Every AI demo works. It is built to, on inputs chosen because they demonstrate well, and it tells you almost nothing about behaviour on your messiest Tuesday. Evaluating an AI vendor is mostly about getting past the demo to the questions whose answers actually vary between suppliers.
The questions below are the ones where a good vendor answers immediately and a weak one changes the subject.
What does it do when it is wrong
Ask for the failure rate on a realistic workload, and ask what happens to the failures. A vendor who has measured this will give you a number and describe the fallback. A vendor who has not will talk about accuracy in general terms.
Then ask whether failures are detectable at the time. A system that is wrong loudly is manageable. One that is wrong silently, producing plausible output nobody flags, is a liability whatever its headline accuracy.
Test it on your own data
Assemble fifty cases from your own records before any demo, including the awkward ones: incomplete records, unusual formats, edge cases your staff argue about. Ask to run those.
- Include cases you know are ambiguous, and see whether the system says so or guesses.
- Include a few where the correct answer is that no action should be taken.
- Score against what your own experienced staff would have done, rather than against a theoretical ideal.
- Keep the set. It becomes your acceptance test and later your regression suite.
A vendor unwilling to run your cases is telling you something. The good ones usually ask for them.
We ask for fifty real records before the first demo. Half the conversations end there, and the ones that continue are with people who have actually run this in production.
Ethan Caldwell, CTO & Co-Founder, Engineered With AI
Where the data goes
Establish which provider is behind the product, where processing happens, whether your inputs are retained, and whether anything you send can be used to improve their models. These have contractual answers and a serious vendor has them ready.
For regulated work, get the processing location in writing rather than in a sales conversation. Retrofitting a residency requirement usually means changing supplier, which is an expensive discovery to make in month four.
What happens when the model changes
The underlying model will be updated or retired on a schedule the vendor does not control. Ask what notice you get, whether you can stay on a version, and how they validate that behaviour has not shifted.
A vendor who has never thought about this will tell you updates are always improvements. In practice a newer model can behave differently enough on your specific task to matter, and finding out from a customer complaint is the bad way.
Ask how you leave
Getting your data out, in a usable format, along with any configuration and prompts you paid to develop, should be a clause rather than a favour. Ask early, while you still have leverage, because the answer also reveals how they think about the relationship.
Who carries it afterwards
Every deployed system needs somebody watching it: an exception queue, an alert, a change upstream that breaks an assumption. Establish whether that is you, them, or unowned, because unowned is the default and it is how systems degrade without anyone noticing.
Pricing deserves the same scrutiny. Usage-based pricing that looks cheap in a pilot can behave very differently at production volume, so model it against your real throughput rather than the pilot. More on how we structure this is in the automation hub.
Ask who built it and who maintains it
A meaningful share of AI products are a thin layer over a provider API, which is fine when priced accordingly and a problem when priced as proprietary technology. Ask directly what they built themselves and what they are reselling.
The answer changes how you evaluate an AI vendor on both price and risk. A wrapper is easier to replace and easier to replicate, and it also means their roadmap depends on decisions made somewhere else.
Ask to speak to a customer who stopped
Reference calls with happy customers are chosen for you. Asking to speak to someone who churned, or who ran a pilot that did not proceed, is unusual enough that the reaction tells you a lot even when the introduction never happens.
Assessing a vendor proposal?
We will review the technical claims and tell you which ones are load-bearing.