Most AI features are reported on with usage statistics: requests per day, users who tried it, tokens consumed. None of those say whether the business is better off, and a feature can be heavily used while making the work slower and more expensive.
Measuring AI ROI properly means deciding, before anything is built, what the feature is supposed to change and how that change will be observed. Retrofitting the measurement after launch almost never works, because the comparison is gone.
Take the baseline first
Whatever the feature is meant to improve, measure it for a few weeks beforehand: how long the task takes now, how often it is done, the current error rate, what it costs in salary time. Without those figures every later claim is an assertion.
Baselines are usually easier to gather than teams fear. Timing a sample of twenty cases and asking the people who do the work is enough for a credible comparison.
Pick one primary measure
Choose the single number the feature exists to move: minutes per document, tickets resolved without escalation, days to close the month, proportion of quotes issued same day. One primary measure keeps the argument honest and stops success being redefined after the fact.
Keep two or three supporting measures for context, including at least one guard against the obvious failure mode, such as quality or rework, so speed gained is not quality lost.
Count the full cost
The model bill is only part of it. Include the engineering time to build and maintain, the review time users spend checking output, integration work, evaluation upkeep and whatever the feature costs when it fails and something has to be redone.
Review time is the most commonly forgotten and often the largest. A feature that saves ten minutes of drafting but adds eight of checking is worth much less than it appears.
Adoption is not value. Plenty of features are used every day by people who would be quicker without them.
David Kwon, Head of Automation, Engineered With AI
Measure quality alongside speed
Track how often output is accepted unchanged, lightly edited or rejected. Those three buckets, recorded by the people already reviewing the work, are the cheapest quality signal available and they show drift long before anyone complains.
A rising edit rate after a model change is a real regression even when nothing errored, and it is invisible in usage statistics.
Compare against the honest alternative
The comparison is rarely AI against nothing. It is AI against a rules engine, a better form, a template, a search box or simply hiring someone. Some of those are cheaper, more predictable and easier to maintain.
Being willing to conclude that a simpler option wins is what makes the rest of your AI programme credible with the people approving it.
Review on a schedule
Set a review date when the feature launches, typically at one month and three months, and put it in the calendar. Features without a review date drift on indefinitely, accumulating cost and quietly becoming part of the furniture.
At each review decide explicitly: expand, leave alone, improve or retire. Writing down which one you chose, and why, is what makes the next review quick.
Be prepared to switch it off
Retiring a feature that does not pay is a good outcome, not a failure. It frees engineering attention, removes a maintenance burden and, importantly, tells the organisation that these decisions are made on evidence rather than enthusiasm.
Keep the evaluation set and the notes when you do. The same idea often becomes viable a year later when models improve or costs fall, and starting from the previous work saves repeating it, as described in taking an AI pilot into production.
Ask the people using it
Numbers miss things a two-minute conversation catches. Ask the people who use the feature daily whether they would object if it were removed tomorrow, and listen to the hesitation as much as the answer. Staff who have quietly built a workaround, or who use it for something other than its intended purpose, are telling you something no dashboard will.
Report it in business terms
Whoever approves the budget cares about cost, capacity and risk rather than tokens or latency. Translating the result into hours returned to the team, work handled without extra headcount, or errors avoided is what turns a technical success into a funded programme, and it makes the eventual decision to retire something easier to explain too.
Can you prove your AI feature works?
We will define the measures and the baseline before you build.