Support automation is usually sold on deflection rate, which is the share of conversations resolved without a person. That number is easy to raise and easy to raise badly, because a conversation can be deflected by answering it or by exhausting the customer into giving up.
A system that scores well on deflection and poorly on whether people got what they needed is worse than no automation, because the damage is invisible in the reporting and visible to every customer.
Deflect the routine, escalate the rest
Most support volume is a small number of repeated questions with defined answers: order status, opening hours, how to change a booking, password resets, where to find a document. That tier automates cleanly and it is usually a larger share of the queue than teams expect, because the difficult conversations are the memorable ones.
Everything else should reach a person quickly and visibly. Complaints, anything involving money going the wrong way, anything the customer has already contacted you about, and anything expressed with obvious frustration all belong with a human immediately rather than after three attempts at a scripted answer.
Make the escape route obvious
The single design decision that separates helpful automation from the version people complain about is how easy it is to reach a person. Hiding that option raises the deflection number and lowers the actual resolution rate, and customers learn quickly to type the words that break out of the flow.
Offer it plainly, early, and without penalty. Counterintuitively, systems that make it easy to reach a human tend to have better automation numbers, because people are willing to try the automated answer first when they know they are not trapped in it.
A deflection rate is only meaningful next to whether the customer came back. Half the conversations we see marked resolved end with the person opening a second ticket an hour later.
David Kwon, Head of Automation, Engineered With AI
Ground answers in your own content
Answers should come from your documentation and policies rather than from the model's general knowledge, which is how a support system ends up confidently describing a returns policy you do not operate. Retrieval keeps the content outside the model so it updates the moment your source changes.
That also means the quality ceiling is set by your help content. A system built on documentation that is out of date will be confidently wrong, and the honest first deliverable in that case is fixing the content, which improves human support at the same time.
The measures that matter
- Reopen rate: how often a resolved conversation comes back.
- Escalation rate, tracked over time as documentation improves.
- Time to a human when one is requested.
- Customer satisfaction split between automated and human resolutions.
That last split is the one that exposes a bad system fastest. If satisfaction on automated conversations is materially lower, the deflection is coming at a cost the headline number is hiding.
Let it say it does not know
A system permitted to say it cannot answer, and to hand over cleanly, converts a class of bad answers into a visible gap somebody can fill. Systems that never refuse are not confidently correct; they are answering things they should not, and the refusal path needs testing deliberately rather than assumed.
Start narrow and widen on evidence
Begin with the ten questions you are certain about, measure for a month, then extend. Widening the scope because the system seemed to be coping is how it ends up answering something consequential badly. Our support agent index covers how that boundary is drawn in regulated settings specifically.
Where the content gaps show up first
The questions an automated system cannot answer are a direct list of what your documentation is missing, and it is the most useful output of the first month. Reviewing them weekly turns the deployment into a content programme that improves human support at the same time, because the same gaps were slowing your own team down.
That loop is what makes deflection improve honestly over time. Systems that plateau usually stopped feeding the unanswered questions back into the source material and started tuning prompts instead.
Want automation that knows when to stop?
We scope what the system may answer and what it must escalate before anything goes live, so deflection never becomes obstruction.