Traditional integrations fail in recognisable ways: a timeout, an error code, a rejected request. Model calls add failures that look like successes. The response arrives quickly, is well formed, reads confidently and is wrong, or is valid JSON with a field quietly missing.
Good AI failure handling assumes all of this will happen in production and decides in advance what the user sees when it does, rather than letting an exception surface as a broken screen.
Enumerate the failure modes
There are more than teams expect: hard errors and rate limits, timeouts, truncated output from a token limit, malformed structured output, refusals, hallucinated content that passes validation, and correct output that arrives too late to be useful.
Writing the list down is most of the work. Each mode needs a decision about what happens, and most teams discover they had only ever handled the first two.
Validate before you trust
Parse and validate every structured response against a schema, then apply business rules: does this invoice total match the sum of the lines, is this date plausible, is the referenced account one that exists? A model can produce perfectly shaped output with impossible values.
Treat a validation failure as an ordinary event rather than an exception. It happens often enough that the retry and fallback path should be as well tested as the success path.
Retry with intent
A blind retry of the whole request costs money and often fails the same way. Retrying with a shortened prompt, a lower temperature, a repaired instruction naming the validation error, or a different model is far more likely to succeed on the second attempt.
Cap attempts and log which attempt succeeded. A feature quietly needing three attempts to work is telling you something about the prompt, and that signal is invisible without the count.
The question is not whether the model will return something unusable. It is what your product does in the two seconds afterwards.
Hannah Berg, Lead AI Engineer, Engineered With AI
Decide the degraded experience
Every AI feature should have an answer to what happens when the model is unavailable. Falling back to a simpler rule-based result, showing the raw source material, queueing the work for later or handing it to a person are all acceptable; a spinner that never resolves is not.
Tell the user plainly when they are getting the fallback. People forgive a system that says the summary is unavailable far more readily than one that silently shows something worse without saying so.
Fail closed where it matters
For anything touching money, safety, health or legal obligations, an uncertain answer should stop and ask rather than guess. Deciding which paths fail closed and which fail open is a product decision, and it should be written down rather than left to whoever wrote the try block.
Related controls belong with it, such as the limits described in access control for AI agents, so a failure never widens what the system is allowed to do.
Protect the rest of the system
A slow model should not exhaust your connection pool or block the request thread for everyone else. Timeouts, bulkheads and a circuit breaker that stops calling a failing provider for a while keep one degraded dependency from taking the product down.
Queue work that can be done later rather than holding a user waiting. Much AI work is not genuinely interactive, and moving it to a background job removes a whole class of failure.
Make failures visible
Track validation failures, refusals, retries and fallbacks as first-class metrics, not just errors. A rise in refusals after a provider update is a real incident even though nothing technically broke, and nobody will notice it in a general error count.
Alert on the rate rather than on individual events, and review the shape of failures monthly. Patterns there usually point at a prompt or a data problem worth fixing at source, which ties into model routing and fallbacks.
Test the failure paths on purpose
Fallbacks rot because nothing exercises them. Force the failures in a test environment: point at an unreachable endpoint, return deliberately malformed output, inject a refusal, and add a delay longer than your timeout. Each one should produce the experience you designed rather than a stack trace, and running those tests in the pipeline keeps it that way after the next refactor.
Does your AI feature fail safely?
We will review your failure paths and make them boring.