Conventional monitoring answers whether the service responded and how fast. For a system built on a language model that leaves the important question untouched, because the endpoint can return quickly, with a valid response, that is quietly worse than it was last week.
AI observability is about detecting that. It needs different signals from the ones an ordinary service exposes, and most teams add them after the first silent failure rather than before.
Log the whole interaction
For every call, keep the input, the resolved prompt including whatever was retrieved or injected, the model and version, the parameters, the raw output, the parsed result, and the latency and token counts. Without the resolved prompt you cannot reproduce anything, and the resolved prompt is the field most often missing.
Sampling is acceptable at high volume, and errors should be kept in full. The cases you most need to study are exactly the ones a uniform sample discards.
Watch the shape of the output
Several cheap signals correlate well with quality drifting, and none require a human reviewer.
- Response length distribution. A sudden shift in average length usually means behaviour changed.
- Schema validation failure rate, which should be near zero and is an early warning when it moves.
- Refusal and fallback rates, which rise when inputs drift or a model becomes more conservative.
- Retry rate, which quietly absorbs a growing problem until someone looks at it.
- Distribution across output categories, where a classifier suddenly favouring one label is a strong signal.
These are all computable without knowing the correct answer, which is what makes them usable as alerts.
The outage you can see gets fixed in an hour. The one that matters is the week where answers were slightly worse and every dashboard stayed green.
Sam Ortiz, Director of Engineering, Engineered With AI
Run a fixed evaluation set continuously
A small set of cases with known good answers, run on a schedule against production configuration, converts quality from something you notice eventually into something you measure hourly.
Keep it small enough to run often and broad enough to cover the main paths. Add every production failure to it once fixed, so the set grows toward the places your system actually breaks rather than the places you imagined it might.
Version everything that affects behaviour
Prompt, schema, retrieval configuration, model version and parameters all change output. If a log entry cannot tell you which combination produced it, investigating a regression becomes guesswork. Stamp the version onto every record.
Cost belongs in the same view
Token spend is a quality signal as well as a financial one. A rising cost per request with flat volume usually means prompts are growing, retrieval is returning more than it should, or retries are climbing, and each of those is a problem worth finding early.
Break it down by feature and by customer where you can. Aggregate spend tells you there is a problem; the breakdown tells you where, and the gap between those two is usually several days of investigation.
Alert on things a person can act on
Alerting on every anomaly produces noise and then indifference. The alerts worth having are the ones with an obvious first action: validation failures above a threshold, evaluation score dropping below a floor, cost per request crossing a line, error rate on a specific path.
Everything else belongs on a dashboard somebody reviews weekly. The distinction between what wakes someone and what waits is worth making deliberately, because a system that cries wolf gets muted and then it is worse than no monitoring at all. This connects closely to keeping agents reliable in production.
Retrieval deserves its own instrumentation
Where a system retrieves context before generating, most quality problems originate in retrieval rather than in the model. Logging what was retrieved, how it scored and whether it was actually used separates a retrieval failure from a generation failure quickly.
Without that split, every investigation starts by re-running the case by hand. AI observability that covers the model call but not the retrieval step leaves the more common fault invisible.
Keep the human signals too
Thumbs-down clicks, edits a user made to generated text, and cases escalated to a person are direct quality evidence that costs nothing to collect. They are also the only signal that reflects what the reader thought rather than what the system measured.
Flying blind on a live system?
We will instrument your pipeline so quality problems surface before a customer finds them.