Most large model bills grow for structural reasons rather than because usage grew. The prompt got longer, retrieval started passing more context, and a capable expensive model is handling work a smaller one could do. None of that is visible from a monthly total.
The first step is finding out where the tokens go, which is usually surprising.
Measure per request, not per month
Log input and output tokens per request type. Almost every system has a small number of request shapes doing most of the spending, and they are frequently not the ones anybody expected. A background summarisation job running on every record can quietly outspend the customer-facing feature that justified the project.
Until that breakdown exists, cost reduction is guesswork and usually targets the visible feature rather than the expensive one.
Prompt size is the usual culprit
- System prompts accumulate instructions over months and are rarely pruned.
- Retrieval passes more chunks than the answer needs, because more felt safer.
- Conversation history is resent in full when a summary would do.
- Examples included for a problem that was fixed months ago are still being sent.
Input tokens are cheaper than output tokens and there are usually far more of them, which is why prompt bloat dominates the bill in most systems.
The largest saving we find is almost always retrieval passing ten chunks where three would answer the question. Nobody tuned it down after launch because nothing was obviously wrong.
Lena Fischer, Solutions Architect, Engineered With AI
Route work to the right model
Not every request needs the most capable model. Classification, extraction, routing and simple rewriting are frequently handled well by a smaller and much cheaper one. The pattern that works is routing by task type, with an escalation path when the small model returns low confidence or fails validation.
Test this properly rather than assuming. Build the evaluation set first, then compare models against it, because the intuition about which tasks need the large model is often wrong in both directions.
Cache what repeats
Identical or near-identical requests are common in production, particularly for reference questions. Caching responses for a sensible period removes that spend entirely. Where the provider supports caching parts of a prompt, a long stable system prompt is exactly the thing to cache.
Constrain the output
Output tokens cost more, so asking for a structured response with defined fields rather than a discursive answer reduces cost and improves reliability at the same time. Instructing for brevity works less well than giving a schema, because a schema removes the room to elaborate rather than requesting restraint.
Know when to stop optimising
Beyond a point, further reduction trades answer quality for a saving that is small against an engineer’s time. Set a target cost per request that the business case supports, reach it, and move on. Systems tuned past that point tend to acquire fragile logic that costs more to maintain than it saves.
For how these decisions are made per sector, our automation index covers the processes that justify the spend.
Streaming and perceived cost
Streaming a response does not reduce token spend and it changes what users tolerate, which matters when the alternative is choosing a faster and more expensive model to reduce waiting. Where latency is the actual complaint, streaming from a cheaper model frequently resolves it at a fraction of the cost of upgrading.
It is worth separating the two problems before spending. Teams routinely move to a more capable model to fix a delay that was a user experience issue rather than a capability one, and the bill increase is permanent while the perceived improvement is modest.
Set a budget alert before you need one
Model spend grows in steps rather than smoothly, usually when a feature is extended to more records or a retrieval setting is loosened. An alert at a threshold you choose deliberately turns a surprise at month end into a question asked on the day it changed, when the cause is still obvious.
Tagging requests by feature makes that diagnosis trivial. Without tagging, a bill that doubled tells you only that something changed, and reconstructing which feature caused it after the fact is far more work than adding the tag would have been.
Bill growing faster than usage?
We will review where your tokens are going and tell you which reductions are safe and which would cost you answer quality.
