Users are used to software that responds almost instantly. Language model calls routinely take several seconds, and chained calls can take much longer. The result is an AI feature that works well and still feels broken, because people give up before the answer arrives.
Reducing AI latency comes down to two separate jobs: making the system genuinely faster where possible, and making the unavoidable waits feel shorter.
Measure where the time goes
Before optimising anything, break a typical request into its parts: retrieving context, building the prompt, waiting for the model, processing the output, and any follow-up calls. Record how long each takes for real traffic.
Teams are often surprised by the result. Slow database lookups or several unnecessary model calls frequently account for more delay than the main model call itself.
Stream the response
Streaming sends text to the user as it is generated rather than waiting for the complete answer. The total time barely changes, but the first words appear within a second or two, and users perceive the feature as far faster.
Streaming suits conversational answers and long explanations. It is less useful when the output must be complete and validated before anything is shown, such as structured data feeding another system.
Streaming is the cheapest latency improvement available. It changes nothing about the model and almost everything about how the feature feels.
Sam Ortiz, Director of Engineering, Engineered With AI
Use a smaller model where it is enough
Larger models are generally slower. Many tasks, such as classification, extraction and short rewrites, run perfectly well on smaller, faster models, and routing those tasks away from the largest model cuts response times considerably.
Test the smaller model on your own evaluation set before switching. A faster answer that is noticeably worse is not an improvement.
Send less
Long prompts take longer to process. Trimming repeated instructions, sending only the relevant sections of documents and summarising long conversation histories all reduce the time before the model starts responding.
Requesting shorter answers helps too, because generation time grows with output length. If users only need three sentences, ask for three sentences.
Run steps in parallel
Systems that make several model calls often run them one after another even when they do not depend on each other. Running independent calls at the same time can cut the total wait dramatically.
Map which steps genuinely depend on earlier results. Only those need to wait.
Cache what repeats
Identical or near-identical questions are common in many products. Caching those answers returns them instantly. Provider-side caching of long, fixed prompt prefixes also reduces the time spent processing the same instructions repeatedly.
Cached answers need an expiry policy, so users are not served stale information after the underlying data changes.
Do slow work ahead of time
Some work can be done before the user asks: summarising documents when they are uploaded, pre-computing embeddings, preparing daily briefings overnight. The user then receives a result that was already waiting.
This pairs well with queuing background work, covered in designing around LLM rate limits.
Design the wait
When a delay cannot be avoided, show progress. A clear indication of what the system is doing, such as searching documents or drafting a reply, reassures users that something is happening.
For long tasks, let users continue working and notify them when the result is ready. Forcing someone to watch a spinner for thirty seconds turns a useful feature into an irritating one.
Track it over time
Latency tends to creep up as prompts grow and features are added. Monitor typical and slowest response times alongside quality, and investigate when either moves.
Setting a target for the slowest acceptable response keeps performance on the agenda rather than something noticed only after complaints.
Consider where the servers are
Network distance adds delay to every call. If your application runs in one region and the model endpoint sits on another continent, each round trip carries a cost that multiplies across chained calls.
Where providers offer regional endpoints, choosing one close to your application can shave noticeable time off each request. Check the data residency implications at the same time, since the region also determines where your data is processed.
AI features feeling sluggish?
We will profile where the time goes and make the experience feel faster.
