Every language model provider caps how much traffic an account can send: requests per minute, tokens per minute, sometimes concurrent connections. Early prototypes never reach those caps. Production systems reach them on their busiest day, which is the worst possible moment to discover how the system behaves.
Designing around LLM rate limits from the start is far cheaper than retrofitting it during an incident.
Know your actual limits
Limits vary by provider, model and account tier, and they change as usage grows. Find the current figures for the models you use and record them where the team can see them.
Then estimate your own peak. Multiply the busiest hour's expected requests by the average tokens per request, and compare it with the limit. If the peak is anywhere near the cap, design for it now.
Retry properly
When a request is rejected for exceeding a limit, retrying immediately makes things worse, because every client does the same thing at the same moment. Exponential backoff, where each retry waits longer than the last, with a little randomness added, spreads the load out.
- Respect any retry delay the provider returns with the error.
- Cap the number of retries so a request cannot loop forever.
- Add random jitter so clients do not retry in lockstep.
- Log every rate-limit event so the pattern is visible.
Retries handle brief spikes. They do not fix sustained demand above the limit, which needs a different approach.
A retry loop without backoff turns a small spike into an outage. Everything waits a second, then everything tries again at once.
Sam Ortiz, Director of Engineering, Engineered With AI
Queue work that can wait
Not every request needs an answer this second. Summarising yesterday's tickets, tagging documents or generating reports can sit in a queue and be processed at a steady pace that stays under the limit.
Separating interactive work from background work is the single most useful design decision here. It keeps the features customers are waiting on responsive while the bulk jobs trickle through.
Prioritise within the queue
When the queue is under pressure, some jobs matter more than others. Give interactive requests and high-value customers priority, and let low-priority work wait. Without priorities, a large batch job can starve everything else.
Reduce what you send
Token limits are often hit before request limits, because prompts grow over time. Trimming repeated instructions, sending only the relevant parts of documents and summarising long conversation histories all reduce the load without changing results.
Caching identical requests removes them from the count entirely. Many workloads repeat more than teams expect.
Spread the load
Higher limits can often be requested from the provider, and they are worth asking for well before they are needed. Using more than one model or provider for different tasks also spreads the load, though it adds complexity and data-handling questions that need thinking through.
The trade-offs of sending different work to different models are covered in model routing and fallbacks.
Degrade gracefully
When limits are hit despite everything, the user should see something sensible: a message that the answer is on its way, a simpler response, or a queued result delivered later. A blank error screen is the worst outcome.
Decide in advance what each feature should do under pressure. That is a product decision as much as an engineering one.
Test at realistic peaks
Load tests against the provider's real limits reveal how the system behaves before customers do. Test the busiest realistic hour, not the average, and watch queue lengths, retry counts and response times together.
Repeat the test when usage grows or a new feature launches. Limits that were comfortable six months ago can quietly become tight.
Watch the limits you share
Limits usually apply per account or per organisation, which means a test script run by a developer can consume the capacity that production relies on. Separate accounts or projects for development, testing and live traffic prevent one from starving another, and they make usage far easier to attribute when the bill arrives.
Hitting limits at busy times?
We will review how your system handles load and redesign the parts that break.