Why LLM APIs break under real traffic: rate limits, retries and timeouts
The Ship AI Engineer team · Published 2026-08-12
Your first integration with a language model API is most likely one line: send the request, print the answer. It works perfectly in development, because in development you send a few requests a minute and the provider is quiet at that hour.
In the first busy hour after you ship, the same line breaks in three different
ways. Some requests come back 429. Some never come back, and the user watches a
spinner. Some fail with a transient error on the provider's side. If your code
handles all three the same way (as "an error happened", like most first
integrations), it handles all three wrongly. This post is about telling them apart
and giving each the right response.
What is the difference between a 429 and a 500 error?
A 429 means "you're going too fast, wait a moment"; a 5xx means "the problem is on our side". Both can be retried, for different reasons.
429 Too Many Requests says your account went over its requests-per-minute or
tokens-per-minute limit. The problem is your traffic, and waiting fixes it. 500,
502, 503 or a provider-specific "overloaded" error say the provider couldn't
handle the request just then; these are usually transient too. The distinction
that really matters is between those two and the 400 family: a 400 (invalid
request), a 401 (bad key) or a request that exceeds the context window won't fix
itself on retry. Retrying them only gets you the same error, more expensively and
more slowly. The first code you should write is the code that sorts errors into
those two groups.
How should you retry?
With growing waits, some randomness, and a ceiling.
Retrying immediately after an error adds load to a provider that is already loaded. The right approach is exponential backoff: a short wait after the first attempt, twice that after the next, twice again after that. Adding jitter matters as much as the backoff itself. If a hundred clients fail at the same moment, wait the same time and retry together, they hit the provider together again. Adding a random share to each wait spreads that wave out.
If the provider sends a Retry-After header, follow it instead of your own
calculation; it knows better than you when you can come back. And always set a
ceiling: giving up after three or four attempts and showing the user a meaningful
message is better than retrying forever.
How long should a timeout be?
As long as the user can reasonably wait, not as long as the model might take.
A request with no timeout hangs forever when the provider doesn't answer, and it eats your server's connections while it does. Many HTTP clients default to a very long timeout or none at all, so set it explicitly. For an endpoint that produces long answers, it helps to separate two things: how long the connection takes to establish (should be short) and how long the answer takes to finish (can be longer). If you stream the answer, the user sees the first word right away, and the number to watch is no longer total time but time to first token.
When is retrying dangerous?
When the request has a side effect.
The worst cost of retrying a plain text-generation request is paying for the tokens twice. But if the model is calling a tool (sending an email, creating an order, starting a payment), the first request may actually have succeeded and only timed out before its answer reached you. Retrying then does the action twice. The fix is to give every operation with a side effect an idempotency key, and when a second request arrives with the same key, return the first result instead of doing the work again. Payment systems have applied this rule for years, and it holds for every AI system that uses tools.
What should happen when the provider is down completely?
The product shouldn't fall over; one feature should switch off for a while.
Every provider has outages. The real question is what the user sees during one. In a well-designed system, the AI-dependent feature says "not available right now" and the rest of the application keeps working. One step further: after a run of consecutive failures, stop calling the provider for a short while (a circuit breaker), so users don't each wait out a long timeout and you don't add load to a provider that is trying to recover. Failing over to a second provider is possible too, but only if you've measured it on the same question set; otherwise you've quietly switched to a model whose quality you don't know.
What should you log?
On every request, at least: which model, how many input and output tokens, how long it took, which attempt succeeded, and the error code if it failed.
Without that record, none of "why is it slow", "why is it expensive" and "why does it sometimes fail" can be answered. Token counts matter most: they're what the bill is calculated from, and watching tokens per request over time is the only way to see a long conversation's cost multiplying. You can try why conversation cost grows faster than the message count in the cost game, and follow a real system breaking under heavy traffic, step by step, in the failure cases.
Why do you need a per-user limit?
Because your account's rate limit is a budget shared by all your users.
One user (or one bot) sending hundreds of requests a minute uses up your account's
limit for everyone, and your other users start getting 429. Putting a per-user
limit and a spending cap on every endpoint that calls a language model protects
that shared budget and keeps your bill out of someone else's hands. It's also the
first step in securing an AI application.
Sources
- Anthropic: errors and rate limits: what each error code means and which ones can be retried. (added 2026-08)
- AWS Architecture Blog: exponential backoff and jitter: the classic post showing, with measurements, why jitter is needed. (added 2026-08)
- MDN: Retry-After header: how a server says when you may try again. (added 2026-08)
- OWASP: API Security Top 10: Unrestricted Resource Consumption: how an API without per-user limits gets abused. (added 2026-08)
Keep reading
2026-10-06
Resources worth following as an AI engineer: what to read, and when
A new post, model or tool appears in this field every day, and trying to follow all of it is the fastest way to learn none of it. A handful of resources that genuinely help, and the stage of learning at which each one is worth reading.
2026-10-05
Being an engineer in the age of AI: code got cheap, structure didn't
Writing code has never been this fast. But code produced quickly amplifies whatever structure sits underneath it. Why engineering matters more today, not less, and where to go deeper.
2026-10-02
Shipping an AI application: spending caps, caching, monitoring and model changes
In an ordinary web app every request costs roughly the same. In an app with a language model inside, it doesn't. Capping cost when you ship, using caching safely, watching the right things, and being ready when the provider retires a model.
2026-09-30
An AI engineer portfolio: what a project that gets you hired looks like
A chatbot demo impresses nobody anymore. Six signs that make an AI project credible, five project ideas that each show a different skill, and a README someone can read in two minutes.
2026-09-27
Prompt injection and abuse: why the server has to be the single source of truth in an AI app
An application with a language model inside carries every security problem of an ordinary web app and adds new ones. Why prompt injection can't be fully prevented, how to limit the damage, protecting the endpoint, and why the model's output can't be trusted.
2026-09-23
Building your first test set for an LLM application: a guide to evaluation
You changed the prompt and the answers look better. Do they? Building a first test set, taking a baseline, seeing what the average hides, the limits of one model grading another, and watching the system once it's live.
2026-09-19
When should you fine-tune, and when is it a waste of time?
Training a model on your own data is an attractive idea. Most of the time it's the wrong tool. What fine-tuning teaches and what it doesn't, how it compares with prompting and RAG, data quality, leakage, LoRA, and licences on open models.
2026-09-14
7 questions to ask before you pick an AI engineering course
Seven questions worth asking before you spend money and months on an AI engineering course. A short answer to each, and the concrete sign you can check in the syllabus before you buy.
2026-09-09
AI with image and audio input: the costs and failures your tests don't show
Handing a model a photo or a recording isn't the same as sending text. Request size, unreadable images, sentences invented in silence, generation that takes seconds, and an image paid for again on every turn.
2026-09-04
When don't you need an agent? Building and stopping a tool-using system safely
"Agent" is today's most requested and most easily misbuilt thing. The difference between a fixed workflow and an agent, why tool descriptions are critical, how to stop the loop, and how to protect irreversible actions.
2026-08-31
Why does RAG give wrong answers? Most failures are in retrieval, not the model
When a system that answers from your documents says something wrong, the model is the first suspect. Most of the time the culprit is the chunk it was given. Finding where the failure is, containing invention, citing sources and measuring properly.
2026-08-26
How to become an AI engineer: a roadmap that builds products, not models
Most roadmaps in the search results describe a different job. What the AI engineers being hired today actually know, the order to learn it in, and what makes each stage "done".
2026-08-21
When do you actually need a vector database? A practical guide to embeddings
The first step people take after deciding to build semantic search is usually picking a vector database. Most of the time it's an early step. What an embedding is, what a similarity score doesn't tell you, and the real reasons to move to a database.
2026-08-17
Managing prompts like code: versions, templates, schemas and tests
A prompt that works in a chat window quietly breaks inside an application. Keeping prompts in the repository, separating user input from instructions, validating the answer against a schema, and shipping a change only after measuring it.
2026-08-07
Do you need maths to become an AI engineer?
The honest answers to the six questions that stop people before they start: maths, degrees, machine learning, Python, hardware and money. Short answers first, reasons after.
2026-08-07
An AI engineer is not a machine learning engineer
Search "how to become an AI engineer" and the first page teaches linear algebra, CNNs and training loops. That is a different job. Here is what the work actually is, and what you can skip.