Shipping an AI application: spending caps, caching, monitoring and model changes
The Ship AI Engineer team · Published 2026-10-02
In an application's first month everything looks fine: the server is up, the error rate is low, pages are fast. The model bill that arrives at the end of the month is several times what was expected. The logs show why: a small share of users had very long conversations, and one had been calling the app from a script nonstop. None of it showed on the server monitoring dashboard, because from the server's point of view everything was fine.
Shipping an AI application involves all the work of shipping an ordinary web app. It adds one more thing: every request's cost is variable, and left unchecked it grows.
Why is shipping an AI application different?
Because each request costs a different amount depending on its length and content, and that cost grows directly with user behaviour.
In an ordinary app, serving a page costs roughly a fixed amount. In a language model call, the cost depends on how many tokens are sent and produced. A long conversation, a big document or an image can multiply the cost of a single request. So infrastructure monitoring on its own isn't enough; you also have to watch how tokens and cost are distributed across requests. You can try why a conversation's cost grows faster than its message count in the cost game.
How do you set up a spending cap?
With a per-user spending limit, an account-level budget alert, and a ceiling on how big a single request can get.
Three layers work together. A per-user limit stops a single user, or a bot, from spending the whole budget. A budget alert, set up in the provider's dashboard or in your own logs, tells you when total spend rises unexpectedly, before the end of the month. A per-request ceiling (how long the input may be, how many turns of history are kept, the most tokens an answer may use) stops one request from becoming unexpectedly expensive. Routing easy requests to a smaller, cheaper model and hard ones to the large model is also one of the most effective ways to cut cost, but only if you've measured that the small model is good enough for those requests.
When does caching help?
In two different places: the provider caching the fixed part of the prompt, and you storing the answers to identical questions.
Most providers can cache the start of the prompt that's identical across requests and process it more cheaply and quickly. For that to work, fixed parts like system instructions and examples have to come first and changing parts last. The second kind of cache is storing the answer to exactly the same question; for frequently asked questions it cuts cost and latency sharply. But there's a serious trap here: if the cache key doesn't separate users or customers, an answer given to one customer, with their data in it, can be shown to another. It's a data leak that no test run with a single user will ever show.
What should you monitor?
Error types, time to first token, tokens and cost per user, and signals about answer quality.
Separate errors by type: rate limits, provider errors, timeouts and answers that come back in an invalid format should be tracked as separate numbers, because each has a different fix. For latency, look at time to first token as well as total time; it's the delay users actually feel. Track cost per user and per feature, so you can see where spending comes from. For quality, follow at least negative user feedback and the rate at which the model says "I don't know". And set alerts only for situations where someone needs to do something; alerting on everything ends with nobody looking at any of it.
Where should secrets and configuration live?
Separate from the code, in environment variables or a secrets manager, with different values for each environment.
API keys, database addresses and model names shouldn't be written into the code. Development, test and production should each use their own values, so an experiment in the test environment doesn't spend the production budget and the production key doesn't sit on a developer's laptop. Keeping the model name as configuration also turns changing the model later into something that needs no code change.
What happens when the provider retires a model?
After a set date, requests to that model stop working; if you're not prepared, the product stops.
Hosted models are renewed at intervals and old ones are retired. Everything you build should assume the model you chose won't exist in two years. That takes two things: the model name living in configuration rather than code, and your evaluation set being ready. Running the same test set against the new model and comparing results before switching turns the migration from a guess into a measurement. We covered how to build that set in our evaluation post, and how to protect the endpoint from abuse in our security post.
Sources
- Anthropic: Prompt caching: caching the fixed start of the prompt, and its effect on cost. (added 2026-10)
- MDN: HTTP caching: the general principles of caching and how keys are built. (added 2026-10)
- Google SRE Book: monitoring distributed systems: what to monitor, and why alerts should be few and meaningful. (added 2026-10)
- The Twelve-Factor App: config: keeping configuration and secrets out of the code. (added 2026-10)
Keep reading
2026-10-06
Resources worth following as an AI engineer: what to read, and when
A new post, model or tool appears in this field every day, and trying to follow all of it is the fastest way to learn none of it. A handful of resources that genuinely help, and the stage of learning at which each one is worth reading.
2026-10-05
Being an engineer in the age of AI: code got cheap, structure didn't
Writing code has never been this fast. But code produced quickly amplifies whatever structure sits underneath it. Why engineering matters more today, not less, and where to go deeper.
2026-09-30
An AI engineer portfolio: what a project that gets you hired looks like
A chatbot demo impresses nobody anymore. Six signs that make an AI project credible, five project ideas that each show a different skill, and a README someone can read in two minutes.
2026-09-27
Prompt injection and abuse: why the server has to be the single source of truth in an AI app
An application with a language model inside carries every security problem of an ordinary web app and adds new ones. Why prompt injection can't be fully prevented, how to limit the damage, protecting the endpoint, and why the model's output can't be trusted.
2026-09-23
Building your first test set for an LLM application: a guide to evaluation
You changed the prompt and the answers look better. Do they? Building a first test set, taking a baseline, seeing what the average hides, the limits of one model grading another, and watching the system once it's live.
2026-09-19
When should you fine-tune, and when is it a waste of time?
Training a model on your own data is an attractive idea. Most of the time it's the wrong tool. What fine-tuning teaches and what it doesn't, how it compares with prompting and RAG, data quality, leakage, LoRA, and licences on open models.
2026-09-14
7 questions to ask before you pick an AI engineering course
Seven questions worth asking before you spend money and months on an AI engineering course. A short answer to each, and the concrete sign you can check in the syllabus before you buy.
2026-09-09
AI with image and audio input: the costs and failures your tests don't show
Handing a model a photo or a recording isn't the same as sending text. Request size, unreadable images, sentences invented in silence, generation that takes seconds, and an image paid for again on every turn.
2026-09-04
When don't you need an agent? Building and stopping a tool-using system safely
"Agent" is today's most requested and most easily misbuilt thing. The difference between a fixed workflow and an agent, why tool descriptions are critical, how to stop the loop, and how to protect irreversible actions.
2026-08-31
Why does RAG give wrong answers? Most failures are in retrieval, not the model
When a system that answers from your documents says something wrong, the model is the first suspect. Most of the time the culprit is the chunk it was given. Finding where the failure is, containing invention, citing sources and measuring properly.
2026-08-26
How to become an AI engineer: a roadmap that builds products, not models
Most roadmaps in the search results describe a different job. What the AI engineers being hired today actually know, the order to learn it in, and what makes each stage "done".
2026-08-21
When do you actually need a vector database? A practical guide to embeddings
The first step people take after deciding to build semantic search is usually picking a vector database. Most of the time it's an early step. What an embedding is, what a similarity score doesn't tell you, and the real reasons to move to a database.
2026-08-17
Managing prompts like code: versions, templates, schemas and tests
A prompt that works in a chat window quietly breaks inside an application. Keeping prompts in the repository, separating user input from instructions, validating the answer against a schema, and shipping a change only after measuring it.
2026-08-12
Why LLM APIs break under real traffic: rate limits, retries and timeouts
Three errors you never saw in development arrive together in the first busy hour. Telling 429 from 5xx, retrying properly, setting timeouts, and keeping the product up when the provider goes down.
2026-08-07
Do you need maths to become an AI engineer?
The honest answers to the six questions that stop people before they start: maths, degrees, machine learning, Python, hardware and money. Short answers first, reasons after.
2026-08-07
An AI engineer is not a machine learning engineer
Search "how to become an AI engineer" and the first page teaches linear algebra, CNNs and training loops. That is a different job. Here is what the work actually is, and what you can skip.