AI with image and audio input: the costs and failures your tests don't show
The Ship AI Engineer team · Published 2026-09-09
A feature that pulls expense entries out of receipt photos is tested with ten photos taken on the team's phones, and it works perfectly. The week it ships, three different complaints come in: some photos are rejected with a "request too large" error, a blurry receipt yields a consistent but wrong total, and when the feature is used inside a chat the bill runs to several times what was expected.
What the three have in common is that none of them showed up in testing. AI with image and audio input carries every problem text-based AI has, and adds a few of its own.
What is a multimodal model?
A model that can understand images, audio or other kinds of input alongside text.
A vision-language model takes a photo and a question together and answers in text: "what's the total on this receipt?", for example. An image generation model does the opposite: it takes text and produces an image. They're different models doing different jobs, and inside the same product they call for different architectural decisions.
Why does an image cause a "request too large" error?
Because the image is usually sent inside the request as text, and that conversion makes the file bigger.
To send an image inside the request rather than as an address (URL), an encoding called base64 is used, and it grows the data by about a third. So a six-megabyte photo hits a six-megabyte request limit. Photos from phone cameras are often above that limit. The fix is simple but needs thinking about up front: shrink the image to a reasonable resolution before sending it. That avoids the limit and reduces the tokens the image costs, which is to say the money.
Sending the image as an address carries an assumption too: that the provider can reach that address. A private file the user uploaded, or an address reachable only from your own network, can't be opened by the provider.
Why does the model still answer from an image it can't read?
Because saying "I couldn't read it" isn't guaranteed; its job is to produce an answer.
Asked for the total on a blurry receipt, the model will often produce a plausible number. The assumption that "the model tells us when it doesn't know" is wrong often enough to build a system around. So the safeguard has to live in your code, not the model: check the extracted values for internal consistency (do the line items add up to the total?), ask the model to flag the fields it isn't sure of explicitly, and get the user's confirmation when things are uncertain.
Why does speech recognition produce text during silence?
Because transcription models also produce likely text, and silence sometimes gets filled with a sentence nobody said.
A sentence that was never spoken turning up in the transcript of a quiet or noisy stretch of a long recording is a known behaviour. The assumption that "the model produces nothing when there's nothing" is wrong most reliably here. The way to contain it is to process the audio before the model: detect silent stretches and don't send them, and split long audio at points of silence rather than at fixed intervals. The latter also stops a word being cut in half between two chunks.
Why is image generation an architectural decision?
Because it takes seconds, and that doesn't fit inside a normal request-response cycle.
A text answer starts streaming within a few seconds. Generating a few large images can take much longer. Trying to do that inside the same HTTP call as the user's request leads to timeouts and pages that hang. The right structure is to hand generation to a background job, tell the user it has started, and show the result when it's ready. Knowing this from the start is the difference between a feature and a rewrite.
One more thing: image generation models still aren't reliable at putting correctly spelled text inside an image. If a slogan on a poster has to be exact, generating the image and placing the text on it yourself is the more reliable route.
How many times do you pay for an image in a conversation?
By default, once on every turn of the conversation.
Because the model remembers nothing, in a chat about an image the image is sent again on every turn and billed again on every turn. A ten-turn conversation pays for the same image ten times. Tests don't show it, because a test usually asks one question and stops. The fix is the "look once, then continue in text" pattern: on the first turn, extract what you need from the image as text, and on later turns send that text instead of the image. The pattern only breaks when the user asks about a different detail of the image; then you have to send it again. You can try how conversation length affects cost in the cost game.
Sources
- Anthropic: Vision: how images are sent, size limits and cost. (added 2026-09)
- Hugging Face: Vision-language models explained: a general explanation of how vision-language models work. (added 2026-09)
- Jay Alammar: The Illustrated Stable Diffusion: a visual explanation that gives an intuition for why image generation takes time. (added 2026-09)
Keep reading
2026-10-06
Resources worth following as an AI engineer: what to read, and when
A new post, model or tool appears in this field every day, and trying to follow all of it is the fastest way to learn none of it. A handful of resources that genuinely help, and the stage of learning at which each one is worth reading.
2026-10-05
Being an engineer in the age of AI: code got cheap, structure didn't
Writing code has never been this fast. But code produced quickly amplifies whatever structure sits underneath it. Why engineering matters more today, not less, and where to go deeper.
2026-10-02
Shipping an AI application: spending caps, caching, monitoring and model changes
In an ordinary web app every request costs roughly the same. In an app with a language model inside, it doesn't. Capping cost when you ship, using caching safely, watching the right things, and being ready when the provider retires a model.
2026-09-30
An AI engineer portfolio: what a project that gets you hired looks like
A chatbot demo impresses nobody anymore. Six signs that make an AI project credible, five project ideas that each show a different skill, and a README someone can read in two minutes.
2026-09-27
Prompt injection and abuse: why the server has to be the single source of truth in an AI app
An application with a language model inside carries every security problem of an ordinary web app and adds new ones. Why prompt injection can't be fully prevented, how to limit the damage, protecting the endpoint, and why the model's output can't be trusted.
2026-09-23
Building your first test set for an LLM application: a guide to evaluation
You changed the prompt and the answers look better. Do they? Building a first test set, taking a baseline, seeing what the average hides, the limits of one model grading another, and watching the system once it's live.
2026-09-19
When should you fine-tune, and when is it a waste of time?
Training a model on your own data is an attractive idea. Most of the time it's the wrong tool. What fine-tuning teaches and what it doesn't, how it compares with prompting and RAG, data quality, leakage, LoRA, and licences on open models.
2026-09-14
7 questions to ask before you pick an AI engineering course
Seven questions worth asking before you spend money and months on an AI engineering course. A short answer to each, and the concrete sign you can check in the syllabus before you buy.
2026-09-04
When don't you need an agent? Building and stopping a tool-using system safely
"Agent" is today's most requested and most easily misbuilt thing. The difference between a fixed workflow and an agent, why tool descriptions are critical, how to stop the loop, and how to protect irreversible actions.
2026-08-31
Why does RAG give wrong answers? Most failures are in retrieval, not the model
When a system that answers from your documents says something wrong, the model is the first suspect. Most of the time the culprit is the chunk it was given. Finding where the failure is, containing invention, citing sources and measuring properly.
2026-08-26
How to become an AI engineer: a roadmap that builds products, not models
Most roadmaps in the search results describe a different job. What the AI engineers being hired today actually know, the order to learn it in, and what makes each stage "done".
2026-08-21
When do you actually need a vector database? A practical guide to embeddings
The first step people take after deciding to build semantic search is usually picking a vector database. Most of the time it's an early step. What an embedding is, what a similarity score doesn't tell you, and the real reasons to move to a database.
2026-08-17
Managing prompts like code: versions, templates, schemas and tests
A prompt that works in a chat window quietly breaks inside an application. Keeping prompts in the repository, separating user input from instructions, validating the answer against a schema, and shipping a change only after measuring it.
2026-08-12
Why LLM APIs break under real traffic: rate limits, retries and timeouts
Three errors you never saw in development arrive together in the first busy hour. Telling 429 from 5xx, retrying properly, setting timeouts, and keeping the product up when the provider goes down.
2026-08-07
Do you need maths to become an AI engineer?
The honest answers to the six questions that stop people before they start: maths, degrees, machine learning, Python, hardware and money. Short answers first, reasons after.
2026-08-07
An AI engineer is not a machine learning engineer
Search "how to become an AI engineer" and the first page teaches linear algebra, CNNs and training loops. That is a different job. Here is what the work actually is, and what you can skip.