Building your first test set for an LLM application: a guide to evaluation
The Ship AI Engineer team · Published 2026-09-23
A team rewrites the prompt of a system that classifies customer messages. Three people on the team try the new version on a few examples and say it's "much better". The new prompt ships. Six weeks later someone notices that a large share of billing complaints went to the wrong team the whole time. The new prompt really was better overall; it was just clearly worse in one category, and nobody looked at that category.
Evaluation is the sharpest line between the person who demos an AI system and the person who ships it. And it's the step most often skipped.
What is LLM evaluation?
Measuring a system's answers on a fixed set of questions, with a repeatable method.
The goal is to answer one question: did my change make the system better or worse? To do that, the same questions, the same definition of "correct" and the same scoring method are used every time. Then the difference between two versions is a difference in the system, not in how it was measured.
Why isn't "I think it got better" enough?
Because a few examples and an impression don't show what the change broke.
A language model's behaviour shifts with its input. A prompt change can improve nine questions out of ten and break the tenth, and a try with a few examples very likely doesn't include that tenth. People are also inclined to rate a new version well because they wrote it. A fixed measurement removes both problems.
How do you build a first test set?
With thirty to fifty examples from real usage, and by writing down what "correct" is before looking at any answers.
You don't need a big set to start. Collect thirty to fifty real user messages (or, if you have none yet, realistically written ones) and write the answer you'd accept for each. Writing those answers before you see the system's output matters; otherwise you grade generously against whatever you were shown. Alongside the easy examples, be sure to include hard and borderline ones, messages with typos, messages asking more than one thing, and questions where the system should say "I don't know".
Why do you need a baseline?
Because a score means nothing without something to compare it to.
"Accuracy is eighty-five percent" doesn't, on its own, say whether that's good or bad. Before you change anything, measure the current system on the same set; that's your baseline. If you can, measure the simplest alternative too: a smaller model, a shorter prompt, or a plain rule that uses no model at all. Sometimes it turns out the expensive solution barely beats the simple one. A score without a baseline is a number that feels like a measurement.
Why does the average mislead?
Because it hides a big drop in one area behind small gains in others.
That's exactly what happened to the team above: overall accuracy rose, accuracy in the billing category fell. So report the score per category, not just as a total. Some errors cost far more than others: an urgent message marked "not urgent" is much more expensive than the reverse. Tracking that kind of error as its own number makes the risk the average would hide visible.
What should you measure?
Format in code, content with a scoring method, and cost and latency every time.
The format of an answer (is it valid JSON, are the expected fields there, is it one of the allowed categories) can be checked exactly in code, and always should be. The correctness of the content is measured either by exact comparison or against a scoring rubric, depending on the kind of answer. On top of that, record average cost and latency on every run: a version that's more accurate but twice as expensive isn't always the right choice.
When can you trust one model grading another?
When it's been compared against human judgement and you've measured how often they agree.
Open-ended answers (a summary, an explanation) can't be compared exactly. In those cases having one model score another's answer against a rubric (LLM-as-judge) is a practical route. But that grader has its own biases: it can prefer longer answers, favour whichever of two options it saw first, and reward answers that resemble its own writing. So before you use it, measure how often it agrees with a human on a small sample that a person has also scored. If agreement is low, the numbers the grader gives aren't a measurement.
How do you watch the system once it's live?
By sampling real traffic regularly and running the test set automatically on every change.
The test set is your measurement before release. After release, real users ask questions that aren't in it. Sampling real traffic regularly and checking it, collecting users' negative feedback, and adding newly found failures to the test set brings the set closer to reality over time. Running the test set automatically with every code change is how you catch a drop in the score before users notice it. We explained why "does it teach evaluation?" is the most important question about a course in our course-choice post.
Sources
- Hugging Face: Evaluation Guidebook: a practical guide to evaluating language models, including the biases of LLM-as-judge. (added 2026-09)
- Anthropic: Create strong empirical evaluations: how to design a test set. (added 2026-09)
- EleutherAI: LM Evaluation Harness: an open tool for measuring models on standard tasks. (added 2026-09)
- Hugging Face: Evaluate library: ready implementations of common metrics. (added 2026-09)
Keep reading
2026-10-06
Resources worth following as an AI engineer: what to read, and when
A new post, model or tool appears in this field every day, and trying to follow all of it is the fastest way to learn none of it. A handful of resources that genuinely help, and the stage of learning at which each one is worth reading.
2026-10-05
Being an engineer in the age of AI: code got cheap, structure didn't
Writing code has never been this fast. But code produced quickly amplifies whatever structure sits underneath it. Why engineering matters more today, not less, and where to go deeper.
2026-10-02
Shipping an AI application: spending caps, caching, monitoring and model changes
In an ordinary web app every request costs roughly the same. In an app with a language model inside, it doesn't. Capping cost when you ship, using caching safely, watching the right things, and being ready when the provider retires a model.
2026-09-30
An AI engineer portfolio: what a project that gets you hired looks like
A chatbot demo impresses nobody anymore. Six signs that make an AI project credible, five project ideas that each show a different skill, and a README someone can read in two minutes.
2026-09-27
Prompt injection and abuse: why the server has to be the single source of truth in an AI app
An application with a language model inside carries every security problem of an ordinary web app and adds new ones. Why prompt injection can't be fully prevented, how to limit the damage, protecting the endpoint, and why the model's output can't be trusted.
2026-09-19
When should you fine-tune, and when is it a waste of time?
Training a model on your own data is an attractive idea. Most of the time it's the wrong tool. What fine-tuning teaches and what it doesn't, how it compares with prompting and RAG, data quality, leakage, LoRA, and licences on open models.
2026-09-14
7 questions to ask before you pick an AI engineering course
Seven questions worth asking before you spend money and months on an AI engineering course. A short answer to each, and the concrete sign you can check in the syllabus before you buy.
2026-09-09
AI with image and audio input: the costs and failures your tests don't show
Handing a model a photo or a recording isn't the same as sending text. Request size, unreadable images, sentences invented in silence, generation that takes seconds, and an image paid for again on every turn.
2026-09-04
When don't you need an agent? Building and stopping a tool-using system safely
"Agent" is today's most requested and most easily misbuilt thing. The difference between a fixed workflow and an agent, why tool descriptions are critical, how to stop the loop, and how to protect irreversible actions.
2026-08-31
Why does RAG give wrong answers? Most failures are in retrieval, not the model
When a system that answers from your documents says something wrong, the model is the first suspect. Most of the time the culprit is the chunk it was given. Finding where the failure is, containing invention, citing sources and measuring properly.
2026-08-26
How to become an AI engineer: a roadmap that builds products, not models
Most roadmaps in the search results describe a different job. What the AI engineers being hired today actually know, the order to learn it in, and what makes each stage "done".
2026-08-21
When do you actually need a vector database? A practical guide to embeddings
The first step people take after deciding to build semantic search is usually picking a vector database. Most of the time it's an early step. What an embedding is, what a similarity score doesn't tell you, and the real reasons to move to a database.
2026-08-17
Managing prompts like code: versions, templates, schemas and tests
A prompt that works in a chat window quietly breaks inside an application. Keeping prompts in the repository, separating user input from instructions, validating the answer against a schema, and shipping a change only after measuring it.
2026-08-12
Why LLM APIs break under real traffic: rate limits, retries and timeouts
Three errors you never saw in development arrive together in the first busy hour. Telling 429 from 5xx, retrying properly, setting timeouts, and keeping the product up when the provider goes down.
2026-08-07
Do you need maths to become an AI engineer?
The honest answers to the six questions that stop people before they start: maths, degrees, machine learning, Python, hardware and money. Short answers first, reasons after.
2026-08-07
An AI engineer is not a machine learning engineer
Search "how to become an AI engineer" and the first page teaches linear algebra, CNNs and training loops. That is a different job. Here is what the work actually is, and what you can skip.