When do you actually need a vector database? A practical guide to embeddings
The Ship AI Engineer team · Published 2026-08-21
Right after the decision "we'll build a system that answers questions about our documents" comes the question "which vector database should we use?". Comparison tables get read, accounts get opened, infrastructure gets set up. Meanwhile the team has a few thousand documents, and every vector fits comfortably in a laptop's memory.
Vector databases answer a real need. But that need usually isn't where people think it is, in scale. This post covers what embeddings are, then when you really need a database.
What is an embedding?
A fixed-length list of numbers that represents the meaning of a piece of text.
An embedding model takes the sentence "how long is the return window?" and turns it into a vector of a few hundred or a few thousand numbers. Texts close in meaning produce vectors close to each other: "how many days do I have to send the product back?" lands near the first one without sharing a single word. That's the whole idea of semantic search: put the question and the documents into the same space and find the documents nearest the question.
What does a similarity score tell you, and what doesn't it?
It tells you how alike two texts are; it doesn't tell you whether a document answers the question.
Closeness is usually measured with cosine similarity, and there is always a nearest result. Even if the answer to your question isn't anywhere in your documents, search returns a "nearest" result. So somewhere in a retrieval system there has to be a step that decides whether what came back is good enough to use.
The second trap is thresholds. A number you read in a blog post, "above 0.8 is relevant", means nothing for another embedding model, another language or another kind of document. The distribution of scores changes from model to model. The right threshold is found by measuring with your own questions and your own documents.
When do you need a vector database?
Usually not for scale, but for persistence, filtering and deletion.
Up to a few tens of thousands of chunks, keeping every vector in memory and comparing each question against all of them is both simple and fast enough. The needs that really justify a database are usually these:
- Persistence: you don't want to re-embed every document each time the application restarts.
- Filtering: you need conditions like "search only this customer's documents" or "only the last year". In a multi-user system that's a security requirement: one customer's question must never retrieve another customer's document.
- Deletion and updates: when a document changes, or a user asks for their data to be deleted, you have to be able to find and remove the matching vectors.
- Real scale: when you have millions of chunks and need low latency, approximate nearest neighbour (ANN) indexes come in.
For most projects a good first step is the vector support in the relational database you already use: your data and your vectors live in one place, and filtering becomes an ordinary query condition.
When is keyword search better?
When you're looking for an exact value: a product code, an error message, a person or place name.
Embeddings are good at capturing meaning but can be weak at catching a code like "ERR-4021" or a rare proper name. For questions like that, classic keyword search is more reliable. That's why a common solution in real systems is hybrid search: run both and merge their results. Which works better for your questions you can, again, only find out by measuring.
Why is changing the embedding model expensive?
Because vectors from different models can't be compared with each other; when you switch, you have to re-embed everything.
The vector space a model produces is its own. Embedding the question with a new model and comparing it against document vectors made by the old model gives meaningless results. That makes the choice of embedding model more permanent than it first looks. Recording which model produced each vector, and working out up front how long and how much a full re-embed would take, prevents the most common surprise later on.
How should documents be split into chunks?
By units of meaning, not by a fixed number of characters.
Cutting a document every five hundred characters is the easiest method and the one that causes the most errors: it splits a table down the middle and a paragraph in mid-sentence. What works better is splitting on headings and paragraphs, leaving a small overlap between chunks, and putting the heading of the section each chunk belongs to at its start. That simple addition often does more for search quality than switching embedding models, and trying it on your own data costs a few lines of code.
You can see every embedding and search concept, with how they connect, in the free knowledge map.
Sources
- Cohere: What are embeddings?: a short introduction to embeddings from a provider's point of view. (added 2026-08)
- Sentence Transformers: semantic search: how semantic search is done without any database. (added 2026-08)
- FAISS wiki: guidelines for choosing an index: which kind of index you need at which scale. (added 2026-08)
- MTEB: Massive Text Embedding Benchmark: a comparison of embedding models; no substitute for measuring on your own data. (added 2026-08)
Keep reading
2026-10-06
Resources worth following as an AI engineer: what to read, and when
A new post, model or tool appears in this field every day, and trying to follow all of it is the fastest way to learn none of it. A handful of resources that genuinely help, and the stage of learning at which each one is worth reading.
2026-10-05
Being an engineer in the age of AI: code got cheap, structure didn't
Writing code has never been this fast. But code produced quickly amplifies whatever structure sits underneath it. Why engineering matters more today, not less, and where to go deeper.
2026-10-02
Shipping an AI application: spending caps, caching, monitoring and model changes
In an ordinary web app every request costs roughly the same. In an app with a language model inside, it doesn't. Capping cost when you ship, using caching safely, watching the right things, and being ready when the provider retires a model.
2026-09-30
An AI engineer portfolio: what a project that gets you hired looks like
A chatbot demo impresses nobody anymore. Six signs that make an AI project credible, five project ideas that each show a different skill, and a README someone can read in two minutes.
2026-09-27
Prompt injection and abuse: why the server has to be the single source of truth in an AI app
An application with a language model inside carries every security problem of an ordinary web app and adds new ones. Why prompt injection can't be fully prevented, how to limit the damage, protecting the endpoint, and why the model's output can't be trusted.
2026-09-23
Building your first test set for an LLM application: a guide to evaluation
You changed the prompt and the answers look better. Do they? Building a first test set, taking a baseline, seeing what the average hides, the limits of one model grading another, and watching the system once it's live.
2026-09-19
When should you fine-tune, and when is it a waste of time?
Training a model on your own data is an attractive idea. Most of the time it's the wrong tool. What fine-tuning teaches and what it doesn't, how it compares with prompting and RAG, data quality, leakage, LoRA, and licences on open models.
2026-09-14
7 questions to ask before you pick an AI engineering course
Seven questions worth asking before you spend money and months on an AI engineering course. A short answer to each, and the concrete sign you can check in the syllabus before you buy.
2026-09-09
AI with image and audio input: the costs and failures your tests don't show
Handing a model a photo or a recording isn't the same as sending text. Request size, unreadable images, sentences invented in silence, generation that takes seconds, and an image paid for again on every turn.
2026-09-04
When don't you need an agent? Building and stopping a tool-using system safely
"Agent" is today's most requested and most easily misbuilt thing. The difference between a fixed workflow and an agent, why tool descriptions are critical, how to stop the loop, and how to protect irreversible actions.
2026-08-31
Why does RAG give wrong answers? Most failures are in retrieval, not the model
When a system that answers from your documents says something wrong, the model is the first suspect. Most of the time the culprit is the chunk it was given. Finding where the failure is, containing invention, citing sources and measuring properly.
2026-08-26
How to become an AI engineer: a roadmap that builds products, not models
Most roadmaps in the search results describe a different job. What the AI engineers being hired today actually know, the order to learn it in, and what makes each stage "done".
2026-08-17
Managing prompts like code: versions, templates, schemas and tests
A prompt that works in a chat window quietly breaks inside an application. Keeping prompts in the repository, separating user input from instructions, validating the answer against a schema, and shipping a change only after measuring it.
2026-08-12
Why LLM APIs break under real traffic: rate limits, retries and timeouts
Three errors you never saw in development arrive together in the first busy hour. Telling 429 from 5xx, retrying properly, setting timeouts, and keeping the product up when the provider goes down.
2026-08-07
Do you need maths to become an AI engineer?
The honest answers to the six questions that stop people before they start: maths, degrees, machine learning, Python, hardware and money. Short answers first, reasons after.
2026-08-07
An AI engineer is not a machine learning engineer
Search "how to become an AI engineer" and the first page teaches linear algebra, CNNs and training loops. That is a different job. Here is what the work actually is, and what you can skip.