When should you fine-tune, and when is it a waste of time?

A company notices its customer support model gives wrong information about its products, and thinks the fix is obvious: train the model on the product documents. Weeks go into preparing data, a training run is done. The new model imitates the company's voice very well, but it still gets the products wrong. And every time the product catalogue changes, the training has to be redone.

Fine-tuning is a powerful tool. But mixed up with the problem it's being asked to solve, it turns into an expensive, slow disappointment.

What is fine-tuning?

Adjusting a ready-made model to a particular task by training it a little further on your own examples.

The model is given pairs of inputs and expected outputs, and it's tuned towards producing answers that look more like those outputs. This isn't training a model from scratch; it's a small adjustment on top of an existing one. The result is a model inclined to answer a particular task in a particular shape and voice.

Does fine-tuning teach the model new knowledge?

Not reliably; what it really teaches is behaviour, format and style.

That's exactly what happened to the company above. Fine-tuning is good at changing how the model answers: in what shape, how long, in what tone, sorting into which categories. It isn't a reliable way to put changing, must-be-correct information (prices, stock, policies) into the model. The model can misremember what it saw in training, mix it up, and keep saying the old version after the information changes. For changing information the right tool is RAG: finding the information in documents for every question and handing it to the model. We covered the difference between the two in our post on RAG.

What should you try before fine-tuning?

A good prompt first, then RAG if needed; fine-tuning only if neither is enough.

The reason for that order is cost and speed. A prompt change is tried and measured in minutes. RAG is set up in a few days, and when information changes you only update the documents. Fine-tuning means preparing data, training, measuring and serving the trained model, and that cycle repeats with every change. The cases where fine-tuning really stands out are: when a very consistent format has to hold across thousands of requests, when you want to teach a smaller, cheaper model the job a long prompt was doing, and when you need a style that prompting just won't produce.

How much data do you need?

Look at quality before quantity: a small number of consistent, correct examples does better than a large number of messy ones.

The pressure in a fine-tuning project is always to increase the volume of data. But the model learns the inconsistencies in the data too. Examples that answer two questions of the same kind in two different shapes teach the model to say "sometimes like this, sometimes like that". Making sure every example's expected output is exactly what you want does more than adding examples. Volume should only go up once quality is in place.

What happens if evaluation data leaks into training?

The model has seen the exam questions in advance, and its score becomes meaningless.

This is called leakage, and it's hard to notice because the results look very good. Its most common form is near-duplicate examples landing partly in training and partly in the test set: two versions of the same customer message written with small differences, for instance. The model appears to solve the test question but has actually recognised an example it memorised. The defence is removing near-duplicates before splitting the data, and keeping the test set completely separate from the training data. A good score means a good model only if that was done.

What is LoRA, and why does it need less memory?

A method that trains small extra matrices added alongside the model, not all of its weights.

Training all of a model's weights takes far more memory than the weights themselves, because during training the gradients and the optimiser's state are also held in memory for every weight. LoRA freezes the main weights and trains only small extra parameters. Because the number of trained parameters drops to a tiny fraction, the memory needed drops sharply too, which makes training on a single graphics card possible. The question "how much memory does training need?" is more about those gradients and optimiser state than about the model's size.

Why check the licence before using an open model?

Because the licence decides whether, and on what terms, you may use the model commercially.

Licences on open-weight models vary enormously: some allow any use, some restrict commercial use, some require separate permission above a certain number of users, and some require you to credit the model by name. For a product you intend to sell, the licence isn't a technical detail but a build-or-don't decision, and making it in the first week is far cheaper than making it once the product is finished.

How much memory does running a model on your own machine take?

The weights themselves, plus a cache that grows as the conversation gets longer.

The model's weights are loaded into memory once and their size is fixed. But while the model generates text, it keeps intermediate state for every token in the context, and that grows as the context grows. A model that runs fine in a demo with short questions can run out of memory with a long document or a long conversation. Accounting for that second line is what separates a working demo from a deployment. You can see the fine-tuning and open-model concepts in the free knowledge map.

Sources

• Hu et al. "LoRA: Low-Rank Adaptation of Large Language Models": the paper that introduced LoRA. (added 2026-09)
• Dettmers et al. "QLoRA": fine-tuning with even less memory. (added 2026-09)
• Hugging Face: PEFT documentation: the library of parameter-efficient fine-tuning methods. (added 2026-09)
• Hugging Face: Fine-tuning a pretrained model: the basic steps of training a ready-made model. (added 2026-09)
• Choose a License: plain-language summaries of common open licences. (added 2026-09)