What we learned building a small-model agent system
Why we tried to shrink the agent
A 400M-parameter model, fine-tuned on about 1,100 real messages, came within 1.6 points of an 80B general-purpose LLM on a production routing decision. With a confidence gate that hands the unsure cases back to the LLM, it matched the LLM. Getting there took about 30 training rounds, and most of what we learned along the way had nothing to do with model architecture.
The setting was a production chat assistant. Every user message went through a full LLM call whose first job was to decide where the message should go. Next to that call sat a set of hand-written regexes for smaller yes/no judgments, and each one grew by a clause after every incident. We wanted to know if small, dedicated models could take over those decisions.
This post covers what held up, what didn't, and the numbers behind both.
Lesson 1: Take the LLM out of the decisions, keep it for the words
If a step in an agent has a fixed set of possible answers, it doesn't need a generative model. Routing a message, spotting a retry, telling a follow-up from a new question: each of these is a classification with known labels. A small classifier handles that kind of step faster and cheaper, and it gives the same answer every time for the same input.
LLMs are still the best tool for producing text: the answer itself, a summary, picking a free-text subject out of a sentence. So the setup we aimed for keeps them there. Typed decisions go to small models. Open-ended text stays with the LLM. When a small model isn't confident enough, the LLM is the fallback.
What this buys you:
- Throughput and latency. A classification is one pass through an encoder, with no token-by-token generation.
- Cost. It runs on CPU inside your own backend, so there's no per-call API bill and no rate limit.
- Determinism. The same input always gives the same label and the same confidence. That makes testing, regression checks and debugging far easier than with a sampled LLM output.
- Fewer regexes. A trained judgment holds up when people rephrase. A regex has to be patched again.
The catch: you only save the call if you replace all of it
We nearly missed this one. The LLM call we wanted to remove didn't return one decision. It returned eight fields: the route plus seven others, some of them free-text extraction. A perfect route classifier still leaves the call running for every message that needs the other seven. When we checked, only about 4% of traffic could skip the LLM call outright on the route alone.
Before you train anything, list every field the call returns and every place in the code that reads them. Then either plan to cover all of them, or split the prompt so the deterministic part can stand on its own. Measure the payoff as LLM calls avoided per 1,000 messages, not as model accuracy.
That audit paid off in a second way. The call we studied sorted messages into six labels, but the agent only ever acted on two of them, and everything else went down the same path. Training a model on distinctions nothing reads wastes capacity and creates errors that don't matter. Our six-way problem became a three-way problem.
Lesson 2: Small models do well, but size buys you the rare cases
A 17M-parameter encoder learned simple, well-separated judgments in one or two training rounds. On a real routing task with lopsided traffic, it hit a ceiling. About 88% of messages took one route, 10% the second and 2% the third. The 17M plateaued at 82 to 86% overall accuracy, whatever we changed in the data.

From 17M to 400M, overall accuracy rose 6 points. Accuracy on the rare routes rose 23 points. A model can score 86% by getting the common route right and still miss the rare routes almost half the time. Judge model size on the rare classes, never on overall accuracy.
Size also bought calibration. At 17M, 100 of the model's 150 errors came with a confidence of 0.99 or higher. At 400M it was 41 of 83. That matters more than the accuracy figure, because the confidence gate in the next section only works if a high confidence actually means something.
What size costs
Size costs latency, and the cost is fixed. We measured single requests on a pinned 2-core CPU:
| Model | Median latency | p95 latency |
|---|---|---|
| 17M | 48 ms | 73 ms |
| 68M | 375 ms | 567 ms |
The 400M needs a GPU. Some things that usually speed up inference didn't help here:
- Prefix caching is impossible. The task instruction was about 70% of every input. In a bidirectional encoder, the instruction's encoding depends on the message after it, so it gets re-encoded on every call. A shorter instruction cut median latency by about 40% but cost 2.2 points.
- int8 quantization flipped 4.6% of the decisions, because of outlier activations. Judge quantization by how many decisions change, not just by how far the probabilities move.
- ONNX was exact but no faster than PyTorch on CPU, and batching was slower per message.
Distillation helped. A 68M student trained on the 400M's soft labels kept more of the rare-route recall than a 68M trained on hard labels. The 17M couldn't learn the rare routes even with a teacher: its recall on them stayed below 50%.
Lesson 3: A fine-tuned small model can match a general one, with a fallback
On its own, our best fine-tuned model (400M parameters, about 200 times smaller than the LLM) scored 92.4% against the LLM's 94.0%. Fine-tuning closed most of the gap but not all of it. What closed the rest was letting the small model decide only when it was sure:

At a confidence threshold of 0.99, the 400M answered 90.6% of messages itself. The LLM decided the remaining 9.4%. Together they reached 94.4%, level with the LLM alone, with nine LLM routing calls out of ten removed.
Two caveats keep this honest:
- The bar moved when we got stricter. We later rebuilt the evaluation with independent labels, a sealed test set and three training seeds. Under those rules the best CPU-sized model (a 68M distilled from the 400M) reached 90.8% gated against the LLM's 93.8%. That isn't good enough to ship yet. The cause was too little data on the rare routes, not too small a model.
- The LLM's own labels were wrong in a systematic way. Of 199 messages the production LLM sent to the small-talk route, an independent jury put 65 in the data route. That was a known production bug. If we had trained on the LLM's labels, the small model would have learned the bug. We took our gold labels from a blind jury instead: two different open models, each asked twice with the answer options in both orders.
So does training beat a general model? On cost, latency and determinism, clearly. On quality, a fine-tuned model in the hundreds of millions of parameters gets within a few points. A confidence gate with an LLM fallback turns that into a tie.
Lesson 4: Your own eval set will flatter you
On our own held-out data, most of the small yes/no judgments scored above 0.9, and many scored a clean 1.00. Then we checked them against a golden set the product team had written independently. The model agreed with it on 6 of 12 cases: 50%. After four targeted fixes it reached 19 of 22 (86%). The 1.00s had only shown that we hadn't tested against anything independent.
The same thing happened with template-generated training data. A model trained mostly on templates scored 11 of 13 on test cases that looked like the templates, and 49% on real traffic.
What we do now:
- Keep several independent references and report all of them: hand-corrected real traffic, the product team's golden cases, and their hand-written test scenarios. Never train on any of them.
- Seal a test set and limit the looks. We cut it by time, so it holds only traffic newer than anything in training. The code allows two looks at it, and every look is logged.
- Split folds by conversation, not by message. Otherwise a follow-up lands in eval while its first message sits in training.
- Know what your baseline is scored against. At first, 70% of our eval labels were the LLM's own answers, never hand-checked. The LLM was right on all of those by definition. Its "94%" was 100% on that part and 80% on the 324 labels a person had checked.
Lesson 5: Real data beats synthetic data, even verified synthetic data
The biggest single jump came from 344 real, hand-labelled messages: accuracy on real traffic went from 49% to 82%. No synthetic data came close.
We gave synthetic data a fair chance. An LLM wrote examples for the rare routes, and each one had to pass a jury label check, a check against a teacher model, a diversity check and a nearest-neighbour ambiguity check. The result was worse. To reach the same recall on the rare routes, the model now gave up 5 points of recall on the common route (82.7% down to 77.7%). The generated examples had correct labels but sat right on the boundary the model already struggled with, and they blurred it further. Our cheap ambiguity check had warned us twice in pilot runs. We should have believed it the first time.
Adding more of the common class doesn't help either. One round added 142 extra common-route examples, and accuracy on small-talk follow-ups fell from 53% to 16%.
So count your rare classes before you plan anything. Our rarest route arrived at about 1.25 labelled examples a day. After the full 90 days of history we had 45 of them to test on. That's too few to measure recall precisely enough to make a release decision. No modelling trick fixes that. Only time or new data sources do.
Lesson 6: Small statistical traps add up
Each of these moved a number we were about to act on:
- Gated accuracy can pass a bad model. A 17M scored 93.3% gated while getting the rare routes right less than half the time. It sent the hard cases to the LLM fallback or got them wrong, and the easy majority carried the score. Report per-class recall with a lower confidence bound, not just the headline.
- One seed lies. Our best config passed every bar on one training seed (92.96%). On three seeds it scored 90.5 to 91.0%. Picking the best of several configs on the same data is a winner's curse.
- Stacking seeds fakes certainty. Pooling three seeds triples the row count without adding a single new example. A rare-class lower bound of 0.80 was really 0.74 once we counted unique examples.
- Benchmark the real request path. An early claim of about 35 ms for the 68M was a batch average. Single requests took 375 ms at the median. Efficiency cores were 35 to 40% slower than performance cores, so pin the CPU you benchmark on.
- Predict on the hardware you ship. GPU predictions in bfloat16 moved probabilities by up to 0.03 compared with CPU float32. That's enough to shift a threshold.
Lesson 7: Models learn shortcuts, so fix causes, not symptoms
Several persistent errors were shortcuts the model had found in our data:
- Every group name in the training data was written in capitals, so the model learned "capital letters mean a group" and misread capitalised item names.
- No negative example in one language mentioned a group, so "this language plus a group name" became a shortcut for yes.
- Every group name looked like a proper noun, so a real group called by an ordinary lowercase word was missed.
Each fix was a small, targeted counter-example, found by reading the exact failing inputs. One regression ran for four rounds because we kept adding more of the same kind of example. The fix that worked was deleting the example that collided with the other class. We also learned not to chase single-round wobbles on small test sets: we only chased a pattern after it repeated for three rounds on the same case.
One more trap: don't write training data from the rules of the prompt you're replacing. We tried it once. Our own test scenarios went to 59 of 59, because they were written from the same rules. Real traffic got worse, and mistakes on follow-up messages doubled.
The whole journey in one chart
Two things moved the rare routes: a bigger model and more real data. Most of the clever data work moved them sideways, and two changes made them clearly worse.

With about 1,100 test messages, each overall figure carries roughly ±2 points of noise, and the rare-route figures carry more. Several rounds we first called wins or losses were noise. Aim for about 1,000 eval rows before you iterate on small changes, and read differences of a point or two as a tie.
If you try this yourself
For a well-defined decision, a small fine-tuned model is a strong option. It won't take over on day one, and almost every hard problem we hit was about data and measurement, not models. The short version:
- List every field the LLM call returns and every place that reads them. You only save the call if you replace all of it.
- Count your rare classes in real traffic and work out how many days it takes to collect enough of them.
- Take gold labels from an independent, blind jury, not from the model you're replacing.
- Set the pass bar before you train, and seal a time-split test set.
- Start at a few hundred million parameters if your classes are imbalanced, then distil down.
- Judge per-class recall with lower bounds, confirm on three seeds, and benchmark single requests on the CPU you'll ship on.
- Prefer real data to synthetic data. If you generate data anyway, gate it early and believe the gates.
- Ship behind a confidence gate with an LLM fallback, in shadow mode first.
Leave the LLM to write the answer. Make the decisions with a model you can test.