Your agent will cheat on its tests
What I learned building Agentest, an open-source test runner for AI agents.
The test that passed for the wrong reason
One of my agent tests went green, and it shouldn't have. The scenario called a tool I had forgotten to mock. My test runner did the right thing and threw an error. The agent framework then did its right thing: it caught the error, handed it to the model as an ordinary tool result, and let the model carry on.
The model apologised, took a different route, and finished the conversation. Every assertion after that point checked a conversation that never really happened. The report showed a vague downstream symptom, not the actual cause.
That bug sums up what building Agentest taught me. An agent is built to recover from failure, and that includes failures in your test setup. You have to design your tests with that in mind.
Agent tests are closer to UI tests than API tests
An API test sends a request and checks the response. An agent breaks all three of the assumptions behind that:
- It's a conversation, not a call. What the agent does on turn four depends on what the user said on turn two.
- The agent picks the tools. Which tools it calls, in what order and with which arguments is the behaviour you want to test.
- "Correct" is fuzzy. Two equally good replies can share no words at all.
When I looked around, eval platforms were good at scoring outputs after the fact. Agent frameworks gave me the building blocks. Neither ran my agent through a realistic conversation in CI with one command. I wanted something that works like Playwright: set up a world, let the agent act in it, then check what it did.
Lesson 1: Your agent will route around your test harness
The opening bug came from treating a thrown error as a dependable signal. In a normal test, an error thrown from a mock goes up the stack and fails the test. In an agent test, it usually doesn't make it that far.
Agent frameworks catch tool errors on purpose. LangGraph's ToolNode does this by default, so the model can recover when a real tool fails. They can't tell "the weather API timed out" apart from "the test harness has no mock for this". Both reach the model as a tool result, and the model does what it's good at: it works around them.
The fix was to stop relying on the throw. The mock resolver now writes every unmocked call to a ledger before it throws. The simulator then checks that ledger after the agent finishes, whatever the agent reports:
// Record BEFORE throwing: the throw may be swallowed downstream.
this.unmocked.push({ toolName, turnIndex, message: error.message })
throw error
// ...after the conversation ends:
const unmocked = unmockedError(mockResolver)
if (unmocked) return { conversationId, turns, error: unmocked }
Three rules came out of this:
- A missing mock is the root cause. If anything else went wrong afterwards, report the missing mock rather than the symptom.
- Fail before you assert. Otherwise a "this tool must never be called" assertion passes trivially on a conversation that got cut short.
- Name every missing mock at once. Calling the same tool twice is one gap, not two. List every distinct tool so a single run tells you everything to fix.
The general point: every error your harness raises becomes input to the system under test. Design the harness so that it gets the last word.
Lesson 2: Be strict about mocks, with one explicit escape hatch
Once unmocked calls failed every time, a new problem showed up right away. Some calls aren't the agent's at all. The integration fires them itself: markers that tell a scenario something happened, which a scenario may assert on and usually ignores. Under strict mocking, every one of those markers became a mock that each scenario had to provide.
The tempting fix is a global "allow unmocked tools" switch. That would have brought the original bug straight back. Instead, the caller can now mark a single call as optional:
await ctx.resolveTool('progress_marker', args, { optional: true })
An optional call with no mock resolves to undefined, doesn't throw, and doesn't go in the ledger. Everything else stays strict. The rule I'd pass on: don't relax strictness globally. Add a named exception at the exact call site where the caller knows the call doesn't matter.
Lesson 3: You need two kinds of agent test
A simulated user driven by an LLM is good at finding failures you didn't think of. It phrases things oddly, changes its mind and gives information in the wrong order. That same variety makes it a poor regression test, because no two runs are the same conversation.
I ended up with two modes, each covering what the other can't:
| Persona-driven | Scripted multi-turn | |
|---|---|---|
| User messages | Generated from a profile, a goal and what the user knows | Written in advance |
| Good for | Finding failures you didn't expect | Catching regressions |
| Determinism | Low, and that's intended | High |
| Assertions | Goal completion, judge scores, tool trajectory | The tool trajectory at each turn |
A persona scenario looks like this:
scenario('user books a morning slot', {
profile: 'Busy professional who prefers mornings.',
goal: 'Book a haircut for next Tuesday morning.',
knowledge: [{ content: 'The salon is open Tuesday 08:00-18:00.' }],
mocks: { tools: { check_availability: () => ({ slots: ['09:00', '10:30'] }) } },
})
When a persona run finds a bug, turn it into a scripted test so it can't quietly come back.
Lesson 4: Assert on the shape of the trajectory, not the exact sequence
My first trajectory assertions demanded an exact list of tool calls. They broke whenever the agent checked availability twice or looked something up in a slightly different order. None of those were bugs.
The exact order is almost never the real requirement. So the assertions became match modes, from strictest to loosest:
| Mode | Passes when | Use it for |
|---|---|---|
strict | The calls match exactly, in order | Protocols where order is the contract |
unordered | The same set of calls, in any order | Independent lookups |
contains | The expected calls appear in order, with other calls allowed between them | Most real flows |
within | Every call comes from an allowed set | Keeping the agent inside a boundary |
Arguments get the same treatment: match the fields that matter and ignore the rest. Choosing the loosest mode that still checks something meaningful is a skill in itself. Too loose and the test proves nothing. Too strict and people learn to ignore it when it fails.
Lesson 5: The hard parts aren't the ones you'd expect
Ending the conversation is a real problem. A simulated user has to decide when its goal has been met. Left alone, it either loops forever or politely signs off too early. What worked was a structured shouldStop field in its response, a stop token as a fallback for when the model ignores the schema, and a hard cap on turns behind both.
Group failures, don't just count them. A failing run can mark 40 turns as bad when there are really only three root causes. One more LLM pass that groups failures by cause made reports readable. Grouping matters more than any single metric.
Rate limits are part of the product. Every scenario spends tokens on the simulated user, the agent and several judges. Provider rate limits made retries with backoff essential, plus a configurable pause between scenarios. Supporting local LLMs kept the inner development loop free.
Shipping an npm CLI is still painful. The ESM build, the shebang, module resolution and tsconfig discovery took about eight commits to get right. None of it was interesting, and all of it blocked users.
What's still unsolved
- Can you trust the judge? LLM-as-judge scores are useful signals, but they drift between model versions. Agentest passes or fails on thresholds rather than exact scores, but a person still has to read the failures.
- Cost grows with ambition. More personas, more turns and more metrics multiply the token bill. Deciding what runs on every commit and what runs nightly is still a judgement call.
- Flakiness can be contained, not removed. Persona runs will always vary. The aim is to make that variation useful, so it surfaces new failures rather than noise.
Try it
Agentest is open source under the MIT license. Install it with npm install @agentesting/agentest --save-dev and run it with npx agentest run. It works with any agent behind an OpenAI-compatible endpoint or a plain TypeScript function.
- Code: github.com/r-prem/agentest
- Docs: r-prem.github.io/agentest
If your own agent tests have ever passed for the wrong reason, I'd like to hear how you found out.