# Are AI-Generated Tests Reliable? Using Mutation Testing to Check Coding Agent Tests

> Canonical: https://vxi.me/en/posts/ai-generated-tests-mutation-testing/
> Locale: en-us
> Published: 2026-10-07
> Updated: 2026-10-07
> Markdown: https://vxi.me/en/posts/ai-generated-tests-mutation-testing.md

Getting Claude Code, Codex, or another coding agent to write unit tests is no longer the hard part. The harder question is what comes after: the tests are green, coverage looks healthy, and you still have to decide whether those tests would actually catch a wrong implementation.

That uncertainty is easy to hide behind a successful CI run. An agent adds a few cases, line and branch coverage go up, and the task looks done. But those signals mostly tell you that code was executed. They do not directly answer the more important question: **if the implementation changed in a plausible but incorrect way, would the test suite object?**

A newly released project called [mutagate](https://github.com/keyboardsamurai/mutagate) approaches that problem by putting mutation testing directly into the coding-agent completion loop. In the author's [Reddit post](https://www.reddit.com/r/ClaudeCode/comments/1wz3wol/i_built_a_hook_that_wont_let_claude_code_finish/), the idea is simple: run mutation testing after the agent writes tests, and if the score stays below a gate, feed the surviving mutants back instead of letting the agent finish immediately.

That caught my attention because “Coverage is 82%” is fairly weak feedback for an agent. “Your tests still pass when `age >= 18` is changed to `age > 18`” is much more useful. The second message tells the model exactly which boundary it failed to constrain.

I did not install mutagate for this experiment. Instead, I built a tiny self-authored Node fixture inside my Content Lab and tested the mechanism itself. The result was more revealing than I expected: **the baseline suite was green and reported 100% line, branch, and function coverage, yet it killed only 3 of 6 controlled mutants.**

## A green suite still does not tell you whether wrong behavior would be rejected

Consider a very ordinary function:

```js
export function isAdult(age) {
  return age >= 18;
}
```

Now imagine the test suite contains only:

```js
isAdult(20) === true
isAdult(10) === false
```

Those two cases can exercise both outcomes and produce 100% branch coverage.

But if I quietly mutate the implementation to:

```js
return age > 18;
```

both tests still pass.

This is where structural coverage can give you more confidence than the assertions deserve. Coverage answers “which code paths ran?” It does not guarantee that the test suite pinned down the behavior at the boundaries.

Mutation testing asks a different question. It deliberately creates small, controlled wrong implementations, for example:

```text
>=  ->  >
>   ->  >=
invert a condition
swap branch results
```

Then it runs the existing tests again.

If the tests fail, the mutant is killed: the suite can distinguish that behavior change. If the tests stay green, the mutant survives. That survivor is especially useful in an agent workflow because it is not a vague “please improve the tests” instruction. It describes a behavior the current suite failed to reject.

The broader idea predates mutagate. The earlier [MuTAP paper](https://arxiv.org/abs/2308.16557) already used surviving mutants as feedback for LLM-based test generation. What is interesting now is how naturally that loop fits into the lifecycle of coding agents.

Claude Code's current [Hooks reference](https://code.claude.com/docs/en/hooks) exposes lifecycle events such as `PostToolUse`, `SubagentStop`, `TaskCompleted`, and `Stop`, including asynchronous hooks. That makes a workflow like “agent writes tests -> deterministic check runs -> completion is allowed or blocked” an engineering mechanism rather than another line in a prompt asking the model to remember to self-review.

This is the same boundary I explored in [When CLAUDE.md Gets Too Big: Rethinking AGENTS.md, Skills, and CI](/en/posts/claude-md-agents-md-skills-ci/): once a rule must happen every time, it should gradually move out of “the model should remember” and into an executable check.

## I ran a small A/B fixture: coverage stayed at 100%, mutation score moved from 50% to 100%

To keep the experiment narrow, I did not execute any third-party repository code and did not install a mutation framework. The fixture used three tiny boundary-sensitive functions I wrote myself:

```js
isAdult(age)          // age >= 18
discount(total)       // total > 100
shipping(total)       // total >= 50 ? 0 : 5
```

I then defined six controlled mutants by hand. Some were obvious condition inversions; others were exactly the kind of boundary changes that structural coverage tends to miss.

The baseline suite had six tests. It intentionally covered both sides of each branch but did not test the exact boundary value.

GitHub Actions produced this result:

| Metric | Baseline |
| --- | ---: |
| Ordinary tests | PASS |
| Line coverage | 100% |
| Branch coverage | 100% |
| Function coverage | 100% |
| Mutants killed | 3 / 6 |
| Mutants survived | 3 / 6 |
| Mutation score | 50% |

All three survivors were boundary changes:

```text
age >= 18        -> age > 18
total > 100      -> total >= 100
total >= 50      -> total > 50
```

For me, that survivor list was more useful than the 50% score itself, because the next move was obvious.

I added only three assertions:

```js
isAdult(18) === true
discount(100) === false
shipping(50) === 0
```

Production code did not change, and the mutation matrix stayed exactly the same. On the next run, all 6 mutants were killed and the score went from 50% to 100%.

The interesting part is not the “100%.” That number can easily become another vanity metric. The useful part is the feedback loop:

```text
Agent writes tests
  -> ordinary tests pass
  -> inject controlled faults
  -> collect survivors
  -> turn survivors into concrete feedback
  -> Agent adds boundary / negative assertions
  -> verify again
```

Coverage stayed at 100% in both rounds. It provided no new signal after the first run. The survivor list identified the three missing behavioral constraints immediately.

After seeing that, I became less interested in mutation testing as “another score.” For coding agents, I think the more valuable framing is **a fault-feedback generator**.

## Feeding survivors back to the agent matters more than copying an 80% threshold

The current mutagate defaults are illustrative: in the pinned version I reviewed, the default mode is `gate`, the mutation threshold is 0.8, and `maxRounds` is 2. When the result is below the threshold, the tool can block completion and give the agent another chance to strengthen the tests. If the mutation infrastructure itself fails, the design favors fail-open behavior so a broken gate does not trap the agent forever.

I like that implementation direction, but I would not copy “80%” into every repository.

Mutation score is still a proxy.

A 2026 [replication study on LLM-generated tests](https://arxiv.org/abs/2607.22880) reaches a deliberately careful conclusion: how well coverage and mutation score track real-bug detection depends on the evaluation setting. They can be useful when comparing test-generation approaches against code assumed to be correct, but become much less straightforward when the code under test may already contain the target bug.

Another 2026 [study of class-level Python test generation](https://arxiv.org/abs/2609.24341) similarly finds that structural coverage can saturate and lose discriminating power. Its practical recommendation is closer to a portfolio of signals: treat executability as a gate, then combine coverage, mutation testing, and structural-quality measures rather than relying on a single number.

So if I were adding mutation testing to a coding-agent workflow, I would treat ordinary tests as the first line for basic execution and regression. Mutation testing would look for behaviors the current assertions cannot distinguish. Integration, E2E, property, security, and domain-specific tests would still cover layers mutation testing does not replace.

I would also derive thresholds from the repository's own baseline rather than copying a tool default. A legacy codebase may have equivalent mutants, unreachable code, or a very expensive suite. A small new library may reach high mutation scores easily. What matters more is which important paths keep producing survivors and whether those survivors lead to meaningful new assertions.

That is also why survivor feedback feels more useful to an agent than a score. A number says “you are below target.” A concrete mutant says “this behavior is not constrained by your current tests.”

## The gate has a real cost

Mutation testing repeatedly changes code and reruns tests, so it is not free.

I ran the ordinary suite and the full mutation loop five times each in GitHub Actions. This fixture is tiny, yet the medians were:

| Run | Median time |
| --- | ---: |
| Ordinary tests | 98.331 ms |
| Mutation loop | 711.798 ms |
| Added time | +613.467 ms |
| Ratio | 7.24x |

The 7.24x figure should not be generalized as “mutation testing is usually seven times slower.” It is only a measurement from this tiny synthetic Node fixture. What it does show is that fault injection has visible cost even when the project is trivial.

On a real repository, I would not begin by running a full mutation sweep on every agent stop. A more practical design is to narrow the target to code the agent just changed, use a fast local gate for the highest-value mutations, and leave the complete mutation suite for CI, scheduled runs, or higher-risk changes.

Equivalent and unreachable mutants are another source of friction. An equivalent mutant changes the code without changing its semantics; an unreachable mutant can never be exercised. If those survive, that does not automatically mean the tests are weak. A tool that cannot handle these cases well can push an agent into chasing an impossible score.

Infrastructure failure also deserves its own state. A mutation runner crashing, a missing test dependency, or a timeout is not evidence that the test suite is bad. Mutagate's fail-open choice at least avoids one ugly failure mode: adding a reliability gate that regularly locks the development loop itself.

There is also a security boundary. Mutagate's security documentation points out that the build and test commands it triggers run with the user's permissions and are not automatically contained inside the agent sandbox. I would not drop a newly discovered third-party hook into a long-lived development environment just because the idea is good. I would read the code, pin a version, and understand exactly which commands it can execute first.

## I would put Mutation Testing in the “completion evidence” layer

I now think of coding-agent test work as two separate jobs.

The first is generating tests. Modern coding agents are increasingly good at producing happy-path cases, mocks, and common boundary checks quickly.

The second is proving that the resulting suite deserves trust. That layer should not remain entirely self-assessed by the model.

A green suite can be the first gate. Coverage can show structural blind spots. Mutation survivors can then ask whether the assertions actually distinguish plausible wrong behavior. For critical paths, integration, E2E, property, or domain tests still matter.

Seen that way, mutation testing has a clear place in an agent workflow. It is not a new truth score and it does not replace coverage. It gives “I finished writing tests” a piece of evidence that is harder to fake with shallow assertions.

Mutagate is still a very new project, and this small experiment is not enough to call it a production standard. But the workflow idea holds up for me: **before an agent is allowed to declare the tests done, do not only ask whether they pass. Deliberately make the implementation a little wrong and see whether the tests object.**

That question has always been worth asking about human-written tests. As coding agents generate more of the test suite automatically, I expect it to become even more important.