The tests that passed against code I deleted: four shapes of a test that cannot fail

How many of your tests can actually fail?

Not how many pass, and not what percentage of your lines they touch. How many would go red if you deleted the code they exist to guard.

I put that question to one of my projects in August. The first answer was uncomfortable: I removed the one element on its dashboard that says, in words, where the month’s money went, ran the suite, and all 867 of its tests passed.

The app is a cash-flow kernel, double-entry bookkeeping for a small company’s books, with Django on the server and a couple of React islands for the charts. I have written about the typed error codes its kernel raises before. It has a coverage gate, a pre-commit gate that runs the whole suite, and a habit of red-first development. All of that was green while that sentence could be deleted.

Mutation testing, in one paragraph

You change the production code on purpose, in a small way a real bug might plausibly take, and run the suite. If a test fails, the mutant is killed, and the guard is real. If everything stays green, the mutant survived, and you have found code your tests execute without checking. Stryker’s documentation puts the useful half plainly: surviving mutants point at lines where tests run the code but do not check its behaviour. Tools exist for this, mutmut on the Python side and Stryker on the JavaScript side, and both compute a mutation score for you. You need neither to start. Every mutation in this post was applied by hand with an editor and undone with git checkout.

Coverage cannot answer the question, and there is a measurement for that. Inozemtseva and Holmes ran five large Java projects at ICSE 2014 and found the correlation between coverage and test-suite effectiveness is low to moderate once suite size is controlled for, and that stronger coverage criteria buy no extra insight (paper). Coverage records that a line ran. A test that renders a page and asserts almost nothing about it covers every line that render touched.

The reason to trust mutants as a stand-in for bugs is also measured. Just, Jalali, Inozemtseva, Ernst, Holmes and Fraser tested 357 real faults across five open-source applications at FSE 2014 and found mutant detection correlates with real fault detection, independently of coverage (paper). A surviving mutant is not a theoretical worry. It is the cheapest honest signal that a real bug could live in that line unnoticed.

The four shapes

The survivors were not random. They fell into four shapes I can now recognise on sight, and each has a fix that generalises.

1. A lone negative with no positive twin

The dashboard prints one plain-language sentence above the charts: this period, this much went out, mostly here. The template hangs it on a data-lead attribute.

That string occurred exactly twice in the whole repository. Once in the template, and once in a test asserting it was absent:

assert "data-lead" not in body

That test is correct and it is worth keeping. It pins the empty state: a period with no expenses must not render “this period R$ 0,00 went out”, which is the sentence a toy app writes. But it was the only test that knew the attribute existed, and a negative assertion is satisfied by deleting the thing it names. Remove the sentence, rename its hook, typo the attribute: all green.

The fix is a positive twin, in the same file, next to its negative:

def test_the_lead_sentence_renders_on_a_period_that_has_expenses(full_chart):
    body = full_chart.get("/visao-geral").content.decode()

    assert "data-lead" in body
    # And it is the computed sentence, not an empty element: the figure it names
    # comes from the same rows the breakdown chart draws.
    assert "Neste período saíram" in body

The rule I took from it: an assertion that something is absent is only meaningful beside an assertion that it can be present. Grep your suite for not in, queryByText(...).not, assertNotContains. Every one that has no partner marks a feature whose presence nothing asserts.

2. A denylist that does not cover its own space

The islands are forbidden from hardcoding a colour, because a literal colour cannot follow the theme, and this app’s theme toggle repaints the page without a reload. A test enforced that by scanning each island’s source for banned prefixes:

for banned in ("oklch(", "rgb(", "hsl(", "#0", "#1", "#f", "#F"):
    assert banned not in source

Read that list against the space it claims to cover. It misses #333, #abc and every three-digit hex that does not start with 0, 1 or f. It misses every named colour, including white, which is precisely what a chart library reaches for when nobody tells it the theme. And it misses rgba( and hsla(, because rgb( is not a substring of rgba(: the a intervenes.

So a test whose entire job was to be a boundary was checking a fraction of the space it named. It now uses a regex over the whole of it:

COLOUR_LITERAL = re.compile(
    r"#[0-9a-fA-F]{3,8}\b"
    r"|(?:rgba?|hsla?|hwb|lab|lch|oklab|oklch|color)\s*\("
    r"|(?<![\w-])(?:white|black|red|green|blue|yellow|orange|purple|gray|grey)(?![\w-])"
)

The mutation that proves a denylist is the same every time: write into your source the exact thing the list is supposed to forbid, in a form the list does not spell. If the suite stays green, the ban is not enforcing anything.

3. A test satisfied by the library’s own fallback

The evolution chart marks its final point with a dot. Not a dot on every month, which is redundant ink, and the code says so at length. The test asserted that the chart rendered a circle.

It passed against the broken version. Recharts does not leave a one-point line unmarked when you pass dot={false}: it falls back to a dot of its own, filled plain white at radius 3, with the fill and stroke roles swapped from what the design asks for. White is a colour chosen inside a library for a screen whose theme it cannot see, and against the dark theme it reads as a hole punched in the line. “A circle exists” is true of both the authored marker and the library’s fallback, so the test could not tell them apart.

The fix is to assert the properties that distinguish yours from theirs: fill, stroke and radius.

A sibling of this shape turned up in the same pass and is worth naming because it is invisible on review: a test asserted color: "ink" on a rendered element. ink is not a valid CSS colour, so the parser discards the declaration, so the assertion compared an empty string to an empty string. It could never fail in either direction. Both of these tests had one thing in common, and I would rather say it than bury it: I wrote them after the fix, to lock it in, which is exactly the moment a test is written against code that already works and never gets to see red.

4. A wire between two files that each have green tests

This one is the reason I now distrust suites that look thorough.

The chart palette follows the theme through a small module: read the CSS custom properties, subscribe to theme changes, re-read on change. It has its own test file, nine tests, all green. The chart components have their own test file, thirteen tests at the time, all green, and they take the palette as a prop.

Between them sits one line, the island that actually wires the observer to the component’s state:

function ChartIsland({ Chart, rows, ticks, empty }) {
  const [palette, setPalette] = useState(readPalette);
  useEffect(() => observeTheme(() => setPalette(readPalette())), []);
  return <Chart rows={rows} ticks={ticks} palette={palette} empty={empty} />;
}

Delete that useEffect and the entire reason the theme module exists is gone: toggle the theme and the charts keep the old colours until a full reload. Both test files stay green. One tests the observer against its own stub, the other passes a palette in as a prop and never mounts the island. Neither is wrong. Neither file owned the wire between them.

I want to be precise about a defence I made at the time, because it did not hold. A browser pass had proved this end to end: page toggled dark, colours followed. That is a measurement, and a measurement is not a regression test. It tells you the code worked once, on one afternoon, on one machine, and it does not go red when someone deletes the line next March. The same distinction decides how much weight an LLM judge can carry in an eval harness: whatever advises you is not what gates you.

The pin has to mount the real island and toggle the real control:

act(() => mount());
expect(painted()).toBe("light-green");

// The theme control is CSS-only: it never changes data-theme, so this change
// event is the ONLY signal the island gets.
current = "dark-green";
await act(async () => {
  document.querySelector("input.theme-controller").click();
});

expect(painted()).toBe("dark-green");

What the mutations say today

Writing this post, I re-ran the experiment rather than quoting the August notes, on two checkouts of the same repository: the tree as it stood immediately before those fixes, and the tree as it stands now. Same mutations, same commands, both suites full. One caveat on the older column, since it flatters the old suite: the endpoint-marker test had already been strengthened one commit earlier, so that row is killed on both sides and only the theme wire reproduces as a survivor.

Three of the five survived on the old tree. Four of the five are killed today. The row that did not move is the control, and it gets its own section below.

Both baselines are green, and that matters more than it sounds. The verdict for a cell is whether the mutation makes tests fail that were not already failing, and my first attempt at this got it wrong: I ran the older checkout without building its stylesheet and island bundle, which several tests assert the existence of, so the suite was red before I touched anything and pytest’s exit code called every mutant killed. Building the artifacts that the project’s own gate builds took that baseline to 867 green, and both Python mutants then survived, exactly as the August notes said they had. If you take one operational detail from this post, take that one: a mutation run against a suite that is already red measures nothing.

The one that matters most in that grid is the theme wire: identical mutation, identical command, survived on the old tree and killed on the new one. The difference is one test.

The survivor I did not plan for

I included a control mutation, expecting it to be killed in both trees. The island loop skips a chart when the template asks for a series the server did not send:

const rows = data[node.dataset.chart];
// An unknown name or a series the server did not send is skipped rather than
// thrown on: the template and the payload are two files, and a screen that
// half-renders beats a screen that does not render.
if (!Chart || !rows) continue;

I cut that to if (!Chart) continue;. It survived on both trees, including today’s.

It is not an equivalent mutant, which was my first hope. The chart components open with rows.length === 0 and give rows no default, so a missing series does not skip: it throws, and it takes the rest of the mount with it. The comment above the guard describes a real decision, and nothing in 42 island tests holds the code to it.

That is the honest state of this practice. Two months after an audit that hardened five guards, three hand mutations and under a minute of machine time found a sixth. It is filed and unfixed as I write this.

Where mutation testing lies to you

A surviving mutant is a question, not a verdict. Some mutants change the code without changing its behaviour at all, and no test can kill them because there is nothing to detect. That is the equivalent mutant problem, and it is the reason a mutation score is not a target to maximise. Yao, Harman and Jia hand-analysed 1,230 mutants across 18 programs at ICSE 2014 and found equivalence distributed very unevenly across operators (paper), which matches the shape of the problem: whether a mutant is equivalent is a property of your code, not of the tool.

The August audit produced exactly one. It computed a value from a folded list rather than an unfolded one, and the two cannot differ, because the function reads only the first row and folding never moves the first five. The right response was to leave the test alone and correct the comment that explained it, which had the reasoning wrong.

Cost is the other honest limit. Every mutation in this post cost a full suite run, and the Python suite takes about two minutes. That is fine for the six full Python runs behind that grid and hopeless as something to run on every commit across a whole codebase. This is why Google’s implementation, described by Petrović and Ivanković at ICSE-SEIP 2018 (paper), is diff-based: it mutates only lines the change touched, skips lines with no coverage at all, and surfaces at most one mutant per covered line to the reviewer. Full mutation analysis of a real codebase is computationally prohibitive; mutating only a diff fits inside code review.

What to do with an afternoon

Pick the three guarantees your project would be embarrassed to break. Not three files. Three promises: money always balances, a tenant cannot read another tenant’s rows, the audit log is append-only.

For each one, find the code that keeps the promise, delete it, and run the suite. Restore it with git checkout and confirm git status --porcelain is empty before you do anything else.

Green is the interesting result. Green means nothing is holding the code to that promise, and you now know which one, for the cost of a coffee. I had done this once before, by hand, against a single benchmark I had published a chart from, and it printed the same flat line with the eager loading deleted. I took the lesson to be about benchmarks. It was about assertions, and it took a suite-wide pass to see that.

The question I would put to anyone with a suite they trust: when did you last watch your most important test fail? Not fail while you wrote it, red-first, on the day. Fail now, against today’s code, with the guarantee removed. If the answer is that you have not, the afternoon above is available, and I would genuinely like to hear what survived.