A green scratch run is not a proof: 127 passing tests and the collision my AI plan called verified

supskill is a sprint conductor for Claude Code. It takes one sprint from a markdown backlog to a merged branch, drawing a fresh context boundary at every stage (SCOPE, REFINE, PLAN, EXECUTE, REVIEW), holding three human gates, and keeping every authoritative fact on disk instead of in the model’s head.

Update 2 packaged it so someone other than me can install it: marketplace slug supskill@supskill, a trigger-description corpus on a borrowed eval harness, a README that stopped one line short of claiming more than the offline suite proved. What it still had not done was the thing the project has been circling since I re-ran the finding it rests on: drive a real sprint against a real backlog, SCOPE through Gate 3, with the model doing the work.

E8 is that epic, and its backlog line is one sentence: validation that “earns its keep or does not ship.” This update is the run. It found real defects in the code the conductor was building and eight more in the conductor itself, and it is the run that finally closed E8, though it took three reports to get there and this was the third. What follows is what I measured.

The target was a throwaway app

The project is called playset: a local, single-user web app that turns a link to a video or a playlist into MP3 files. Localhost only, no auth, fixed MP3 output, no database, because the output directory is the state. It exists so E8 has something to drive.

supskill cannot produce the backlog it requires. I wrote playset’s by hand in conversation before the conductor touched anything. That front-of-chain gap sits in supskill’s own backlog, deferred with one line: “a separate product, revisit after v1.” It is still the honest start of every real run until that changes.

Sprint s1 committed epic E1: the tool seam and the output store. Three stories, 13 points at commit time. By the end of the run: 14 commits on branch s1 (off 4e8255e), 165 tests passing in 0.13s with ruff clean and mypy clean on five source files, and a Gate 3 generative writeback that inserted 27 lines, removed none, and added a new epic E1b (5 rows, 10 points).

Token spend, from .supskill/runs/s1/costs.jsonl:

Stage	Tokens
SCOPE	49,973
REFINE	82,818
PLAN	164,426
EXECUTE	1,469,251
REVIEW	202,669
Total	1,969,137

EXECUTE is 75% of the run. PLAN is about 8%. Before I saw the file I expected planning to be the expensive part. That instinct was wrong in the boring direction: the money sits where the code gets written.

The three gate decisions, from .supskill/gates.jsonl:

• G1 approved: set the output folder to ./output, not ./download
• G2 approved
• G3 replan: generative writeback

The G1 amendment travelled all the way into a test named test_constants_match_the_gate_1_amendment. An operator sentence at a human gate became a named assertion in the suite.

The plan that laundered a false proof

PLAN produced 1,867 lines. Before saving, the planning agent assembled every source and test block into a scratch project outside the repo and ran it: 127 passed, ruff clean, mypy clean. It even caught three real defects in its own text that way, including seven E402 import-order violations from blocks it had appended.

Its self-review then mapped acceptance criteria to test names, including this pairing: “two distinct items never collide,” an acceptance criterion on the output-store story, mapped to test_distinct_items_never_collide.

The shipped derive_path was not injective. Two collisions, both reproduced against the built package:

id='a/b' and id='a\b' with one title  ->  Same Title.a_b.mp3
title='a.b', id='c' and title='a', id='b.c'  ->  a.b.c.mp3

A green scratch run became a claim of proof for a property the suite never actually checked. The tests exercised the hostile cases someone thought of. They never searched for collisions. Pre-execution converted an assumption into the word “verified,” and every stage downstream read that word as evidence.

The cost objection is small. The correctness objection is the one that matters. And EXECUTE still caught it. The worry that a plan full of working code would turn EXECUTE into transcription did not hold: EXECUTE spent roughly nine times what PLAN did and found two blockers the green scratch run had missed.

I fixed the injectivity blocker with an encode_id that percent-encodes every byte outside [A-Za-z0-9_-]. Both collisions died. It also tripled the byte cost of non-ASCII ids. The overflow threshold moved from about 124 characters to 43:

 40 non-ASCII chars -> old  80B | encode_id 240B
 43 non-ASCII chars -> old  86B | encode_id 258B
124 non-ASCII chars -> old 248B | encode_id 744B

I noticed the expansion while writing the fix, decided it fell inside an already-deferred gap, and did not flag it. Parallel adversarial review (PAR) found it instead. The whole project is an argument about not letting an agent quietly decide something mid-task. I did exactly that to my own operator on a human-owned decision.

What 165 green tests still miss

PAR returned 14 findings (1 Critical/high, 3 Important/high, 1 Important/actionable, 5 Minor/high, 4 Minor/actionable), nine of them matched across both reviewers. The Critical is the cleanest illustration of what a green suite can hide.

derive_path spends the full 255-byte filename budget on the final name. Then temp_path_for adds six more bytes for a leading dot and a .part suffix. Any title long enough to be truncated produces a final path that writes and a temp path that does not:

title len 230 -> final 246B, temp 252B -> writes OK
title len 300 -> final 255B, temp 261B -> OSError: File name too long
title len 500 -> final 255B, temp 261B -> OSError: File name too long

"x"*500 is in the project’s own hostile-input table. It passes test_hostile_input_fits_the_byte_budget and can never be fetched. Hostile-input tests exercise derive_path alone. Every atomic_fetch test uses a short title. The two halves are never composed, and that is the hole the Critical lives in.

The same review planted posix_spawn, fork, spawnv, pty, and multiprocessing past the seam guard’s denylist. All returned NOT CAUGHT. The guard is playset’s version of running generated code inside a boundary you control, and this boundary leaks: os cannot be import-banned, because os.replace is what makes the atomic write atomic, so the call denylist is the only barrier there, and it was incomplete.

Scaffolding became the specification

Every prior data point I had for “refinement against live source grows scope” came from a codebase that already existed. Playset’s first run opened on three files: a .gitignore and two docs. Scope still grew, 13 to 21. The shape of the finding inverted. Normally REFINE catches a document that drifted from code that moved underneath it. Here there was no code, and the sprint document had assumed a project already existed to bolt three modules onto.

REFINE also derived the whole toolchain from the only on-disk evidence available: my .gitignore. Pytest from .pytest_cache/, ruff from .ruff_cache/, a local venv from .venv/, and the default output root from downloads/. That last one is the uncomfortable part. I typed downloads/ as boilerplate. The refinement pass correctly treated the only evidence as authoritative and turned it into a specification. Gate 1 exists because of that accident; the amendment to ./output is a human override of a machine reading my scaffolding as intent.

A second sprint, s2, ran the remediation epic Gate 3 wrote back. Before it started I predicted refinement would look weakest on the empty repo, where there is nothing live to read. The measured growth went the other way:

Run	Input	After REFINE	Growth
s1	empty repo, rows I wrote by hand	13 → 21	+62%
s2	real codebase, rows from PAR findings	9 → 10	+11%

The largest growth came from the repo with three files. The smallest came from the codebase with confirmed defects written into the rows. Provenance is the most plausible explanation: E1b arrived already carrying file, symptom, reproduction, and in one case a measured byte count, so refinement had almost nothing left to invent. E1 arrived from a brainstorming conversation about work nobody had looked at yet. One pair of runs is not a trend. Three things also changed at once between s1 and s2 (maturity, provenance, and a conductor fix), so provenance is plausible, not proven. If it holds up, the growth number is a proxy for how well-sourced the input was, and a row written back from review is cheaper to refine than a row written by a human imagining the work.

What I installed was not what I validated

Before playset, I bumped supskill to 0.2.0, pushed, tagged, and confirmed CI green on the exact SHA. An earlier validation run had already shown the marketplace plugin lagging the development binary, so I thought I had closed that loop.

I then told the operator the reload had picked up 0.2.0. It had not. I had verified ./scripts/supskill-state from inside the supskill checkout: the development copy. The installed copies were both still stale:

• ~/.claude/plugins/cache/supskill/supskill/0.1.0/
• ~/.claude/plugins/marketplaces/supskill/ on a July 14 commit

Both exposed ten subcommands. The four that 0.2.0 added (config, review, replan-guard, worktree) were missing. The mechanics that bit me are easy to state once you have paid for them:

• /reload-plugins reloads what is already on disk. It does not re-fetch from the remote.
• claude plugin marketplace update supskill updates the marketplace clone.
• claude plugin update supskill fails with Plugin "supskill" not found. The qualified name is required: claude plugin update supskill@supskill.

An installed-version check is a different question from a source-version check, and being the author made me worse at telling them apart. Update 2 already flagged that bare-profile cross-marketplace dependency fetching is still unverified; that gap is still open, and this run is not evidence it closed.

Append-only safety versus a document that agrees with itself

Gate 3’s generative writeback is append-only. Verified: 27 insertions, 0 removals. Prior rows and point totals untouched, which is the safety property the shape is supposed to guarantee.

The resulting backlog does not agree with itself. Story rows in the body: 22 rows, 73 points. Summary table: 5 epics, 63 points, and no E1b. Stated total still “Total: 63 pts.”

That is not cosmetic. The SCOPE prompt tells the agent its scope is the first epic in the backlog’s recommended build order whose stories are still unchecked, and the recommended build order is that summary table. E1b is not in it. E1’s rows still read unchecked on main because the work sat on an unmerged branch, so the next SCOPE would re-pick E1 and never see E1b.

Correcting “Total: 63 pts” to 73 requires deleting a line, which append-only forbids. The safety rule and document consistency cannot both hold, and nothing in the shape resolves the tension. Two invariants of the same design are fighting, and I do not yet know which one should lose.

Branch s1 did merge, at 3588e5f. The reconciliation happened by hand at merge time, filed as a story of its own: the branch copy of the backlog still marked two stories unchecked and “blocked” for the seam-guard alias gap and the derive_path injectivity gaps, both of which had been fixed on that same branch and confirmed by review. PAR flagged the contradicting status rows as Important. The S1 sprint document and dev plan were untracked too, while their committed siblings were tracked, so the plan the branch implemented was not in the branch.

Then it happened again. S2’s sprint document and dev plan came up untracked the same way, which playset’s backlog records as the exact recurrence of that same finding one sprint later, and it is now carried as a named merge ritual rather than a numbered story: whoever merges the branch commits both files as part of the merge. So the two invariants do get reconciled, by a person, every sprint, from memory.

The findings in the conductor

The report of record lists eight findings against supskill itself. The seven below are my own running list from the same day, and it is a different cut rather than the same eight renumbered. It adds the false-proof finding, the missing skill preflight, and the append-only tension. It leaves out three the report carries: review-final.diff going stale between EXECUTE and REVIEW, supskill-audit --proofs passing vacuously, and the SCOPE template hard-coding supskill’s own capacity model.

1. Prompt templates hardcode the SK-0xx prefix while the parsers honor the configured one. The REFINE agent noticed and overrode it, which is the only reason the run survived.
2. writing-plans emits pre-executed implementations and presents a green scratch run as proof (the false-proof finding above).
3. No runtime preflight that the four dispatched skills actually resolve. A missing dependency makes the stage improvise, which is the failure this project exists to prevent. Playset’s own backlog carries a story for exactly that: “refuse to start when a required external tool is missing.”
4. A subagent dispatched to apply review fixes ran git add -A and swept unrelated harness state into a commit. The controller caught it and reset to the three intended files.
5. One plan instruction was unconditionally wrong (tick all three backlog stories when two were blocked). The controller overrode it.
6. There is no inverse of block. After both blockers resolved, show --json still reported open blockers, and Gate 3 re-presented settled decisions as live.
7. Append-only writeback versus document consistency (above).

Findings 1, 3, and 6 are what a new user would hit first. 2 and 7 are the interesting design ones.

The report’s own first finding is one I have hit before. Subagents returned degenerate final messages again, placeholders instead of their actual report, while having genuinely done the work. Both PAR reviewers hit it; the conductor recovered findings from transcripts. This is the third independent sighting (after earlier fixture and blinkebot reports), which is enough to treat it as a property of the environment rather than a fluke. Write the deliverable to a file and confirm it with a read before replying: still the right workaround.

What I learned

A green pre-execution is evidence that the cases you encoded ran, not that the property those cases stand for holds. Mapping an acceptance criterion to a test name is the moment that gap turns into a claim, and every later stage trusts the claim harder than the suite earned it.

Both defects here are the same mistake at different scales. “Two distinct items never collide” was checked against the collisions someone imagined. “Hostile input fits the byte budget” was checked against derive_path in isolation, never against the temp_path_for call that consumes the same budget six bytes later. A property asserted by name and sampled by hand looks exactly like a property that was tested.

What’s next

E8 is closed, on three reports rather than one. A throwaway fixture repo cleared the first of E8’s two stories on 2026-07-19. The second took two attempts: the blinkebot sprint proved every mechanism against a project I actually depend on, but it never reached Gate 3, never merged, and its feature shipped through a non-supskill path, which the backlog records as “a pass with a caveat, not a pass.” This run answered that caveat. A greenfield project, three tracked files and no code, driven through all five stages and all three gates to merged code on main.

What makes it the report I would defend is not that the run went well. It is that PAR caught a Critical in code that had already passed 165 tests, two rounds of task review, and nine tasks of execution, and the process then routed that defect into the backlog as scheduled work instead of losing it.

Playset still has not fetched a single track. E1 is pure Python, there was no operator run, and no MP3 has ever been produced.

The holes a new user hits first went into the backlog as E9, the epic that turns E8’s own principle on supskill itself, and two have closed since: a run now refuses to start when a dispatched skill will not resolve, and block finally has an inverse, so Gate 3 stops re-presenting settled decisions as live ones. Still ahead are the front of the chain, the brainstorm-to-backlog step that would take the hand-written backlog out of every run, and an installed-version preflight. The conductor drives a sprint end to end already. What it cannot do yet is keep the evidence it produces in step with the words it uses about that evidence.

So the question I would actually like answered. When your suite claims a property under a named test, what makes you believe it searched for counterexamples rather than checking the inputs someone happened to think of? Property-based testing is the textbook answer and I have not wired it in yet, which is why I am asking rather than recommending. The whole run, defects included, is in the open at the supskill log.