Cross-turn regression testing for multi-turn, channel-native AI agents
A harness that records a real agent conversation — its LLM and tool calls — into a deterministic fixture, then replays it to assert cross-turn invariants like "never books before the user confirms": the whole-conversation bugs that per-turn tests miss, on the channel the agent actually runs on.
Update 5 of the ReplayGate build log: adding an LLM-as-judge for the dimensions a deterministic predicate can't score (goal completion, relevance, tone), and then spending most of the sprint making sure it can never fail a build on its own. The judge is a real Claude call via forced tool_use, but its verdicts are recorded and replayed offline like every other call, keyed by the conversation and dimensions. `--judge` is advisory (prints scores, never touches the exit code); `--judge-gate` is a separate opt-in with its own exit code 4. Writing the post caught a bug in it: with no recorded verdict the gate found nothing below threshold and passed the build without running, so it now refuses with exit 2 rather than failing open. Plus the golden test that pinned the deterministic gate byte-for-byte before any of this was written, and why I flipped my own roadmap to build the judge before the WhatsApp channel.
Update 4 of the ReplayGate build log: the regress gate from #3 grows the headline feature, replaying a fixture recorded from one agent against a different candidate. The catch is that a key-miss isn't a pass or a fail, it's a third outcome (the candidate left the recorded trajectory), resolved by a pinned/live policy and a distinct exit code 3. Three support-agent candidates demonstrate OK, DIVERGED, and FAILED against one recording. Plus the fixture-mutation bug a live fallback introduced, and finally wiring the OpenTelemetry spans I'd deferred since #1 through the capture loop.
Update 3 of the ReplayGate build log: the record-and-replay from #1 and #2 becomes an actual regression test. A pure-function invariant suite judges a whole replayed conversation (booked-before-confirm, re-asked order id, forgotten dietary constraint), a replaygate regress command turns a violation into an exit code, and a GitHub Actions gate fails the build on a cross-turn bug. Plus the punctuation bug hiding in the flagship invariant's own happy path, and a review finding that made the CI gate honest.
Update 2 of the ReplayGate build log: the record half from #1 now replays. A recorded conversation re-runs offline with zero network (no provider SDK imported, no API key read), and the same capture seam records against five providers (Anthropic, OpenAI, OpenRouter, Ollama, Gemini) through one provider-agnostic client. With an honest note on the cross-turn detector that's still ahead.
Update 1 of a new build log. ReplayGate does conversation-level regression testing for multi-turn, channel-native AI agents — it records an agent's LLM and tool calls into a deterministic fixture, then replays them offline to catch the cross-turn bugs (like 'booked before the user confirmed') that per-turn tests miss. This first slice is the record half: the trace contract, a DuckDB span store, the record/replay wrappers, and a CLI — 20 tests, no network.