Creator & maintainer
ReplayGate
Cross-turn regression testing for multi-turn AI agents
Records a real agent conversation, its LLM and tool calls, into a deterministic fixture, then replays it to catch the cross-turn bugs per-turn tests miss.
ReplayGate is a regression-testing harness for multi-turn AI agents that run on real channels like WhatsApp. It records a live conversation — every LLM call and tool call — into a deterministic fixture, then replays that fixture offline to assert cross-turn invariants: properties that span the whole conversation, such as "never book an appointment before the user has confirmed". These are the bugs per-turn unit tests miss, because they only emerge from the sequence of turns. It is built in Python with Pydantic v2 for typed fixtures, DuckDB for storage, a Typer CLI and a pytest integration — and it is being built in public.
Highlights
- Records a live agent conversation — LLM and tool calls — into a deterministic fixture
- Replays fixtures offline to assert cross-turn invariants (e.g. "never book before the user confirms")
- Catches whole-conversation bugs that per-turn unit tests miss
- Built in public — follow the build log for ongoing updates
Stack
- Python
- Pydantic v2
- DuckDB
- Typer
- pytest