Run it a hundred times. Report what held.
Deterministic callers replay a scenario across seeds; pass^k reports whether it held on every run.
Each run lands in one of three buckets: pass, if the scenario rendered and the agent held; fail, if it rendered and the agent broke; or SIMULATOR_INVALID, if the fixture itself never faithfully rendered, and is excluded from the count instead of scored either way. Per-run seeds derive from a hash of the scenario and its variation tuple, never wall-clock, so a fixed scenario and seed reproduce the same conversation bytes: a red run points at a real change in behavior. Every produced conversation is stamped origin=simulated, so a synthetic run never lands in a bucket of recorded calls.
$ hotato simulate scenario.yaml --out runs/Holds only if every run passed: one flake sends reschedule to 0.00. The hatched cell is SIMULATOR_INVALID, excluded from n.
What ships today, and what’s gated.
Deterministic simulation runs now, with no credentials required. Driving a live call needs your own credentials and an explicit opt-in. Both paths score against the same seventeen deterministic assertion kinds, so a scenario written once works either way.
Deterministic simulate
Scripted and state-machine callers render a scenario with no model, no TTS, no network: offline, seeded, and byte-reproducible from one machine to the next.
- hotato simulate
Drive a live call
Place a live billed call through Vapi or Twilio, gated behind your own credentials and an explicit opt-in flag. The same per-dimension scorecard covers whichever call you point it at, live or replayed.
- gated · billed
Ship the run that held every time.
Seeded, byte-reproducible replays with a Wilson 95% interval, in one command.
$ uvx hotato start --demo
One recording is enough to start. Get started → · workspace →