The evidence layer for voice agents
Your suite went green.
Prove what it proves.
hotato composes the checks you ran into one portable proof, headlined by the claim its evidence supports — and it refuses to headline a claim the evidence does not reach. Offline, byte-reproducible, MIT.
Any recording: mono, dual-channel, or a transcript.
- Two runs, one sha256
- Four claim scopes, enforced
- Nothing leaves your machine
- Free & MIT, any scale
The problem
A transcript records the words. It does not record what happened.
Two artefacts decide a voice call: the audio, which carries who held the floor and for how long, and your trace, which carries what the agent actually did. A transcript has neither. A hosted judge reads the words and returns a score you cannot re-derive.
A model judge is not reproducible.
Ask the same judge the same question twice and you may get two answers. Nothing in that pipeline lets you show a third party that today's verdict and last month's were produced by the same rule.
A green build with no scope is a screenshot.
“247 tests passed” does not say whether the run tested a suite, a candidate build, or the revision serving traffic. Those are four different claims and only one of them answers “is the fix live?”
A service cannot read your system of record.
The agent said the refund was sent. Deciding whether it was means reading your tool trace and your order row. That evidence lives inside your boundary, so the check has to run there too.
The loop
One call, one report, one pin.
Find it, share it, prevent it.
# install pip install hotato # find what broke hotato check ./call.wav FAIL #2 barge-in t=2.99s talk-over 1.81s # stop it coming back hotato pin apx-cc33f46fad58 exit 0 1 incident pinned
What a green build proves
Four claim scopes. The proof headlines the one its evidence reaches.
hotato prove composes the lanes you already ran — contracts, suite,
before/after, gauntlet — into one proof, and prints the rung it is standing on.
Adding a flag does not promote it. Only adding evidence does.
| Rung | Reads | What it takes to reach it |
|---|---|---|
| 01 | Captured Evidence | A contracts lane alone. It re-measures stored evidence; it does not exercise the current agent. |
| 02 | Test Suite | A suite or gauntlet lane executed. Tests ran and passed. |
| 03 | Candidate Revision | A before/after lane plus a bound candidate identity: --candidate-config-hash and --provider. |
| 04 | Deployed Revision | All of rung 03, and a --deployment-id on top of the bound candidate. Never the deployment id on its own. |
A gauntlet run with a deployment id attached. The id is recorded in the proof — it changes the content address — and the headline does not move.
hotato prove --gauntlet --deployment-id deploy-7f3a Test Suite: PASS A test suite executed and every activated lane passed. hotato prove: proof -- overall PASS (exit 0) claim_scope: test_suite evidence_authority: measured lane verdict counts gauntlet pass passed=10 total=10 content_id: sha256:0f5acfd641a8478bdce0c0c823cb72c22bb5479410b706af3a6f9785c06011f0 [proof path elided]
A before/after lane with fewer previously-failing fixtures than
--min-n. Too thin to characterise, so it is inconclusive rather than
a pass, and the exit code keeps CI from reading “could not tell” as green.
hotato prove --before before.json --after after.json --min-n 5 Test Suite: INCONCLUSIVE A lane refused or came back inconclusive; see the table. hotato prove: proof -- overall INCONCLUSIVE (exit 2) lane verdict counts verify inconclusive paired=8 used_to_fail=0 min_n=5 claim_supported=False
the same command with contracts instead, and with a bound candidate
hotato prove --contracts contracts/ Captured Evidence: FAIL claim_scope: captured_evidence evidence_authority: measured lane verdict counts contracts fail contracts=1 passed=0 failed=1 tampered=0 The contracts lane re-measures stored evidence. It does not test the current agent: a fresh recording of the same moment, scored against the same policy, is what clears a pinned bug. hotato prove --before before.json --after after.json --min-n 1 --candidate-config-hash sha256:9c1f --provider livekit Candidate Revision: INCONCLUSIVE claim_scope: candidate_revision evidence_authority: measured
Binding the candidate raised the headline to rung 03; the thin lane still refused the verdict. Scope and verdict are independent, and the proof prints both.
A proof with zero activated lanes is a usage error, not an empty pass:
a proof of nothing is refused. The verdict is fail-closed — overall
pass only when every lane you activated passed.
Say-do
The trace decides, never the agent's words.
The agent said the refund was sent. No issue_refund span succeeded, and
the order's post-call refund_status still reads none. Two
outcome assertions fail, and the dimensions stay separate rather than averaging into
one number.
hotato test run saydo/test.json --agent demo-agent --transcript saydo/transcript.json --trace saydo/trace.jsonl --state saydo/state.json hotato test run: demo-refund-claimed-not-issued (agent demo-agent) -- exit_code=1 success: FAIL (required: all_deterministic_assertions_pass) [X ] all_deterministic_assertions_pass per-dimension (grouped view; never blended): outcome 0 pass / 2 fail / 0 inconclusive policy 0 pass / 0 fail / 0 inconclusive conversation 1 pass / 0 fail / 0 inconclusive speech 0 pass / 0 fail / 0 inconclusive reliability 0 pass / 0 fail / 0 inconclusive deterministic: 1 pass, 2 fail, 0 inconclusive rubric (model-judged, advisory): 0 pass, 0 fail, 0 inconclusive, 0 error (deterministic:false; never merged into the deterministic counts)
The model judge reports in its own lane, marked advisory, and its verdicts are never folded into the deterministic counts. The gate reads the deterministic lane.
In your boundary
The egress map is derived from the code, not written for a page.
Every urllib, socket and networked subprocess call site in the source is
mapped to the command that reaches it, and published as
docs/EGRESS.md.
A command in the local column has no network call site to reach.
| Command | Reaches | When |
|---|---|---|
run, scan, analyze, verify, contract verify, contract pack, contract unpack, describe | nothing | local files only; no urllib import, no socket |
connect | nothing | validates and stores the credential at mode 0600, with no network round trip |
rubric run, the test run judge lane | your local Ollama | default is localhost:11434; a remote endpoint is an explicit flag |
capture, pull, sweep --stack | the stack you configured | when fetching the call is the command's job |
apply | nothing without --yes | the default is a dry run that prints the staging clone it would create |
- 0 runtime dependencies
- audio stays on disk
- MIT
Generate a suite from your prompt
A suite you can diff, from the prompt you already wrote.
Point it at your agent's system prompt. It extracts the capabilities the prompt claims, crosses them with the shipped failure taxonomy, and writes the suite. Pure string matching — no model, no network — so the same prompt writes byte-identical files and a regenerated suite is a diff you can read.
hotato scenario generate --prompt agent.txt --out scn/ read the prompt (sha256 4a4744e5e241): 5 capabilities, 2 required data field(s), 1 stated constraint(s) wrote 55 scenario file(s) to scn/, drawn from 11 failure modes capability: check order status capability: issue refund capability: book delivery slot capability: transfer caller to human pharmacist capability: verify caller's identity with date a starting point you edit: the facts are synthetic placeholders, and a generator cannot know your agent's domain rules or which caller behaviours matter for your call next: hotato lab scenario validate scn/
Five capabilities crossed with eleven failure modes wrote 55 files. Generating the same prompt into a second directory produced a byte-identical tree, so a regenerated suite reviews as a diff. The extraction rules are documented in the module that implements them.
Evidence you can re-derive
Two runs, one digest. Then pack it and carry it.
The scorer is arithmetic over two lanes of energy, so a run is reproducible by construction and cheap enough to put on every call. Below, the battery that ships with the package scores twice to the same digest; a contract packs into a sha256-manifested bundle that unpacks and re-measures somewhere else.
# two independent runs of the bundled self-test battery hotato run --suite --format json > runA.json hotato run --suite --format json > runB.json sha256sum runA.json runB.json 7464e6a87e5896a644515f5e9111d6af4d68bf3bb8a8234a397b85161d4b920a runA.json 7464e6a87e5896a644515f5e9111d6af4d68bf3bb8a8234a397b85161d4b920a runB.json # pack the contract, unpack it elsewhere, re-measure hotato contract pack contracts/demo-missed-interruption.hotato --out demo.hotato.tgz packed -> demo.hotato.tgz (17 files, sha256-manifested) unpacked demo.hotato.tgz -> elsewhere (17 files, sha256-verified; integrity: intact) [FAIL] demo-missed-interruption (expect yield): did_yield=False talk_over=2.51s | integrity: intact
The bundle re-measured to the same verdict after the round trip. The contract pins a known failure, so it stays red until the agent is fixed and the call recaptured — the way a snapshot test stays red until you update the snapshot.
| A 70.6s call, scored | 3.65s | wall clock, one core, 118MB peak |
|---|---|---|
| A 94.3s call, scored | 4.91s | about twenty times faster than the call ran |
| Required runtime dependencies | 0 | dependencies = []; the extras are opt-in |
| Tests in the suite | 5,013 | Python 3.9 through 3.13, MIT |
Measured with /usr/bin/time on one desktop core, over
dual-channel recordings of a live agent on the phone. Your machine will differ; the
point is the order of magnitude, and that nothing here waits on a queue, a GPU or an API.
Where the instrument stops
What it measures, and what it hands to you instead.
The boundaries are the product. An instrument that reported past them would be reporting an opinion.
It measures turn timing and say-do, not intent.
Whether the agent stopped for the caller, and whether the thing it claimed to do shows up in your trace and your state. Whether the caller was satisfied is a different instrument.
Two channels, or it refuses.
Separating who spoke when needs the caller and the agent on their own lanes. Handed a mono file it says so and stops, rather than guessing a diarisation and scoring the guess.
A measured overlap is a candidate a human labels.
The scorer reports coincidence, never causation. It hands you the moment and the numbers under it; the verdict on what the moment meant is yours.
The criterion reads energy, and energy has limits.
A breath and a handover can look alike to an energy threshold, so a hold that pauses can measure as a yield. The confusion matrix, the per-clip error and the command that re-derives both from the labelled corpus are on the evidence page, where the fixtures ship alongside the scorer.
Run your suite. Then prove what it proved.
hotato composes alongside whatever you already use to find problems: run that to discover them, run hotato so the one that hurt you last week stays fixed on every push, inside your boundary, on evidence that outlives both tools.
pip install hotato then: hotato check your-call.wav