Getting started
One file describes a conversation; one command scores it across five dimensions, offline. Your audio, transcripts, and traces stay on your own infrastructure.
1. See it work with zero setup
The flagship first run needs no account, no network, and no credentials. It sweeps the two bundled demo calls, turns the strongest failing moment into a portable failure contract, verifies that contract (it fails, because the recording shows the agent talking past a caller interruption), and prints the exact next commands.
# zero-setup first run: sweep the bundled calls, mint a contract, verify it uvx hotato start --demo [start] demo: swept 2 bundled calls, 5 candidate moments; wrote hotato-sweep.json, hotato-sweep.html, hotato-no-single-threshold.svg, contracts/demo-missed-interruption.hotato hotato start: swept the 2 bundled demo calls offline. sweep result: hotato-sweep.json sweep dashboard: hotato-sweep.html funnel card: hotato-no-single-threshold.svg demo contract: contracts/demo-missed-interruption.hotato verified contract: FAIL as expected -- the demo call missed the interruption; a CI gate on this contract catches any change to the evidence or policy (start --demo itself exits 0 because setup succeeded; the next command runs the contract's CI gate) Next steps (all offline, no credentials): 1. Save a candidate as a permanent regression test: hotato fixture promote hotato-sweep.json#1 --expect yield --id my-first-fixture --out tests/hotato 2. Run your fixtures in CI: hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio 3. Re-verify the demo failure contract in CI: hotato contract verify contracts/
Step through the demo call it catches → · Rank every candidate at once →
2. The unit of work: a conversation-test.yaml
One file names the agent under test, a simulated caller, the two separate assertion lanes, and a boolean success condition. hotato scenario init writes a starter you edit.
uvx hotato scenario init refund-flow --agent support-v3 --out refund.conversation-test.yamlDeterministic checks and the model-judged rubric live in separate lanes. Each deterministic check carries a dimension tag — one of the five — so results group by dimension. The tag is a grouping key, never a weight, and there is no blended field anywhere.
kind: hotato.conversation-test version: 1 id: refund-flow agent: support-v3 caller: persona: a customer whose order arrived damaged and wants a refund goal: get a refund for order A-1001 facts: {order_id: A-1001} assertions: # DETERMINISTIC lane: pure, offline, no model. Each check is TAGGED with a dimension. deterministic: - {id: disclosure-said, kind: phrase, regex: "recorded for quality", role: agent, dimension: policy} - {id: refund-issued, kind: tool_call, name: issue_refund, dimension: outcome} - {id: lookup-then-refund, kind: sequence, steps: [{tool: lookup_order}, {tool: issue_refund}], dimension: conversation} - {id: refund-latency, kind: latency, tool: issue_refund, max_ms: 1500, dimension: speech} # RUBRIC lane: a pinned LOCAL model, advisory (never gates unless --gate-judge). rubric: - {id: was-empathetic, kind: judge_rubric, dimension: conversation, criterion: "Did the agent acknowledge the caller's frustration before proposing a fix?", evidence: [transcript]} repetitions: 1 success: # Success is the conjunction of named conditions -- never a score. required: [all_deterministic_assertions_pass, no_rubric_failure] report_dimensions: [outcome, policy, conversation, speech, reliability]
| Dimension | What its checks establish | Reads |
|---|---|---|
| Outcome | Did the task get done — the refund issued, the appointment booked. | --trace |
| Policy | Were the rules followed — disclosure read, PII handled, no forbidden phrase. | --transcript |
| Conversation | Did turn-taking and tool sequence hold — the timing proof-point below. | --audio, --trace |
| Speech | Was the spoken exchange within its timing bounds — latency, response gaps. | --audio, --trace |
| Reliability | Does the same call hold up across repeated runs — pass@1 / pass@k with a Wilson interval. | --repetitions |
Structurally validate a file, or a directory of them, before you rely on it: uvx hotato scenario validate refund.conversation-test.yaml.
3. Run it across the five dimensions
hotato test run loads the file, builds a context from the call you supply, evaluates the deterministic lane, checks the file’s success conditions, and prints a per-dimension scorecard. A missing input leaves a check inconclusive rather than guessed, and the exit code honors the file’s inconclusive_policy.
uvx hotato test run refund.conversation-test.yaml --agent support-v3 \ --transcript call.transcript.json --trace voice_trace.jsonl hotato test run: refund-flow (agent support-v3) -- exit_code=0 success: PASS (required: all_deterministic_assertions_pass, no_rubric_failure) per-dimension (grouped view; never blended): outcome 1 pass / 0 fail / 0 inconclusive policy 1 pass / 0 fail / 0 inconclusive conversation 1 pass / 0 fail / 0 inconclusive speech 1 pass / 0 fail / 0 inconclusive reliability 0 pass / 0 fail / 0 inconclusive deterministic: 4 pass, 0 fail, 0 inconclusive rubric (model-judged, advisory): 0 pass, 0 fail, 1 inconclusive (deterministic:false; never merged into the deterministic counts)
Every result carries an evidence tier (0–4): none, asserted, measured, paired, attested. A phrase match measured off the transcript sits above a bare assertion; a paired input-and-output check sits above a single-sided one. The tier travels with the result so a reader knows what stands behind a PASS. Add --out ./artifact --format html to bind the evidence into a shareable report.
4. Bring your own call
The Conversation and Speech dimensions read the audio; Outcome and Policy read the trace and transcript. Capture a call from your stack — caller and agent on separate channels — and feed it to the same conversation-test.
# pull the dual-channel recording from your stack and score it offline export VAPI_API_KEY=... uvx hotato capture --stack vapi --call-id <id> # no key yet? run the whole capture→score loop on a bundled call uvx hotato capture --stack vapi --demo
One mixed mono file can’t tell whose overlap is whose, so the Conversation timing lane doesn’t take it. If your stack only exports a mixed file, switch on separated recording; the capture guides show the setting per stack.
--stack accepts vapi, twilio, livekit, pipecat, retell, or generic. It names the fix knobs for your stack; it never changes a measurement.
5. The Conversation dimension, up close
Turn-taking timing is the deterministic engine behind the Conversation dimension: whether the agent yielded the floor to a caller interruption, how long the yield took, and how long the two overlapped. hotato run scores those signals directly on a dual-channel recording. --expect yield means stop for the caller; hold means keep the floor through a backchannel like “mhm”.
# two-channel WAV (caller on ch0, agent on ch1) uvx hotato run --stereo call.wav --stack vapi --expect yield # two separate mono files + a caller onset label (seconds) uvx hotato run --caller caller.wav --agent agent.wav --onset 2.4 --expect yield
Each event reports three timing signals — did_yield, seconds_to_yield, talk_over_sec — and a PASS/FAIL. --format json emits the machine envelope for CI; --dump-frames writes re-derivable per-frame evidence. To exercise the scorer itself on synthetic fixtures: uvx hotato run --suite barge-in.
6. Or let doctor do all of it
One command scores your recording (or the self-test), renders the visual report, and opens it. Found a bad call? Turn it into a regression test → · see the same loop run against twelve recorded calls →
uvx hotato doctor --stereo call.wav # score, render, open uvx hotato doctor # self-test fallback, same flow
0 all pass (or --no-fail) · 1 a required condition failed · 2 usage/I/O error, or an input that is not scorable: the CI gate; see CI integration.