hotato
GitHub

What Hotato validates.

The evidence behind the deterministic Conversation dimension: three checks, each read on its own.

Generated from Hotato 1.0.0 on 2026-07-09: bundled corpus and demo fixtures, not customer calls.

hotato · one corpus, three commands
hotato's bundled corpus 01 hotato test run timing reproducibility 02 hotato suite run candidate moments 03 hotato start --demo fixture pass-fail Same corpus in. Three unrelated questions out.

Five stages of evidence, weakest to strongest.

Every finding names its stage and claims only what it shows.

Stage 1 Candidate A timing event worth review; a person confirms it. Stage 2 Human-labeled failure A reviewer confirmed the recording violated a yield-or-hold expectation. Stage 3 Stored-evidence check Stored audio still produces the expected result under the pinned scorer. Stage 4 Fresh-recapture comparison A newly captured call passed the same contract; no paired guard regressed.
Stage 5 External proof Only an outside team supplies this: their regression caught, or their recapture passing.

Timing reproducibility, candidate discovery, fixture agreement.

Reproducibility sits outside the ladder: a scorer property, not a finding's maturity.

Timing reproducibilityOutside the ladder

Does the same recording produce the same measurements every run? Yes: no learned weights, no sampling, no RNG. Pin version, audio, and config, and results are byte-stable in CI; a changed result means a pinned input changed.

What is reported

did_yield, seconds_to_yield, and talk_over_sec with exact thresholds: the measurement is stable, not that 0.51s is the true latency.

Candidate-discovery usefulnessStage 1

Does hotato suite run surface the moments a reviewer wants to check? Each candidate is an overlap or a gap, ranked by size, never a verdict or an intent label.

hotato suite run
$ hotato suite run --stereo 02-backchannel-mhm.example.wav --top 5
hotato suite run: 02-backchannel-mhm.example.wav  (6.0s, 3 candidate moments)
Candidates are timing events. You decide the expected behavior.
[ 1] t=2.09s  overlap_while_agent_talking  overlap=1.58s  agent did not go silent
[ 2] t=3.19s  overlap_while_agent_talking  overlap=1.07s  agent did not go silent
[ 3] t=4.29s  overlap_while_agent_talking  overlap=0.56s  agent did not go silent
ranked by overlap size
t 0s 1s 2s 3s 4s 5s 6s [1] t=2.09s overlap=1.58s [2] t=3.19s overlap=1.07s [3] t=4.29s overlap=0.56s
Shareable card: Hotato found a candidate talk-over of 1.58 seconds of overlap while the agent was talking, at t equals 2.09 seconds in the recording. Footer reads: Hotato reports timing candidates, not intent.
Every candidate exports as a card.
What is reported

A ranked list of timing facts: a quiet region is not proven clean.

Fixture pass-fail agreementStage 3

Once you label the expected behavior, does the verdict agree on the audio? Both failures below are caught, and no single threshold fixes both: a missed interruption and a false stop need opposite repairs.

hotato start --demo
$ hotato start --demo --no-open --format text
  0/2 events pass  (failed=2)
  [FAIL] fd-01-missed-interruption: did_yield=False talk_over=0.25s
         fix[config]: the agent kept talking over the caller
  [FAIL] fd-02-backchannel-yielded: did_yield=True t=1.28s trailing_silence=0.46s agent_stop_no_caller
         fix[engagement-control]: a backchannel was treated as a bid for the floor
  note: no single sensitivity threshold satisfies this battery
  exit_code=1
two fixtures, two different fixes0/2 pass
fd-01-missed-interruption FAIL · kept talking talk_over=0.25s fd-02-backchannel-yielded FAIL · stopped too soon silence=0.46s 0.0s 0.6s
Shareable card: Hotato found a candidate false-stop of 0.46 seconds of silence after the agent stopped, with no caller nearby, at t equals 1.28 seconds in the recording. Footer reads: Hotato reports timing candidates, not intent.
Surfaced at discovery, then labeled into the Stage 3 verdict above.
What is reported

Per fixture: the verdict, the signals behind it, and a named fix class when the failure maps to a config family.

Each mechanism does one job, and names it.

The Conversation scorer measures turn-taking timing to the frame; everything else has its own dimension or your label.

measures
timing / turn-taking: who held the floor, for how long. Same audio, same numbers.
you label
semantic intent: whether a sound means stop or keep going, declared in a conversation-test.yaml.
Outcome dimension
task success: whether the call booked the appointment, scored on its own pass.
model judge
satisfaction / sentiment: tone belongs to the advisory model judge, never gating a verdict.
candidate
root cause: a slow yield can be TTS, transport, or VAD; diagnose names the likely layer.
one call
vendor comparison: a provider-default example is one assistant, one config, one date.

More evidence, one link each

Live probe calls

Twelve probe calls

One stock configuration, scored offline, with full timelines and audio.

 Demo report

A generated artifact

Two events scored offline, zero pass: a Hotato report.

 Trust and limits

The input contract

Ten input conditions mapped to score, caveat, or refuse.

 Corpus

Every clip, licensed

The consent and licensing ledger, MIT and CC BY 4.0 only.

Run the same corpus. Read the same output.

One bundled corpus, three independent commands, byte-stable on a fixed version.

$ uvx hotato start --demo

One recording is enough to start. Get started → · the full corpus →