Twelve maintainer-recorded live probe calls, scored against one default configuration
Twelve calls on one stock assistant, scored offline with Hotato: turn-taking timing measured to the frame.
Scored with Hotato 1.0.0 · captured 2026-07-09 · corpus revision 33985d1 (v0.8.0) · reproduce: hotato test run on corpus/probe-defaults
One human caller worked from a written script against a synthetic voice assistant; script labels are not model output. Hotato measures timing from the two audio channels only. A field note is the operator’s own per-moment observation, logged separately; where the two disagree, this page says so, and the measurement is what gets scored. Clips are MIT licensed, in the repository.
| Moment | What happened | Scored result |
|---|---|---|
| Backchannel (“mhm”, “yeah”), 4 moments | Three of four backchannels yield as fast as a real interruption; the fourth is held. | Fail · 3 of 4 false yields |
| “Stop.” command | Agent runs 1.26s past the command; yield scored at 1.46s, 0.46s past the 1.0s bound. | Fail · 1.46s vs 1.0s bound |
| Quiet interrupt (half-volume), moment 10a | Never crosses the default energy threshold; agent talks through the whole attempt. | Fail · missed entirely |
| Quiet interrupt, louder retry, moment 10b | Same caller, same call, retried louder: crosses the threshold and yields. | Pass · 0.59s talk-over |
| Thinking pause (4s) | Caller falls silent mid-sentence to keep the floor; agent starts 3.44s in. | Latency-only, not scored pass/fail |
| Correction retry | First attempt not picked up, dead air until the caller re-prompts; the retry yields in 1.16s. | Fail · needed a retry |
| Hard interrupts, double-talk, greeting overlap (7 moments) | Clean floor holds or a genuine interruption is caught, 0.00-0.40s. | Pass · 7 of 7 |
| Call 11, clean floor | 82.5 seconds, no caller-during-agent onset anywhere: the baseline every measurement departs from. | Clean-floor baseline |
The same reflex speed, one correct and one not.
A genuine interruption caught in 0.35 seconds; a backchannel mistaken for one in 0.34.
The moments that expose the funnel
A command ignored, a quiet interrupt missed, a thinking pause misread, and a correction that needed a retry.
The full “Stop.” timeline, zeroed at the caller’s command:
The funnel verdict
The battery fails on both axes at once: it misses a genuine interruption and false-yields on backchannels in the same run.
Run against the battery, hotato plan reports exactly that, and refuses to propose a threshold change:
hotato plan [stack] finding=threshold_funnel decision=do_not_tune_single_threshold config_only_safe=false production_apply=false (approval: manual) hypothesis: The battery missed a genuine interruption and also stopped for a backchannel. One sensitivity threshold cannot satisfy both: raising it to hold through backchannels drops real interruptions, lowering it to catch interruptions yields to backchannels. The failure class is discrimination, not calibration. recommended fix class: engagement-control platform mutation: performed=false (hotato plan is read-only)
Trimmed to the decision and hypothesis. The full JSON and manifest are in the repository: the corpus on GitHub →
One assistant, one configuration, one date, one scripted caller: enough to demonstrate the threshold funnel, not enough to rank platforms.
Score your own calls the same way.
Twelve calls, one open corpus. Point Hotato at your own next.
$ uvx hotato start --demo
One recording is enough to start. Get started → · the corpus →