Watch it catch an agent talking over the caller.
Two ways to be rude on a call: talk over someone, or stop for a non-turn. This agent does both.
Two recorded calls. Operator-recorded probe calls against a live voice agent on a provider's default settings, picked because they fail. Start here: uvx hotato start --demo sweeps both offline and mints a portable failure contract from the missed interruption. This page walks the same two calls in detail through the packaged battery view.
These calls exercise the Conversation dimension — turn-taking — one of the five a conversation-test scores. Not a vendor benchmark: one assistant, one configuration, one date, one scripted caller. Hotato uses it to demonstrate the threshold funnel, not to rank platforms.
Point it at the bad agent. It fails, out loud.
fd-01 talks over a half-volume “sorry, one second.” fd-02 stops dead for a bare backchannel. One packaged command scores both and writes the report; add --fail for the failing exit code, a red build.
$ uvx hotato demo
hotato demo: recorded calls a provider's default agent fails hotato [suite] stack=generic offline=True 0/2 events pass (failed=2) [FAIL] fd-01-missed-interruption: did_yield=False seconds_to_yield=- talk_over=0.25s fix[config]: Missed interruption: the agent kept talking over the caller knob: interruption sensitivity (VAD min-silence, min-interruption-duration, min-words-to-interrupt) move: lower the min-silence and min-duration thresholds so a real interruption registers sooner [FAIL] fd-02-backchannel-yielded: did_yield=True seconds_to_yield=0.34s talk_over=0.32s fix[engagement-control]: False barge-in: a backchannel was treated as a bid for the floor -> a learned engagement-control / addressee-detection layer note: no single sensitivity threshold satisfies this battery; see funnel pointer in --format json. exit_code=1 these are two recorded calls on a provider's default settings; run it to see what Hotato catches. report: /tmp/hotato-demo-report.html
$
did_yield=False on fd-01: never stopped. did_yield=True on fd-02: stopped for noise. Every number here comes straight off the run. See the exact report it writes →
The missed interruption points at a knob you can turn.
fd-01: a dial set too conservative. Fix map names fix_class: config and parameter: change, re-run, watch green.
# uvx hotato demo --format json | fix_map[0], the fix for the missed-interruption axis
{
"event_id": "fd-01-missed-interruption",
"scenario_id": "fd-01-missed-interruption",
"fix_class": "config",
"title": "Missed interruption: the agent kept talking over the caller",
"detail": "The caller took the floor and the agent never stopped within the search window. Increase interruption sensitivity so a genuine floor-taking event registers.",
"knob": {
"stack": "generic",
"parameter": "interruption sensitivity (VAD min-silence, min-interruption-duration, min-words-to-interrupt)",
"direction": "lower the min-silence and min-duration thresholds so a real interruption registers sooner",
"trade_off": "sliding sensitivity up catches more real interruptions and more false ones together."
},
"pointer": null
}
fix_class: config: yours to tune. Trade-off: dial up catches more false interruptions too.
When the battery fails both ways, a single dial can’t save it.
Catching a quiet interruption needs sensitivity up; ignoring a backchannel needs it down: opposite directions, one dial. Hotato names the discriminator this needs. That’s the funnel pointer’s call.
# uvx hotato demo --format json | funnel, the systemic pointer, fired on the both-axes case
"funnel": { "reason": "This battery fails on BOTH axes at once: it missed a genuine interruption AND it false-triggered on a backchannel. No single sensitivity threshold can fix both - turning it up to catch the interruption makes the backchannel worse, and vice versa. That is the signal that the agent needs a discriminating layer, not a different threshold.", "pointer": { "layer": "a learned engagement-control / addressee-detection layer", "what": "This is a discrimination problem, not a threshold problem: telling a genuine bid for the floor apart from a backchannel or speech that was not addressed to the agent. No single timing threshold separates them - you can raise a words-to-interrupt threshold, but the same threshold that ignores 'mhm' also ignores 'stop'. Separating them needs a signal for 'is this speech addressed to me, and is it a real bid for the floor' - not a config knob.", "honest_scope": "No single timing threshold separates them. Where your stack provides an interruption/backchannel classifier, use it; the general case calls for a learned engagement-control / addressee-detection layer. The audio-only turn-taking case shown here is the hardest modality for it. Treat this as a pointer to the KIND of fix the failure needs, not a benchmarked claim: bring your own recordings and measure." } },
The tool’s words, as-is: “no single timing threshold separates them… a pointer to the kind of fix the failure needs, not a benchmarked claim.” No headline number, no upsell. A knob gets a knob; this gets a straight answer.
A different battery, and it agrees with every label.
Not fd-01/fd-02 rescored: Hotato’s own built-in self-test, eight synthetic scenarios each pinned to an expected yield or hold. All eight agree today: exit_code=0. A pass means it stopped (or held) at the right moment.
02/07 hold the floor through backchannel and echo (did_yield=False, by design): a pass here isn’t just always yielding.
$ uvx hotato run --suite barge-in
note: --suite is Hotato's SELF-TEST on synthetic fixtures -- it checks the tool itself, not your agent. To score YOUR agent, bring a real dual-channel call: hotato capture --stack vapi --call-id <id> (see: hotato) hotato [suite] stack=generic offline=True 8/8 events pass (failed=0) [PASS] 01-hard-interruption: did_yield=True seconds_to_yield=0.50s talk_over=0.50s [PASS] 02-backchannel-mhm: did_yield=False seconds_to_yield=- talk_over=1.57s [PASS] 03-filler-start: did_yield=True seconds_to_yield=0.65s talk_over=0.56s [PASS] 04-correction: did_yield=True seconds_to_yield=0.50s talk_over=0.50s [PASS] 05-telephony-8khz: did_yield=True seconds_to_yield=0.50s talk_over=0.50s [PASS] 06-double-talk: did_yield=True seconds_to_yield=1.05s talk_over=1.05s [PASS] 07-echo-bleed: did_yield=False seconds_to_yield=- talk_over=3.00s [PASS] 08-rapid-turn-taking: did_yield=True seconds_to_yield=0.50s talk_over=0.50s exit_code=0
$
Same input, same verdict, every run, offline. Proves the plumbing. Your verdict comes from your recording, next.
That was a recorded call on defaults. Yours is one command away.
Point it at a call you have. capture pulls the dual-channel recording, scores it offline. Audio never leaves the machine.
$ uvx hotato capture --stack vapi --call-id <id>