How it works

Hotato scores a conversation across five independent dimensions — Outcome, Policy, Conversation, Speech, Reliability — from a single conversation-test.yaml. Deterministic assertions and the model-judged rubric stay in separate lanes, results group per dimension, and there is no blended score.

This page details the deterministic engine behind the Conversation dimension: turn-taking timing from an energy-based voice-activity detector. It measures when there was speech-level energy, not what was said, or who said it — every threshold an exposed parameter, every frame inspectable.

Five dimensions, two lanes, one file

A conversation-test.yaml is the unit. Every check it declares lands in one of two lanes and carries one of five report dimensions as a grouping tag — never a weight.

The two lanes never merge. Deterministic assertions are pure regex, checksum, span-lookup, and timing math, with no model call. The rubric lane is a pinned local model, advisory by default (it gates only under --gate-judge). Success is a boolean conjunction of named conditions; no dimension is folded into a percentage. The rest of this page is the Conversation dimension in full.

The pipeline, in short

Runs offline, standard library, no network. Deterministic under pinned inputs: the same version, audio, channel map, onset, label, and scoring config give the same result in the verified environments (CI on Linux x86_64). A changed result means a pinned input, policy, or scorer component changed.

1. Per-frame RMS. Each channel is cut into 20 ms windows stepped by 10 ms; the linear root-mean-square amplitude is computed per window. 10 ms is the time resolution.
2. RMS → dBFS. Each value becomes decibels relative to full scale, with a floor to avoid log(0).
3. Per-channel energy VAD. The quietest 10% of frames estimates the noise floor; the threshold is noise_floor + 15 dB, but never below a −60 dBFS absolute gate. A dynamic-margin guard keeps the threshold below speech on a rarely-silent channel. A 150 ms hangover keeps one utterance whole across brief gaps.
4. Caller onset. Supplied as a label (--onset) or taken as the start of the caller’s earliest sustained active run. The label is preferred, because on mixed audio the earliest energy is not always the caller.

What it reports

Nothing to score (silent caller, agent not talking at onset)? The event returns scorable: false with a plain reason, never a verdict. Single-recording runs exit 2: CI reads an input problem, never a pass.

Two additive signals: echo and resume

Two further checks run on the same VAD tracks, entirely in hotato’s own layer. Both add an optional signals block; neither changes did_yield, seconds_to_yield, or talk_over_sec for any event.

• Echo. Leaked TTS looks like a delayed, scaled copy of the agent’s own audio arriving back on the caller channel. signals.echo searches lags from 0 up to 0.5 s and reports the best cross-channel coherence found; echo_suspected flags a match at or above 0.7 cosine similarity with at least 8 overlapping frames. Independent speech, real turn-taking or a backchannel, does not correlate with the agent’s own envelope at any lag, so it scores low. --echo-gate (opt-in, on hotato run) holds an echo-suspected yield out of the verdict instead of counting it as a clean pass; without the flag signals.echo is still computed and reported, it just does not gate.
• Resume. On an event where the agent yielded, signals.resume looks for a fresh agent onset within 4 seconds of the yield: resumed (found one or not), resume_gap_sec (seconds to it, null if none), and restart_suspected, whether the longest contiguous run from that onset reaches 2 seconds, the timing fingerprint of re-answering a whole paragraph from the top rather than finishing the interrupted clause. Whether the resumed words literally repeat the earlier ones is a transcript question, out of scope here.

Out of scope

Everything above is timing. None of it is meaning. What the scorer never claims, on any call:

• No speaker identification: an opt-in diarizer (mono files only, see below) assigns anonymous speaker labels; it never says who a person is.
• No speech-to-text / transcription.
• No emotion or intent detection.
• No claim about any specific vendor’s internal accuracy.

This is the out-of-scope list a generated report or card links back to here instead of restating.

Why it is reproducible

• Deterministic given (audio, config): no randomness, no network, no hidden state.
• Every threshold that drives the active decision is a named, overridable parameter, echoed back in the output.
• --dump-frames writes, per frame: t_sec, per-channel dBFS, active flags, and the constant thresholds and noise floors, so you can re-derive any reported number by hand.

# every number on the table, re-derivable from the frame dump
uvx hotato run --stereo call.wav --dump-frames frames.json

Two channels is the high-validity mode

The scorer reads one track per party: one two-channel WAV, or two mono WAVs starting at the same instant. With separate tracks, an overlap has an owner: you see who was already talking and who cut in.

A single mixed mono file cannot show whose speech an overlap belongs to on its own, so by default it is refused (exit 2). capture, pull, and sweep can still fetch or score one with --allow-mono: stamped indicative_only, no contract-grade talk-over attribution.

hotato run --mono call.wav --diarize takes a different opt-in path: a speaker diarizer splits the mix into caller and agent activity first, and the same scorer runs on the result. Pick the backend with --diarizer: pyannote (local, offline, the default) or sortformer (local, GPU) run entirely on your machine; pyannoteai (hosted) is the only one that sends audio off it, and only behind --egress-opt-in. Confidence gates the result: above the bar the verdict is stamped diarized-mono; below it, indicative_only with no SLA gate; a file that will not separate is refused. Check separability before scoring with hotato trust --stereo call.wav --diarize. The diarizer assigns anonymous speaker labels, never a person’s identity, and neither mono path is ever equivalent to the dual-channel reference for sub-second talk-over attribution.

Optional neural cross-check

The energy VAD above is the reference: every published, golden, and bundled number comes from it, deterministically. --backend neural (the default is energy) re-runs the identical timing math over Silero VAD, a learned speech model, run locally and offline, instead. Install it with pip install 'hotato[neural]', then hotato run --stereo call.wav --backend neural. The --suite self-test always scores with energy, so --backend neural is ignored there.

It returns the identical result shape as energy, aligned to the same 10 ms hop grid. A learned model has no dB threshold, so the threshold_db / noise_floor_db it reports are the energy-domain description of the same audio, for inspection only; the neural active decision is a learned speech probability, not a dB crossing.

It tightens onset precision on clean speech but does not close the energy-vs-intent gap: a cough, a laugh, or crosstalk still carries speech-band energy under either backend, and no accuracy percentage is claimed for either one. Without the [neural] extra, --backend neural errors cleanly; it never silently scores with energy and presents that as the neural result.

Validity is measured, not claimed

The bundled fixtures are synthetic, rendered from exact declared segment boundaries, so the scorer’s own measurement error is computable.

Report that: timing error in milliseconds per scenario, and a four-cell confusion matrix for did_yield (correct yields, missed yields, correct holds, phantom yields).

Never blended into a single accuracy percentage: a missed interruption and a phantom yield are different failures with different fixes. One number would hide that distinction.