Tiered corpus suites

Four labelled suites, 112 scenarios, every one synthetic shaped noise rendered deterministically from its own labelled timings. The timings are the ground truth, and the defect suites exist to fail.

Suite	Scenarios	Conditions	Expected exit
`silver`	40	clean, 16 kHz, default noise floor	`0`
`silver-defects`	16	clean conditions, deliberate defect renders	`1`
`gold`	40	hard conditions: noise floors, 8 kHz, gain extremes, echo, edge timings, endurance	`0`
`gold-defects`	16	hard-condition defect renders, plus two labelled capture-defect cases	`1`

The families cover hard interruptions (onset, speed, duration, resume), backchannels (position, density, repeats, long acks), double-talk, one-word interruptions, stutter onsets, multi-turn exchanges, resume-then-reinterrupt, and latency prompts.

corpus/suites/manifest.json is the machine-readable inventory.

Defect suites fail by design

Every scenario in a defect suite is rendered to fail on its labelled axis: an agent that keeps talking through a real interruption, yields to a backchannel, or misses its latency budget. Exit 1 there is the scorer catching what it claims to catch.

The negative control

A suite that can only pass proves little. The defect suites are the negative control that gives the passing suites their meaning: the same scorer, the same thresholds, and a known-bad render must come back red.

Run a suite

Any suite is a scenarios directory plus an audio directory, on every surface: run, report, export, and the pytest gate.

bash

hotato run --suite barge-in \
  --scenarios corpus/suites/gold/scenarios \
  --audio corpus/suites/gold/audio

Deterministic builder

The suites are generated by one script, and the generator is the proof. Each scenario renders from a seed derived from sha256(scenario_id), so the audio on disk is exactly what the labelled timings say it is; the same seed renders byte-identical audio, verified by --check.

The same seed renders byte-identical audio; --check regenerates to a temp directory and byte-compares against the checked-in corpus.

bash

python3 corpus/suites/build_suites.py           # rebuild in place
python3 corpus/suites/build_suites.py --check   # regenerate to a temp dir, byte-compare
build_suites --check: regenerated output is byte-identical to disk

This is timing reproducibility, the one claim on this site made outside the evidence levels: it proves the measurement is stable, not that any flagged behavior is real. Byte-identical re-runs are verified in CI on Linux x86_64 across Python 3.10, 3.11, and 3.12; macOS and Windows are now checked in the same workflow but not yet green.

Additive scenario classes

corpus/classes/ ships four small, deterministic classes on top of the four suites above, built the same way: synthetic shaped noise rendered from each scenario's own labelled timings, seed sha256(scenario_id), so a rebuild is byte-identical, verified by --check. 13 scenarios across four classes.

Class	Scenarios	Axis	What it holds
`mid-utterance-pause`	3	latency	The caller pauses mid-turn for a multi-second thinking gap, then resumes. Scored with a widened `turn_end_silence_sec` so the pause is not mistaken for the caller's true turn end. One defect: the agent grabs the floor inside the pause.
`backchannel-multilingual`	5	should not yield	Short non-English acknowledgement tokens (Hindi, Telugu, Spanish, Japanese) over agent speech. The VAD is energy-based, not lexical: it does not detect language. One defect: a false yield on the Spanish token.
`noise-hold`	3	should not yield	Sustained non-speech ambient energy on the caller channel (cafe, TV), not a brief backchannel. One defect: a false yield triggered by the ambient energy.
`telephony-degraded`	2	should yield	An existing gold scenario re-rendered through G.711 mu-law plus a fixed packet-loss schedule. One pass, one defect fail: the verdict holds across codec degradation.

corpus/classes/manifest.json is the machine-readable inventory.

Kept separate from corpus/suites/ because mid-utterance-pause needs a non-default turn_end_silence_sec that the generic, dynamically-discovered suite tests do not apply.

bash

python3 corpus/classes/build_classes.py          # write labels + render audio
python3 corpus/classes/build_classes.py --check  # regenerate to a temp dir, byte-compare

Full per-class detail: corpus/classes/README.md.

Synthetic is the floor

These suites prove the plumbing and catch regressions. Validity on your system comes from your own labelled calls: the walkthrough is docs/SUBMITTING.md, and running suites against a live stack is docs/BENCHMARK-STACKS.md. For how far the scorer's own measured event times land from each labelled ground truth, see docs/BENCHMARK.md.

Contribute

New synthetic families are welcome through the normal PR path. The highest-value contribution stays a real, consented, labelled call: open the corpus-submission issue form and it walks you through record, label, validate, submit.