Home/Docs/The regression loop

Turn a bad call into a regression test

A production call where the agent talked over someone is your best test asset. Eight steps, straight hotato commands, from that call to a CI gate that catches the bug if it comes back.

Already know the bad call? Start here. A folder of recordings or a connected stack to search first? See the closed loop for candidate discovery, then come back to keep the one that matters.

Fix in your stack, check in Hotato

Your stack’s runtime controls fix turn-taking. Hotato re-scores the stored evidence on every push and fails CI when a pinned input, policy, or scorer result changes. That confirms the stored evidence and scorer are still intact, not that your current agent still avoids the bug. That takes a fresh recapture.

Stored-evidence checkLevel 3

The historical call plus your frozen policy: the stored recording still produces the expected result under the pinned scorer. Does not establish current-agent behavior.

Fresh-recapture checkLevel 4

A newly recorded call against your current agent: the new recording matched the same contract. Does not establish causation, general reliability, or future behavior.

The loop in commands: hotato capture pulls the call, hotato diagnose and hotato plan name the fix, hotato scan finds an unlabeled moment, hotato compare measures the before/after, hotato fixture create keeps the call, hotato run re-scores it on every push.

1. Export the dual-channel recording

Pull the bad call with caller and agent on separate channels. The capture guides show the setting per stack.

bash

# auto-pull from your stack (the download is the only network call)
uvx hotato capture --stack vapi --call-id <id>
uvx hotato capture --stack twilio --recording-sid RE...

2. Score it

One command scores offline, writes the visual report, and opens it.

bash

uvx hotato doctor --stereo call.wav

3. Read the failure and the fix class

diagnose names exactly one of two fix classes per failure: a config knob you can turn, or a discrimination case no single threshold settles. The loop below tunes the config case; the engagement-control case gets a pointer, not a patch.

fix[config]: a stack setting is off. The output names the knob.
fix[engagement-control]: a discrimination case no single threshold settles.

For the full picture, score to a file and hand it to diagnose: a per-failure explanation plus one battery-level decision.

bash

uvx hotato run --stereo call.wav --format json > result.json
uvx hotato diagnose result.json

Read-only: diagnose fetches nothing and changes nothing. A battery that misses a real interruption and false-stops on a backchannel gets named outright: do_not_tune_single_threshold.

4. Tune your stack’s controls

The fix lives in your runtime, in one of these control families:

Interruption settings: sensitivity, words-to-interrupt or minimum-duration thresholds.
Endpointing: min-silence and end-of-turn latency.
VAD sensitivity on the input path.
TTS buffering: how much queued speech plays after a stop signal.

The same four places a call’s audio moves through your stack: sensitivity on the way in, two threshold decisions, buffering on the way out. Hotato never sets any of them, and only ever pins a direction on two: interruption settings or endpointing. A slow yield that could be VAD smoothing, TTS buffering, or transport latency can’t be told apart from one recording, so plan hands you an instrumentation checklist there instead of a guessed knob.

Per-stack knob field names, bounds, and worked examples: FIX-PLANS.md in the repo.

Or skip naming the knob yourself: plan combines the diagnosis with your stack’s live config into one guarded, bounded step.

bash

uvx hotato plan result.json --stack vapi --assistant-id <id>

One bounded move on one setting, never an absolute value: from → to is one step relative to the inspected current value. production_apply is always false; nothing is applied for you. When the battery fails on both axes at once, plan refuses to name a threshold and points at a discriminating layer instead.

5. Not sure exactly where the moment is? Scan the callLevel 1

You know the call was bad, but not the second. scan walks the whole recording and lists candidate turn-taking moments as timing facts only, ranked by salience; nothing is labeled for you.

bash

uvx hotato scan --stereo call.wav
# narrow it down, or hand every candidate to a script:
uvx hotato scan --stereo call.wav --top 5
uvx hotato scan --stereo call.wav --format json --out candidates.json

Candidates are overlap onsets, agent starts during caller speech, and long response gaps. You pick the one that matters and supply the yield or hold label in the next step.

6. Re-run and compare

talk_over_sec on the same clip, before and after the config change: 2.65 seconds down to 0.42. The compare run that produces these two numbers follows.

Replay the same caller audio against your changed agent, record the new call, and let compare score both takes with the identical expectation and bounds.

bash

uvx hotato compare --before call.wav --after call-after.wav \
    --onset 42.18 --expect yield --out compare.html

output

hotato compare: call.wav -> call-after.wav
  verdict:           FAIL -> PASS
  did_yield:         false -> true
  seconds_to_yield:  - -> 0.42s
  talk_over_sec:     2.65s -> 0.42s  improved -2.23s

result: fixed

One machine-stable result word every time: fixed, regressed, improved, worse, unchanged, still_pass, or not_scorable. compare measures; it does not gate CI by itself. Pass --fail-on-worse if you want it to.

7. Keep the recording as a fixtureLevel 2

One bad call, kept forever: fixture create clips the moment and writes the label with provenance; every push re-scores it.

fixture create clips the moment, re-bases the onset to the clip, writes the scenario label with provenance, and scores it immediately: an input that cannot be judged is refused with the reason, never written as a fixture that would report a meaningless verdict.

bash

uvx hotato fixture create --stereo call.wav --id prod-2031-talk-over \
    --onset 42.18 --expect yield \
    --max-talk-over 1.0 --max-time-to-yield 0.9 \
    --out your/hotato

This writes your/hotato/scenarios/prod-2031-talk-over.json (the label, with provenance) and your/hotato/audio/prod-2031-talk-over.example.wav (a two-channel clip around the event, onset re-based to the clip). Backchannel case? Pass --expect hold instead; --category is derived from it unless you set it directly.

Your suite, your repo

This is your own regression suite; nothing leaves your machine. Strip PII before committing audio anywhere shared. To donate a fixture to the public corpus, see Contributing.

8. Gate CI on it

Every failing fixture is a red build, with exit codes CI already understands.

bash

# run your fixtures: exit 1 on a regression
uvx hotato run --scenarios your/hotato/scenarios --audio your/hotato/audio

# or gate the pytest run you already have
pytest --hotato-suite --hotato-suite-scenarios your/hotato/scenarios --hotato-suite-audio your/hotato/audio

The GitHub PR check posts one sticky results comment per pull request and fails the job on a regression. Red again? Score the new failure to JSON and hand it back to hotato plan (step 4).

The durable loop

Every bad call you keep makes the suite stricter: that frozen recording gets re-scored on every push, confirming the evidence, policy, and scorer are still intact (the stored-evidence check). It does not confirm your current agent still avoids it; that takes a fresh call, via another sweep or capture (the fresh-recapture check).

See this loop fire on twelve live calls against one default configuration →