Find the candidate bad calls, then close the loop

hotato analyze and hotato connect / pull / sweep rank candidate turn-taking failures across a folder or your recent calls. Label the ones that matter; plan, patch, verify, and loop carry the fix from proposal to paste-ready artifact to a before/after check. The two irreversible decisions stay in your hands.

1. Find the candidate bad calls in a folder

hotato analyze is zero-config: no scenarios, labels, onset, or flags. Point it at a folder of dual-channel recordings; it walks every .wav with the same whole-call scanner as hotato scan, then ranks candidate moments by salience so the worst float to the top.

# one self-contained, offline HTML dashboard, ranked by salience
uvx hotato analyze ./recordings

# a bare folder as the first argument routes to the same command
uvx hotato ./recordings

# ranked candidates for an agent instead of a browser
uvx hotato analyze ./recordings --format json

The dashboard embeds the audio around the top moments (base64, nothing uploaded, zero external requests); a playhead sweeps the timeline in lockstep with playback, so you hear the talk-over or dead-air gap land exactly where the chart marks it. Files that cannot be scored (mono mixes, unreadable audio) land in a Skipped section with their reason; a bad file never crashes the run.

2. Or find them across your recent calls

Already have API access to your stack? Connect once, then pull and score recent calls in one shot, no per-call ids. Credentials are stored locally at ~/.hotato/connections.json (file mode 0600) and sent only to the vendor’s own API, never to Hotato.

uvx hotato connect vapi --api-key <key>
# --stack and the key become optional once exactly one stack is connected
uvx hotato sweep --since 7d

sweep is pull (list recent recordings, download each with the same fetch hotato capture uses) then the same zero-config analyze over the pulled folder, into one dashboard. pull and sweep both work standalone too.

LiveKit and Pipecat are capture-in-your-infra: there is no vendor recording to pull, so they are not connectable. See LiveKit / Pipecat capture. Mono/mixed stacks cannot attribute overlap to caller or agent, so their calls surface in the dashboard’s Skipped section, with the reason shown. Full per-stack endpoint provenance: ADAPTER-STATUS.md in the repo.

Set and forget: a webhook that scans every completed call

A sweep is a pull you run. To make it continuous, scaffold a small self-hostable worker that turns your platform’s call-ended webhook into a scheduled batch scanner: every finished call is fetched read-only and scanned for candidate moments, no CLI to remember.

# scaffold a self-hostable worker for your stack
uvx hotato init webhook --stack vapi --target fastapi --out hotato-webhook

# each payload flows to the same composable primitive
uvx hotato ingest --stack vapi --event payload.json   # + VAPI_API_KEY

3. Label the moments that matter

Hotato never infers intent, in a folder scan or a single call. You listen to a candidate, decide the expected behavior, and freeze it as a fixture. The lowest-friction path promotes a candidate straight off the ranked result from step 1 or 2: fixture promote FILE#N carries the recording, onset, and kind; you add only the yield or hold label.

# step 1 or 2, but as a ranked result file instead of a dashboard
uvx hotato sweep --demo --format json > hotato-sweep.json

# promote candidate #1 by its rank; you supply only the label
uvx hotato fixture promote hotato-sweep.json#1 \
    --expect yield --id refund-cutoff-001 --out tests/hotato

Starting from a single recording with no ranked result? fixture create takes the stereo file and an explicit onset instead:

uvx hotato fixture create --stereo call.wav --onset 42.18 \
    --expect yield --id refund-cutoff-001 --out tests/hotato

Either way, a candidate it cannot judge is refused with the reason (exit 2), never written as a fixture that would report a meaningless verdict. To vet a raw export first, hotato trust --stereo call.wav reports input-health only, never a verdict. The full walk from one clip to a CI gate, including --expect hold for a backchannel: The regression loop.

File the work in GitHub

Two commands hand the loop off to the workflow your team already lives in. Both are a dry run by default: they render the body and print the exact gh and git commands, and change nothing until --yes with an explicit --repo.

# file a sweep's ranked candidates as a triage issue
# (each carries the exact promote command for both a yield and a hold label)
uvx hotato issue create hotato-sweep.json --repo owner/repo --top 3

# open a PR that adds the fixtures you promoted
# (always a new feature branch, never the default branch, never a force-push)
uvx hotato pr create --fixtures tests/hotato --repo owner/repo \
    --title 'Add turn-taking regression fixtures'

4. loop: one command, with memory

hotato loop drives the parts Hotato can drive and remembers where it left off in a small local state file (.hotato/loop-state.json by default). It never runs ahead of you: it advances exactly one stage per call, and the next run picks up where the last one stopped.

# run 1: discovery (analyze -> scan -> rank) -> awaiting_label
uvx hotato loop ./recordings

# you label the candidates that matter
uvx hotato fixture create --stereo rec.wav --onset 12.4 \
    --expect yield --id refund-001 --out tests/hotato

# run 2: plans a guarded fix from the labeled fixtures -> awaiting_verify
uvx hotato loop ./recordings --fixtures tests/hotato

If every fixture you labeled already passes, loop plans nothing: it reports loop complete: all labeled fixtures already pass; there was nothing to fix. and stops, rather than inventing a fix for a battery that was never broken.

5. patch: the paste-ready artifact

Reads a fix plan (from hotato plan, schema hotato.fixplan.v1) and renders its abstract {field, from, to} recommendation into a literal artifact for the target stack. For Vapi and Retell, whose config lives behind a REST API, that means a JSON merge-patch body plus a ready curl against the platform’s own config-update endpoint, using fixmap’s verified knob field names.

uvx hotato plan result.json --stack vapi --assistant-id <id> --out fixplan.json
uvx hotato patch fixplan.json

hotato patch [vapi] finding=missed_real_interruption decision=propose_one_step
  config_patchable=true  applies_change=false
  change: stopSpeakingPlan.numWords  4 -> 3  (decrease, bounds [0, 10])
  apply method: rest-merge-patch
    curl (you run this; hotato does not):
      curl -X PATCH https://api.vapi.ai/assistant/<assistant-id> \
        -H "Authorization: Bearer $VAPI_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{"stopSpeakingPlan": {"numWords": 3}}'
  next: review the artifact above and apply it yourself; hotato never applies it
  next: re-capture the failing moment through your stack after applying it
  next: prove the movement across the battery: hotato verify --before before/ --after after/

For LiveKit and Pipecat, whose config lives in your agent source, patch emits the exact source edit instead (for example InterruptionOptions(min_words=1)), never a fabricated endpoint. For an unknown stack it names the knob family and asks for a concrete target; it emits no literal body it cannot stand behind.

hotato patch [vapi] finding=threshold_funnel decision=do_not_tune_single_threshold
  config_patchable=false  applies_change=false
  no patch: The plan refused single-threshold tuning: the battery misses a real
    interruption AND false-stops on a backchannel, so no one config value fixes
    both. No config patch is produced.
  recommended fix class: engagement-control
    This is a discrimination problem, not a threshold problem: telling a genuine
    bid for the floor apart from a backchannel or speech not addressed to the
    agent. No single timing threshold separates them.
  next: This is the both-axes case: no single config threshold fixes it, so no
    config patch is emitted. See the engagement-control pointer.
  next: Verify any change with a battery, not one clip: hotato verify --before
    before/ --after after/

patch makes no network call and pins applies_change to false. It produces the change; it never applies it.

apply --clone: the one guarded rung that can touch your platform

Every step so far is offline or read-only. apply is the single exception, and the most conservative rung: it can create external state, but only a fresh staging clone, never your source. By default it prints exactly the clone it would create and touches no network. Only --yes with credentials creates a new staging assistant: your config with the patch applied.

uvx hotato patch fixplan.json --format json --out patch.json

# dry run by default: prints the staging clone it WOULD create, changes nothing
uvx hotato apply patch.json --clone --name staging-refund-fix \
    --battery tests/hotato

6. verify: the before/after check

After you apply the change, in your own stack, and re-capture the same failing fixtures, verify scores the old and new run envelopes against each other. Fixtures pair by event_id, then scenario_id; each side is a single hotato run envelope or a directory of them.

# the failing take, before the change
uvx hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio \
    --format json > before.json

# ... apply the patch, re-capture the fixtures ...
uvx hotato run --scenarios tests/hotato/scenarios --audio tests/hotato/audio-new \
    --format json > after.json

uvx hotato verify --before before.json --after after.json

hotato verify: 5 fixtures paired (before -> after)
  3 of 3 that used to fail now pass; 2 of 2 hold fixtures still pass
  results: fixed=3, still_pass=2
  CLAIM: 3 of 3 fixtures that used to fail now pass, and 2 of 2 hold fixtures
    still pass. This improvement COINCIDES with your change; hotato measures
    timing and does not attribute cause.
  talk-over before: p95 2.10s (n=5)
  talk-over after: p95 0.35s (n=5)
  hotato reports coincidence, not causation.

It reuses the hotato compare taxonomy per fixture (fixed, regressed, improved, worse, unchanged, still_pass, not_scorable). The rollup is the two axes that matter: the regression axis, how many previously-failing fixtures now pass, and the hold axis, how many hold-labeled guard fixtures still pass.

• Coincidence, never causation. verify says the improvement coincides with your change, never that it caused it. Hotato measures timing; it does not run a controlled experiment.
• Refuses low-n claims. Below --min-n (default 3) the per-fixture facts still print, but the battery-scale headline is withheld and said so.
• Never invents a verdict. An unjudgeable side is not_scorable; a fixture on only one side is reported unpaired, never silently folded into the rollup.

By default verify measures and exits 0; --fail-on-regression exits 1 if any fixture regressed or got worse, for gating a rollout on the before/after check itself.

The anti-bandaid gate: --policy

--policy hotato.verify.yaml turns the same rollup into a pass/fail CI gate. A policy declares target.improve success criteria (a metric such as talk_over_sec_p95 or failed_count, with a required delta or a direction keyword) and hard guardrails (max_new_false_yields, max_not_scorable, require_hold_fixture, require_yield_fixture). verify exits 1 unless every guardrail holds AND every target is met, so a patch that cuts talk-over by yielding to everything still fails on max_new_false_yields.

uvx hotato verify --before before.json --after after.json \
    --policy hotato.verify.yaml