# hotato

> Open-source, self-hosted conversation QA for voice agents. Test every
> conversation your agent must handle: verify task outcomes and policy, catch
> voice failures, simulate calls at scale, and keep every result reproducible on
> your own infrastructure. A run yields a per-dimension scorecard across five
> dimensions -- Outcome, Policy, Conversation, Speech, Reliability -- never one
> blended score. Deterministic checks and the advisory model judge report on
> separate lanes, so a model opinion can never stand in for a measurement.
> Turn-taking timing (talk-over, time-to-yield) is one deterministic proof-point
> inside the Conversation dimension, not the whole product. Offline by default;
> the only egress is a command whose job is to fetch from your own provider.
> 17 deterministic assertion kinds, 5 dimensions, 5 evidence tiers, a 5-view
> workspace, 0 runtime dependencies, 3,544 tests in CI, MIT. Callable by an AI
> agent as a CLI or over MCP.
> Version 1.7.0. This file is the machine-readable index; `hotato describe
> --format json` emits the same command surface generated straight from the
> CLI's own argparse structure (every subcommand, its args, its exit codes, plus
> the schema URLs and the version), kept in sync with the flags by regenerating
> it from the parser.

## Start here (zero-install)

- `uvx hotato start --demo` -- the zero-setup guided first run on bundled
  recordings; no account, no keys, no network. Start here.
- `hotato test run conversation-test.yaml --agent <id>` -- run a conversation
  test and get the per-dimension scorecard plus a bound conversation artifact.
- `hotato sweep --demo` -- connect, pull recent calls, and analyze them into one
  self-contained HTML dashboard (swap for `--stack vapi --since 7d` on your own).
- Self-host the workspace: `docker compose up -d`, then open
  http://127.0.0.1:8321.

## Every command (byte-runnable; the console script is `hotato`; `--format json` is for machines)

- `hotato test run conversation-test.yaml --agent <id>` -- run a conversation
  test; emits the per-dimension scorecard and a conversation artifact. Flags:
  `--transcript`, `--trace`, `--state`, `--audio WAV`, `--repetitions N`,
  `--out DIR`, `--format {text,html,md,json}`, `--judge-model`,
  `--judge-provider {ollama,hosted}`, `--gate-judge`, `--no-store`.
- `hotato assert init --from-trace trace.jsonl` and
  `hotato assert run --assertions assertions.yaml` -- the deterministic,
  no-model assertion DSL (the 17 kinds below). Flags: `--stereo`,
  `--transcript`, `--transcribe`, `--trace`,
  `--inconclusive-policy {report,fail,refuse}`. Exit 0/1/2.
- `hotato simulate scenario.yaml --out runs/` -- render a scenario into
  deterministic simulated conversation artifacts (scripted / state-machine
  callers; per-run seeds from sha256; no model, no TTS, no network; every
  artifact stamped `origin=simulated`). Flags: `--matrix scenario.yaml`,
  `--conversation-test TEST.yaml`, `--seed N`, `--repetitions N`,
  `--parallel N`, `--agent ID`.
- `hotato rubric run --rubrics rubric.yaml --transcript t.txt` -- the advisory
  model-judge lane; every result is `deterministic:false` with full provenance.
  Add `--gate` to let it gate, `--sign` to sign the result. The default judge is
  a local Ollama model; a hosted judge is refused without `--judge-egress-opt-in`.
  `hotato rubric calibrate --labeled corpus/` calibrates the judge against a
  human-labeled held-out set.
- `hotato serve` -- the self-hosted local team workspace, read-only, binds
  127.0.0.1, default `--port 8321`. Flags: `-w`/`--workspace ID`, `--host`,
  `--port`, `--registry PATH` (default `~/.hotato/fleet`), `--token`,
  `--token-file`. Five views: Release readiness, Scenario matrix, Conversation
  inspector, Failure clusters, Production health.
- `hotato contract verify contracts/ --junit contracts.xml` -- re-score portable
  CI contracts. `--junit` takes a PATH; also `--html PATH`,
  `--format {text,json}`. Exit 0/1/2.
- `hotato conversation verify <dir>` -- re-hash a conversation artifact's bound
  children (audio, transcript, trace, evals) and refuse on tamper.
- `hotato investigate <call.wav>` (or
  `hotato investigate --stack vapi --call-id <id>`) -- authenticated capture ->
  channel-trust gate -> ranked candidate moments you then label. Follow with
  `hotato investigate label <STATE#N> --expect yield` to turn a labeled moment
  into a signed, CI-ready contract.
- `hotato sweep --demo` (or `hotato sweep --stack vapi --since 7d`) --
  connect -> pull -> analyze recent calls into one self-contained HTML dashboard.
- `hotato capture --stack vapi --call-id <id>` (also `--demo`) -- fetch and score
  one recorded call. `hotato pull --stack vapi --since 7d` -- bulk-fetch your
  recent recordings into a folder.
- `hotato connect vapi` -- store a stack's credentials once at file mode 0600
  (no network round-trip on store), sent only to the vendor's own API, never to
  Hotato.
- `hotato describe [--format json]` -- the generated capability manifest of the
  whole CLI: every subcommand's name, purpose, arguments, and documented exit
  codes, plus the version and the schema URLs. Deterministic; generated straight
  from the parser.
- Transcription is a FLAG, not a command: `--transcribe` (with
  `--transcribe-model base.en`) on `run`, `assert run`, and `test run`, gated by
  the `[transcribe]` extra (faster-whisper, local). There is NO
  `hotato transcribe` command.
- MCP server: `hotato-mcp` (install the `[mcp]` extra); copy-paste client
  configs live in `docs/mcp.html`.
- Self-host: `docker compose up -d` -> http://127.0.0.1:8321. Add the local
  Ollama judge with `docker compose --profile judge up -d`; confirm no egress
  with `deploy/verify-zero-egress.sh`.

## The 17 deterministic assertion kinds (exact strings; all `deterministic:true`, no model)

phrase, pii, policy, entity_accuracy, tool_call, tool_result, tool_error,
handoff, dtmf, termination, state, state_change, latency, timing_contract,
sequence, count, outcome.

- Transcript-reading: phrase, pii, policy, entity_accuracy. The pii check never
  echoes the matched value (detectors: ssn, card_luhn, email, phone).
- Trace spans (the authenticated voice trace, never the transcript): tool_call,
  tool_result, tool_error, handoff, dtmf, termination.
- State adapter (the post-call system of record): state, state_change.
- Timing: latency, timing_contract. Ordered / counted: sequence, count.
  Composite: outcome.
- `--inconclusive-policy` is one of {report, fail, refuse} (default report):
  refuse exits 2; fail gates like a FAIL (exit 1).

## What a run reports

- A per-dimension scorecard across the five dimensions, each answered on its own
  terms: Outcome (did the task succeed, grounded in tool and state evidence),
  Policy (did it follow the workflow and rules), Conversation (did it listen,
  take turns, clarify, escalate -- turn-taking timing lives here), Speech
  (transcript, latency, audio delivery), Reliability (does it hold across
  repetitions, scenarios, and releases: pass_at_1, pass_at_k, and pass^k held
  across all k runs, with a Wilson confidence interval). The five are a grouping
  key, never a weight -- there is no blended score anywhere.
- Outcome state has three authorities: authenticated trace spans, the post-call
  state adapter (an absent adapter yields INCONCLUSIVE), and human-attested,
  Ed25519-signed label records. A model verdict lives only in the advisory rubric
  lane and can never stand in for one of them.
- The machine-readable result is a JSON envelope (schema
  `https://hotato.dev/schema/envelope.v1.json`, `schema_version` "1",
  additive-only). Structured errors share one shape (schema
  `https://hotato.dev/schema/error.v1.json`, `ok:false`, a stable `error_code`,
  and an actionable `message`) -- never a raw uncaught exception.
- Exit codes: `0` all pass, `1` a regression, `2` usage or IO error (or a single
  input that is not scorable). Not scorable means the recording cannot answer the
  question, so no verdict is asserted -- `scorable:false` with a plain
  `not_scorable_reason`, never a fake pass or fail.

## Evidence kernel (5 ascending tiers; how strong the evidence is)

- The five tiers: none -> asserted -> measured -> paired -> attested. measured =
  recomputed from one clean recording; paired = a manifest-bound before/after,
  both recomputed under one pinned policy; attested = paired plus a signed
  policy/contract and an attested capture origin (Ed25519).
- The public tier is the WEAKEST required dimension (a minimum, never blended or
  averaged); uncertainty only pulls it down. A tier is earned only when a
  signature verifies -- never from a field merely being present.
- Signing is Ed25519, opt-in via the `[sign]` extra; a verifier needs only the
  public key. Altered audio, a tampered manifest, or the wrong key is REFUSED
  (tier none), not quietly downgraded.

## Self-host, offline, and sovereignty

- `docker compose up -d` -> http://127.0.0.1:8321. Zero runtime dependencies,
  offline by default, no telemetry, no phone-home. Credentials sit at file mode
  0600. The workspace is stdlib-only (http.server + sqlite3), bearer-token auth,
  a SQLite registry plus a content-addressed evidence store, no per-agent cap.
- Egress happens only when it is the command's job: capture / pull / sweep fetch
  from YOUR provider; a hosted judge or diarizer is refused without an explicit
  `--egress-opt-in`. `deploy/verify-zero-egress.sh` confirms the workspace still
  serves with the network unplugged.
- Air-gap with `docker save` / `docker load` (seed the judge model first); back
  up by tarring the single `/data` volume. Your conversation artifacts,
  contracts, and dashboards move between self-hosted and cloud without changing a
  line.
- No SOC2 / HIPAA / ISO / PCI certification is claimed; the answer is audit the
  code, or run it in your own VPC.

## Integrations (per-stack capability matrix)

- CAPTURE (score a recorded call) covers all five stacks: Vapi, Twilio, Retell,
  LiveKit, Pipecat. Vapi / Twilio / Retell auto-pull the recording with YOUR API
  key (Retell by explicit `--call-id` only). LiveKit / Pipecat capture in your
  own infrastructure -- your deployment writes the dual-channel WAV and Hotato
  scores the local file (extras `[livekit]` / `[pipecat]` for the capture
  helpers).
- DRIVE a live (billed) call is Vapi and Twilio only, double-gated behind stored
  credentials AND `HOTATO_DRIVE_OPT_IN=1`. Retell / LiveKit / Pipecat are
  capture-only.
- Per-stack pages: /integrations/vapi, /integrations/twilio,
  /integrations/livekit, /integrations/pipecat, /integrations/retell.

## Scope and method (each boundary is a property of how it works)

- The timing detector reads audio energy over time and nothing else: it cannot
  tell who spoke or what was said. No speaker identification, no diarization
  scoring, no transcription grading, no emotion or intent inference.
- Two separate channels make overlap a fact of the recording. On a mono mix the
  scorer needs a caller-onset label and marks its overlap numbers as estimates.
- Transcription (`--transcribe`) supplies context for the transcript-reading
  assertions; it is never graded as a capability of its own.
- The advisory model judge is an opinion, gated off by default; INCONCLUSIVE
  never gates even when `--gate` is set, and a human rubric is never
  model-scored.
- Every number is a measurement against a documented, overridable threshold, each
  with an explicit ceiling; the envelope reports reproducible measurements, never
  a blended figure.

## Site map (clean URLs)

- / -- the homepage: the five-dimension scorecard, the command surface, the
  evidence kernel.
- /success -- the Outcome and Policy dimensions: did the task succeed and follow
  the rules, grounded in trace and state.
- /timing -- the Conversation dimension's turn-taking proof-point: talk-over,
  time-to-yield, barge-in, and the finding-maturity ladder.
- /transcript -- the Speech dimension: the transcript-reading assertions,
  latency, and audio delivery.
- /rubric -- the advisory model-judge lane and its provenance.
- /simulation -- deterministic simulated conversations and reliability (pass^k).
- /dashboards -- the self-hosted workspace and its five views.
- /self-host -- docker compose, air-gap, backup, and the zero-migration promise.
- /security -- offline default, credential handling, signing, and the evidence
  tiers (trust is folded in here).
- /compare -- where Hotato fits alongside other approaches.
- /changelog -- dated releases backed by commits.
- /about -- the maintainer (Attention Labs, Inc.) and the project's posture.
- /integrations/ -- the per-stack capture and drive matrix, plus a page per
  stack (/integrations/{vapi,twilio,livekit,pipecat,retell}).
- /docs/ -- the deep documentation index (getting-started, how-it-works, capture
  per stack, ci, pytest, mcp, report, team, export, suites, closed-loop,
  security).
- /validation -- the evidence gallery captured from a pinned build.
- /corpus -- the tiered corpus suites and the additive scenario classes.
- /real-calls -- example calls recorded during operator sessions, shown with the
  report Hotato writes for each.

## Files

- /docs/ -- the deep documentation (indexed above).
- Schemas: /schema/envelope.v1.json (the result envelope) and
  /schema/error.v1.json (the shared error shape).
- /llms.txt -- this index. /sitemap.xml -- every page.
- /LICENSE -- MIT; the open core stays MIT.