CI integration
A prompt tweak or a VAD threshold change ships clean through review, then callers start getting talked over again three weeks later. Wire Hotato into the PR gate and that regression fails the build instead of shipping.
Exit codes
| Code | Meaning |
|---|---|
0 | Passed (or --no-fail set). |
1 | An event failed. |
2 | Usage or I/O error: bad args, missing file, or a single recording that is not scorable (silent caller, or agent silent at onset). |
# fails the job on a turn-taking regression uvx hotato run --stereo call.wav --stack vapi --expect yield --format json echo $? # 0 green · 1 red
The GitHub PR check
Copy .github/workflows/hotato.yml into your repo at the same path. That's the whole setup: every PR gets a turn-taking gate and a running comment.
- Scores on every PR: installs the package and scores the suite to a JSON envelope.
- Posts one sticky comment: pass/fail, a per-scenario results table, a regressions section. Updates in place on every push; never spams the thread.
- Deltas against the base branch: any scenario overlapping more or yielding slower is listed with its delta. Best effort; the gate holds either way.
- Fails the job on a regression, read straight from the scored envelope.
exit_code: pass in green, a regression in red.Rendering is scripts/pr_comment.py, stdlib only, so you can preview the comment locally: hotato run --suite barge-in --format json | python3 scripts/pr_comment.py. See the workflow →
Gate on your agent, keep the plumbing
The bundled suite is a self-test of the harness. Replace one step, Score turn-taking (head), with your own capture and score. Envelope shape and exit codes stay identical, so the comment and gate stay too.
# play each corpus *.caller.wav into your agent, record its reply, then: hotato run --stereo your_call.wav --expect yield --format json --no-fail > head.json
Your own fixtures gate the same way: turn a bad call into a regression test →
Promote a candidate straight into the gate
A moment hotato sweep already ranked does not need a raw call and a hand-typed onset. hotato fixture promote takes the ranked result and your yield or hold label, and writes the same scenario and audio clip that hotato run --scenarios DIR --audio DIR scores on every push. A hotato scan candidate labels through hotato fixture create instead: copy its t_sec in as --onset.
# candidate #3 from a sweep result, labeled and pinned hotato fixture promote hotato-sweep.json#3 --expect yield \ --id refund-cutoff-001 --out tests/hotato
Candidate-to-fixture walkthrough, labels and thresholds included: The closed loop.
Re-scoring a frozen recording in CI is a check that the evidence, the policy, and the scorer are still intact: same audio in, same events out. It is not evidence that your live agent still behaves the same way today; that needs a fresh capture through Score turn-taking (head), not a re-score of yesterday's file.
Already running pytest?
One flag adds the same gate to the test run you already have: pytest --hotato-suite scores the battery after your tests and fails the session on a regression. Details: Pytest plugin.
Gate any CI with a failure contract
hotato contract create turns one labeled moment, from a hotato sweep candidate or a raw recording plus --onset, into a portable <id>.hotato bundle: the audio, the frame evidence, and a pass or fail policy, in one directory. hotato contract verify re-scores a directory of bundles against each one's own recorded policy, so the gate travels with the bundle instead of living in one repo's workflow file.
# create once, from a candidate a sweep already ranked hotato contract create --from-candidate hotato-sweep.json#1 --expect yield \ --id refund-cutoff-001 --out contracts # verify on every push, in any CI that reads JUnit XML hotato contract verify contracts/ --junit contracts-junit.xml
--junit writes one test case per contract, so any CI system that reads JUnit XML gates on it like any other test suite; no GitHub-specific wiring required. Exit codes: 0 every contract passes, 1 a contract regressed or is no longer scorable, 2 usage error, an empty directory, or an unreadable contract.json. Re-verifying bundled audio proves the evidence and the policy are intact, the same limit as above: it does not re-check today's live agent. Failure contracts, full mechanics →
Other CI systems
The workflow above is one wiring of a portable gate: install Hotato, verify any failure contracts, run any regression fixtures, then publish hotato.xml as JUnit so the CI shows per-contract pass or fail. Here is that same gate for four other systems. Each installs with python -m pip install hotato (swap in pipx run hotato or uvx hotato for a throwaway run), fails the job on a nonzero exit from hotato contract verify, and runs on push, on a pull or merge request, and on a weekly schedule (0 6 * * 1) to catch drift with no code change.
0 every contract passes, 1 a regression, 2 a usage or I/O error, matching the Exit codes table at the top of this page. Each job gates on a nonzero exit from hotato contract verify; the fixtures run and the JUnit publish report results but never mask that exit.
GitLab CI
A single job in a voice-gate stage, publishing hotato.xml through artifacts: reports: junit and running via rules on pushes, merge requests, and scheduled pipelines.
# install, gate on contracts, publish JUnit as a report stages: - voice-gate hotato-gate: stage: voice-gate image: python:3.12-slim script: # 1. install the CLI (or: pipx run hotato / uvx hotato) - python -m pip install hotato # 2. the gate: re-score failure contracts when any exist; nonzero here fails the job - | if ls contracts/*.hotato >/dev/null 2>&1; then hotato contract verify contracts --junit hotato.xml --format json > contracts-verify.json else echo "no contracts yet - add contracts/*.hotato to gate the build" fi # 3. run regression fixtures only when they are present - | if [ -n "$(ls -A fixtures/scenarios 2>/dev/null)" ]; then hotato run --scenarios fixtures/scenarios --audio fixtures/audio --format json > fixtures-run.json fi # 4. publish hotato.xml so GitLab shows per-contract pass/fail artifacts: when: always reports: junit: hotato.xml paths: - contracts-verify.json - fixtures-run.json # run on branch pushes, merge requests, and the weekly scheduled pipeline rules: - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - if: '$CI_PIPELINE_SOURCE == "push"' - if: '$CI_PIPELINE_SOURCE == "schedule"'
The weekly run is a pipeline schedule: add cron 0 6 * * 1 under CI/CD → Schedules, and the schedule rule above runs this job in it.
Jenkins
A declarative pipeline: one stage per step, a cron trigger for the weekly drift check, and junit in post so the build shows per-contract pass/fail.
// declarative pipeline, gates on hotato contract verify pipeline { agent any triggers { // weekly drift check, Mondays around 06:00 (H spreads load across the hour) cron('H 6 * * 1') } stages { stage('Install hotato') { // or: pipx run hotato / uvx hotato steps { sh 'python -m pip install hotato' } } stage('Verify contracts') { // the gate: nonzero from contract verify fails the build steps { sh ''' if ls contracts/*.hotato >/dev/null 2>&1; then hotato contract verify contracts --junit hotato.xml --format json > contracts-verify.json else echo "no contracts yet - add contracts/*.hotato to gate the build" fi ''' } } stage('Run fixtures') { // regression fixtures, only when present steps { sh ''' if [ -n "$(ls -A fixtures/scenarios 2>/dev/null)" ]; then hotato run --scenarios fixtures/scenarios --audio fixtures/audio --format json > fixtures-run.json fi ''' } } } post { always { // publish hotato.xml so Jenkins shows per-contract pass/fail junit allowEmptyResults: true, testResults: 'hotato.xml' archiveArtifacts artifacts: '*.json', allowEmptyArchive: true } } }
Push and pull request builds come from a Multibranch Pipeline or a webhook trigger; the cron trigger above adds the weekly drift run.
Azure DevOps
YAML steps on a hosted Ubuntu agent, publishing hotato.xml with PublishTestResults@2 in JUnit format and a schedules block for the weekly run.
# install, gate on contracts, publish JUnit results trigger: - main # run on pushes to main pr: - main # run on pull requests targeting main schedules: - cron: "0 6 * * 1" # weekly drift check, Mondays 06:00 UTC displayName: Weekly hotato drift check branches: include: [ main ] always: true # run even with no code change pool: vmImage: ubuntu-latest steps: # 1. install the CLI (or: pipx run hotato / uvx hotato) - script: python -m pip install hotato displayName: Install hotato # 2. the gate: re-score failure contracts when any exist; nonzero fails the step - script: | if ls contracts/*.hotato >/dev/null 2>&1; then hotato contract verify contracts --junit hotato.xml --format json > contracts-verify.json else echo "no contracts yet - add contracts/*.hotato to gate the build" fi displayName: Verify failure contracts # 3. run regression fixtures only when they are present - script: | if [ -n "$(ls -A fixtures/scenarios 2>/dev/null)" ]; then hotato run --scenarios fixtures/scenarios --audio fixtures/audio --format json > fixtures-run.json fi displayName: Run regression fixtures # 4. publish hotato.xml so Azure shows per-contract pass/fail - task: PublishTestResults@2 condition: always() inputs: testResultsFormat: JUnit testResultsFiles: hotato.xml failTaskOnFailedTests: false
The schedules block runs the pipeline weekly with no code change; trigger and pr cover pushes and pull requests.
CircleCI
One job in a voice-gate workflow. CircleCI reads a directory of JUnit XML, so the report is written under test-results/ and handed to store_test_results.
# install, gate on contracts, store JUnit results version: 2.1 jobs: hotato-gate: docker: - image: cimg/python:3.12 steps: - checkout # install the CLI (or: pipx run hotato / uvx hotato) - run: name: Install hotato command: python -m pip install hotato # the gate: re-score failure contracts when any exist; nonzero fails the job - run: name: Verify failure contracts command: | mkdir -p test-results if ls contracts/*.hotato >/dev/null 2>&1; then hotato contract verify contracts --junit test-results/hotato.xml --format json > contracts-verify.json else echo "no contracts yet - add contracts/*.hotato to gate the build" fi # run regression fixtures only when they are present - run: name: Run regression fixtures command: | if [ -n "$(ls -A fixtures/scenarios 2>/dev/null)" ]; then hotato run --scenarios fixtures/scenarios --audio fixtures/audio --format json > fixtures-run.json fi # store the JUnit report so CircleCI shows per-contract pass/fail - store_test_results: path: test-results - store_artifacts: path: contracts-verify.json workflows: voice-gate: jobs: - hotato-gate
The weekly run is a Scheduled Pipeline: add one with cron 0 6 * * 1 in the project settings so this workflow runs with no code change. Push and pull request builds run automatically once the project is set up.
Richer artifacts on a red build
hotato report renders the visual report (attach it as a CI artifact), and hotato team turns a directory of nightly envelopes into a trend.
Keeping a run green on purpose
--no-fail always exits 0 (reports, doesn't fail). --max-talk-over / --max-time-to-yield set pass/fail bounds per recording.