hotatodocs
Home Security Changelog Docs GitHub
Home/Docs/Pytest plugin
Integrate

The pytest plugin

Two ways a turn-taking regression should show up in a test run: as one assertion inside a test you already wrote, or as a whole-session gate you didn't have to write at all. Install hotato and the plugin registers both, no extra config.

your test hotato_score(...) assert on envelope pytest --hotato-suite battery, after tests session failson regression
The fixture is for measurements your own test cares about; --hotato-suite is a session-wide gate you opt into once and never write again.
bash
pip install hotato   # the plugin auto-registers (pytest11 entry point)

The hotato_score fixture

Same inputs as the CLI, returns the same envelope. Assert on whichever measurement your test cares about.

test_turn_taking.py
def test_call_yields(hotato_score):
    env = hotato_score(stereo="call.wav", expect="yield")
    assert env["exit_code"] == 0
    assert all(e["verdict"]["passed"] for e in env["events"])

def test_split_channels(hotato_score):
    env = hotato_score(caller="caller.wav", agent="agent.wav", expect="yield")
    assert env["events"][0]["verdict"]["passed"]

def test_selftest_battery(hotato_score):
    env = hotato_score(suite="barge-in")
    assert env["exit_code"] == 0

hotato_score(**kwargs) takes the same keyword arguments as the CLI: stack names the voice stack for fix wording, max_talk_over_sec and max_time_to_yield_sec tighten a bound for one test without touching the CLI default.

The --hotato-suite session gate

Opt in on the command line and the battery runs after your tests, printing a summary and failing the session (exit 1) on a regression.

terminal
$ pytest --hotato-suite -q
.                                                     [100%]
================== hotato suite: barge-in ==================
8 of 8 events pass (failed=0)
1 passed in 0.54s

On a regression the gate is explicit:

terminal
FAIL svd-st-missed: fix[config] expected the agent to yield but it kept talking
hotato: regression detected; failing the session.

Gate on your own labelled sets

The bundled battery is a self-test of the harness. Point the same flag at your own scenario and audio directories, including the tiered suites:

bash
pytest --hotato-suite \
  --hotato-suite-scenarios corpus/suites/gold/scenarios \
  --hotato-suite-audio corpus/suites/gold/audio
Insufficient coverage is not a regression

An empty or wrong --hotato-suite-audio path leaves every scenario not-scorable: no caller onset detected, or the agent already silent where a should-yield case needs it talking. The gate still fails the session (scoring nothing is a setup failure), but the summary says so directly: hotato: no scorable events, never regression detected. A missing file is an input problem, not a measured miss.

One flag, both gates

If CI already runs pytest, this flag is the whole integration. Prefer a PR comment with the results table? The GitHub workflow posts one and gates on the same envelope. Use either, or both.

A regression here means a frozen battery scenario that should pass didn't: a config change, a prompt edit, or an upstream dependency bump to go look at, not a diagnosis of what broke.