The model judge, quarantined by design.

Every verdict is advisory, cached with full provenance, and walled off from the deterministic gate.

rubric.v1 · acknowledged-frustration

$ hotato rubric run --rubrics rubric.yaml --transcript t.txt
{
  "schema": "rubric.v1",
  "deterministic": false,
  "verdict": "INCONCLUSIVE",
  "provider": "ollama",
  "model": "qwen2.5vl:3b",
  "model_digest": "sha256:7b1c…e04a",
  "prompt_sha256": "9f2adb…41c",
  "temperature": 0,
  "input_sha256": "c41d8c…9e2",
  "votes": ["pass", "fail", "inconclusive"],
  "disagreement": true,
  "confidence": 0.33,
  "confidence_required": 0.85,
  "review": { "human_required": true }
}
# exit 0 (advisory): verdicts are reported, never gate

Run locally at http://localhost:11434, zero egress. The votes disagreed, so the judge abstained and flagged the case for a person.

Two lanes that never touch.

Seventeen deterministic assertion kinds on one wall; a model verdict cannot become a deterministic result.

Advisory is the default: a rubric FAIL is reported and gates nothing, unless a team opts in with --gate. Even then, INCONCLUSIVE never gates: an abstention cannot fail a build.

one report, two shelves

assert.v1 · the deterministic walldeterministic:true · no model

phrase
policy
tool_call
outcome
+ 13 more

kept apart on purpose

rubric.v1 · the advisory shelfdeterministic:false · local model

model judge · advisory

It abstains, then defers to a human.

The deterministic lane decides the verdict; a judge that cannot agree with itself says so and sends the case to review.

rubric.v1 · scorecard · support-bot

rubric.v1 · agent support-bot · call 8f2a

Outcomerefund posted (trace span)deterministicpass

Policyrequired disclosure readdeterministicpass

advisory shelf · never gates the verdict above

Model judgeacknowledged the caller's frustration?advisoryinconclusive → review

The advisory shelf never changes the deterministic verdict.

The model's opinion is visible, attributed, and boxed off from the decision. The deterministic lane →

Refused off-box. Never scored by a person’s call.

The default judge never leaves the machine, and a rubric marked for a human is never asked of a model at all.

Point the judge at a non-local endpoint and the run is refused until --judge-egress-opt-in makes the trade-off explicit. Mark a rubric kind: human_rubric and it stays INCONCLUSIVE until a person signs the record; an overall_score is rejected at load.

Run the judge in your own environment →

terminal

$ hotato rubric run --judge-provider hosted --judge-endpoint https://api.example.com/v1
refused: judge egress requires --judge-egress-opt-in (exit 2)

$ hotato rubric run --rubrics rubric.yaml  # kind: human_rubric
acknowledged-frustration   INCONCLUSIVE   human_required: true
overall_score                              rejected at load

Refusal is the default. No blended score, ever, on either lane.

A judge that shows its work, and knows its place.

One local model, full provenance, and a lane that never gates without a person.

$ uvx hotato start --demo

One recording is enough to start. Get started → · Self-host →