How judging works

Methodology

The rules are published before the results. If you think a rule is wrong, say so — the workload is versioned and changeable, in the open.

1. 01

   Same fight

   Every contestant in a fight runs the same frozen workload — same prompt, same frozen sources, same output schema, same acceptance threshold. No contestant gets a friendlier question.

2. 02

   Repeats and medians

   Three repetitions per contestant by default. Reported metrics are medians across runs, never the single best run. One lucky run proves nothing.

3. 03

   Deterministic validation

   Validation is pure code. For the pricing workload, validators recompute every plan total from frozen sources, check the winner, reconcile the landed cost, and verify evidence citations. The model being judged is never consulted.

4. 04

   Mandatory tripwires

   Fabricated sources, unreconciled arithmetic, malformed output, missing evidence, or a wrong winner fail the run outright — no partial credit, no averaging away a fabrication.

5. 05

   Landed cost, honestly labelled

   Landed cost = direct API cost + schema-repair cost + retry cost + human-review cost + human-intervention cost + infrastructure cost. Every cost is labelled by basis: provider-reported, calculated, estimated, or manually entered.

6. 06

   Failures are evidence

   A contestant that fails every run still appears — with status, tripwires fired, and full evidence. Cherry-picked leaderboards are how buyers get burned.

7. 07

   Fixture vs real

   Fixture runs execute the entire platform with canned responses and zero spend. They prove the machine works. They are labelled fixture everywhere and are never benchmark evidence. Real publication requires explicit human confirmation of the roster and the paid run.

8. 08

   Spend controls

   Per-run, per-contestant, and per-suite budget ceilings with a 25% contingency allowance. Dry-run estimates precede any spend. The suite stops before crossing the ceiling.

9. 09

   Portable by design

   The lab is plain Python with pinned dependencies and a static site. It runs under uv or pip, in Docker, on any host. No Replit-specific services, no proprietary platform lock-in.