Reproducibility

Reproduction has a narrow meaning here: the same harness in another environment, on calibration tasks, with the reference solution at full marks. The difference in pass^3 between the two environments, env_delta, is reported with its sign and as an absolute value. Differences between harnesses are not a reproducibility failure — they are what MedGauge measures.

Fixed run conditions

Every result record fixes seven conditions; changing any of them makes a different record.

1. Model and version
2. Service snapshot or fingerprint returned by the interface
3. Reasoning setting
4. Runtime environment (container image digest)
5. Tool permissions and tool list
6. Harness id and version
7. Budget: step limit, output tokens, timeouts, token budget

Run records and re-grading

Each run writes a run record: model, interface, actions under the harness version's own definition and the step limit, tool-call counts, token use, timing, the amount and its price table version where amounts apply, temperature, system-prompt hash, whether the artifact exists, whether inputs were modified, checkpoints passed, veto categories, any run error with its attribution, and the grade. The stored artifact can be re-graded by the verifier at any time; re-grading must agree with the recorded grade on every checkpoint, or the result is not published. A result record carries the hash of its run records.

Recomputation

Every published number can be recomputed from the public record set. A result record carries every run of every task under an opaque task key, with its checkpoint counts, continuous score, veto categories and error attribution. Interval methods, the paired test and the letter display carry versions; bootstrap seeds and quantile rules are published, so that an interval can be recomputed digit for digit. A frozen benchmark version keeps the algorithm versions in force when it froze.

Run modes

Site-run (default)

The site runs the submitted model × harness on its own infrastructure. Tasks never leave the site.

Submitted-run re-grading

An enrolled team receives a task package under agreement and runs it; the site re-grades from the returned artifacts and run records. Task packages are watermarked and must be destroyed at the agreed date.

In both modes the score is computed by the site.

Adapter check

Before a harness's results are accepted, the reference solution must score full marks through it. A second-environment run on calibration tasks gives env_delta, reported with its sign and as an absolute value. A claim that env_delta is at most 0.05 is an equivalence claim; tested properly it needs some 309–734 tasks, which no benchmark version holds. Until a threshold is set, env_delta is reported and not judged; once one is set, a result above it is labelled not reproducible and is not withdrawn. Registered harnesses currently record the first check; second-environment measurements are pending and shown as such on each harness card.

External reproduction

External teams can reproduce calibration tasks in their own environment with a registered harness. A reproduction record lists organisation, harness, model, reference-solution result and env_delta with its sign; pass or fail is stated only once a threshold is set. None has been recorded yet.

Submitting results

Results enter the registry only as site-generated records via pull request, after review: reference solution full score, re-grading agreement on every checkpoint, every run present with its error attribution, env_delta recorded where a second environment was used, run records verifiable, amounts tied to a price table version, trajectory sample checked, canary check. Submission is not open yet.