Answer-key quality comes first

Evidence: verified Methodology 0.1.0 · external sources checked 2026-10-27 · what the grades mean

Label errors in the test sets of widely used benchmarks are pervasive and destabilise their rankings. MedGauge treats the key as the first thing to verify, and measures how often keys change.

What a public study found

A study of the test sets of widely used machine-learning benchmarks found label errors to be pervasive, and found that they destabilise the benchmarks’ rankings (Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, arXiv:2103.14749). When the key is wrong, the board measures agreement with the key’s errors as well as the ability it was built to measure.

What MedGauge does about it

Reference solution must score full marks through the same verifier subjects face.
Wrong artifacts must all fail, built by class rather than by hand-picked example.
Two independent expert keys per task. Two experts from different institutions each set the key; every disagreement is resolved and recorded, and their agreement is published per family version as Gwet’s AC1.
Independent acceptance: a reviewer who sees only the package and the checklist, not the builder’s notes.
Re-grading: every stored artifact is re-graded, and the result must agree with the recorded grade on every checkpoint, or the result is not published.
Grading references checked on the original: a reference is listed only once a frozen version uses it and its clause has been checked on the original source; only such a reference can be a veto.
Audit after every round, with the share of keys changed published.

What remains open

None of these steps proves a key correct. They make a wrong key less likely and easier to find, and they make the rate at which keys change visible. When an error is found after release, the task is corrected in the next benchmark version, and results across versions are marked not comparable. When a key or task fault is found during a round, that task version is voided for every subject and every subject is run again on the corrected version; the voided versions are listed with their attribution.