Binary pass, continuous score, and how rows are ordered

Why MedGauge reports a binary pass and a continuous score for every run, orders rows by pass^3, and groups them by paired comparisons rather than by overlapping intervals.

Two numbers, two questions

Every run produces two numbers: a binary pass (did the artifact clear every gate and every checkpoint?) and a continuous score s, the share of the task’s checkpoints that pass, set to 0 when the run hits a veto. That is the only definition of the continuous score; no family defines its own.

The binary pass answers the question a department actually asks: could this artifact be accepted as it stands? A temporal audit that finds four of five defects still sends the fifth one through. The continuous score answers a different question: how much of it was right? It separates a run that missed one subtle item from a run that missed everything.

How they are aggregated

Task: the mean over its three runs. A run error is one of the three runs and counts as a failed run; it is never re-run or removed.
Family: the mean over its tasks, each task weighted equally.

There is no third step. No figure is averaged across families: the continuous score depends on how many checkpoints a task has, so each family’s score is its own scale.

Why rows are ordered by pass^3

The board’s primary key is pass^3, the share of holdout tasks passed in all three runs. Disagreement between repeated runs of the same subject is part of what is measured, and pass^3 is the figure that carries it. Among adjacent rows that carry an identical set of group letters, rows with an amount are ordered by cost per success. The continuous score is shown beside these figures; it is not the ordering key.

Whether the interval of the continuous score is narrower than that of the binary rate depends on the pass rate: in MedGauge’s own computation the ratio between the two reverses at low pass rates. MedGauge therefore makes no general claim that one interval is narrower than the other.

Groups instead of strict order

Two rows whose 95% intervals overlap are not thereby the same: “not distinguishable” defined by overlap is not transitive. In MedGauge’s own simulation with 13 tasks, two rows whose true values differ by 0.09 are merged 84% of the time. MedGauge instead compares every pair of rows on the paired difference over the same tasks, adjusts for the number of comparisons by Holm’s procedure, and shows each row with every group letter it belongs to. Rows sharing a letter are not shown to differ; groups can overlap, and a row can carry several letters.

What this does not replace

Neither number replaces the hard gates. A run that fabricates evidence fails regardless of its continuous score, and veto hits are reported in their own columns and marked on the row. Capability levels are to be thresholds on the lower bound of a passed exam, not on the continuous score; methodology 0.1.0 sets no threshold and issues no level.