The harness is part of the measurement

Why every MedGauge result names a harness, and what we expect to learn by measuring the same model under different harnesses.

The observation

Studies on agentic coding benchmarks report that changing only the harness — the tools, limits, prompt and loop around a model — can move the same model’s score by many percentage points. The paper Stop Comparing LLM Agents Without Disclosing the Harness (arXiv 2605.23950) argues that reports should disclose the harness for exactly this reason.

What MedGauge does

A result row is always a model inside a registered harness version: the same model under two harnesses is two subjects, two rows, and the gap between them is reported as a measured quantity. Each harness counts actions by its own definition, so action counts and redundancy rates are compared only within one harness version; across harnesses they are reported, not ranked. Reproduction is defined narrowly: the same harness in a second environment, on calibration tasks, with the difference in pass^3 reported with its sign and as an absolute value.

What we expect, and have not yet tested

Hypothesis 1. On medical audit tasks the harness effect is at least as large as on coding tasks. Audit artifacts are long lists where one extra or missing item fails the run, so step limits and tool access should matter a great deal.
Hypothesis 2. Harnesses that encourage the model to re-read inputs before writing will reduce fabricated evidence more than they raise recall.
Hypothesis 3. Step limits interact with over-reporting: very low limits produce incomplete artifacts, while unbounded runs produce more speculative findings.

These are statements to test, not results. Testing them requires running the same model under at least two registered harnesses with fixed conditions, compared on the paired difference over the same tasks. A large effect needs some 30 tasks to show; an effect of 0.075 needs about 200.