MedGaugeHarness and tool registries
Harness registry
A harness is the layer that puts a model to work: runtime, tools, limits, sandbox and prompt. On MedGauge a capability belongs to a model inside a harness, so every result names a registered harness version, and the same model under two harnesses is reported as two rows.
Published studies show why: changing only the harness moves the same model's score by several to more than twenty percentage points on agentic coding benchmarks (see "Stop Comparing LLM Agents Without Disclosing the Harness", arXiv 2605.23950). MedGauge treats that gap as something to measure, not as a reproducibility failure.
Registered harnesses
| Harness | Kind | Interface | Tools | Step limit | Sandbox | Autonomy | Status |
|---|---|---|---|---|---|---|---|
| Early executor (archived) [email protected] | API tool loop | Other | — | 40 | container, no network, readonly inputs | A3 | Deprecated |
| MedGauge reference driver (containerised tool loop) [email protected] | API tool loop | Chat-completions-compatible API · Messages-style API | 6 | 100 | container, no network, readonly inputs | A3 | Active |
What a harness card must state
- Base runtime and version; one file per version, never edited after publication.
- Model slots, tool components (each pinned to a tool registry version), system-prompt hash if the prompt is not public.
- Limits (steps, output tokens, temperature, timeout) and sandbox (container, network, filesystem).
- Autonomy class: A1 assistant, A2 co-pilot, A3 executes with run records, A4 executes and self-verifies. The A scale describes the harness; it is separate from the capability levels L0–L5.
- Reproduction command and adapter check: the reference solution scores full marks through the harness, and the same harness in a second environment gives
env_delta(difference in pass^3 on calibration tasks), reported with its sign and as an absolute value and not judged until a threshold is set.
Tool registry
Tools, skills, plugins and MCP servers that harnesses use. Each entry records its protocol, medical scope, safety profile and conformance testing.
| Tool | Kind | Summary | Side effects | Conformance | Status |
|---|---|---|---|---|---|
| [email protected] | tool · function | Replace an exact string in a workspace file. | writes | schema valid · smoke test not run | Active |
| [email protected] | tool · function | List workspace files matching a glob pattern. | none | schema valid · smoke test not run | Active |
| [email protected] | tool · function | Search workspace files for a regular expression. | none | schema valid · smoke test not run | Active |
| [email protected] | tool · function | Read a file from the task workspace. Task inputs are mounted read-only. | none | schema valid · smoke test not run | Active |
| [email protected] | tool · function | Run a shell command inside the task container (no network; fixed toolchain; per-command timeout). | writes | schema valid · smoke test not run | Active |
| [email protected] | tool · function | Write a file under the task workspace, typically the submission artifact. | writes | schema valid · smoke test not run | Active |
The six task-environment tools are function tools defined by the reference driver; their source is held in the private grading repository.
Registration
Harness and tool registration will work as pull requests validated by the same schemas used on this site. Applications are not open yet.