MedGaugeMedical AI agent evaluation
MedGauge
A closed-set, open-method benchmark and harness registry for medical AI agents. Tasks are modelled on real medical work — record audits, evidence checks, bioinformatics QC, with prescription review and registry QC planned — and graded by program against the task rules supplied with each task and, where a task relies on an external clause, a registered grading reference.
Counts are computed from the registries when the site is built; none are typed by hand.
First holdout round in preparation. The leaderboard lists site-verified results only. Rows appear when the first reviewed result records are merged.
What MedGauge measures, and how
Artifact tasks, graded by program
Every task gives static materials and requires a structured artifact. A program grades the end state. No judge model, no conversation rubric.
Grading references
Where a checkpoint rests on an external clause, the clause is registered with its locator and the rule applied. Every reference is checked on the original source before a frozen version uses it.
Expert key cards
Each task is to carry an expert key card prepared with clinical or domain experts: the expected key, what counts as over-reporting, and why. Card records are held privately and referenced from the first holdout round.
Closed set, open method
Methods, scoring rules and schemas are public; hash manifests are published with the first holdout round. Tasks, answers, environments and trajectories are not.
Model × harness rows
A leaderboard row is a model running inside a registered harness. The same model under two harnesses is two rows.
Safety constraints in their own column
Fabricated evidence, altered inputs, corrected values and followed injections are vetoes. They are reported separately and never averaged away.
Conflicting guidelines side by side
Where regional standards disagree, a grading reference names both sources, and a task must state the regional standard it is graded against.
Where to start
Model and harness teams
See how results are recorded and what a row means.
Researchers
Benchmark cards, the capability registry and the schemas are citable and machine-readable.
Evaluation institutions
Evaluation rounds with fixed conditions and auditable records. Applications are not open yet.
Data users
Access tiers, isolation rules and de-identification method for data that serves registered capabilities.
How to read a MedGauge score
- Row = model × harness × benchmark version. Different benchmark versions are separate boards and are not comparable.
- pass@3 and pass^3 together. pass@3: at least one of three runs passes. pass^3: all three pass. Single-run scores are not published.
- Vetoes are separate. Hits on X.1–X.4, unsafe advice, and veto-flagged grading references fail the task and are listed in their own column.
- No AI-doctor grades. Levels exist only for (model, harness, capability, benchmark version). No page rates a model "for a disease".