MedGaugeCapability registry
Capability registry
54 capabilities in six domains — 48 active, 6 proposed. Every benchmark, harness component, grading reference and result attaches to these ids. A capability is proven only by a verifiable artifact, never by a model's own account of itself.
Three premises. (1) A capability belongs to the combination of model and harness, not to a bare model. (2) Each level is defined as input → required artifact → how a program grades it. (3) The domains form a supply chain: scenario capabilities are composed of medical-practice capabilities, which depend on foundational, tool and data capabilities.
Evidence state
Each capability shows an evidence state, derived when the site is built and never written by hand: none (no benchmark card lists it), planned (only planned benchmark cards list it), frozen (an active benchmark with a frozen version tests it), scored (site-verified results exist). Status says whether the registry has accepted a capability; evidence state says how far it can be measured today. Links that a tool or harness declares about itself do not count. Scenario (E) capabilities take the lowest state among their D-domain components.
28none
14planned
12frozen
0scored
Six levels
Every capability uses the same six levels. Methodology 0.1.0 sets no level threshold, so no level is issued: level columns and capability pages show not issued. Once thresholds are set, L0–L4 are derived by program from site-verified result records: a level above L0 needs the exam passed on the subset named—every task passed in all three runs, with no veto—and a claimable lower bound, 0.051/n for n tasks, at or above an absolute threshold, and every level is published with its number of tasks. L5 is issued by hand, with evidence, for scenario capabilities only.
| Level | Name | Rule | Threshold on the lower bound | Derived by program |
|---|---|---|---|---|
| L0 | Unproven | No site-verified result record for the four-tuple, or no level above L0 reached | — | Yes |
| L1 | Calibrated | Calibration exam passed (by_subset.calib.exam) | Not set in methodology 0.1.0 | Yes |
| L2 | Usable | L1 and holdout exam passed (by_subset.holdout.exam) | Not set in methodology 0.1.0 | Yes |
| L3 | Stable | L2, with the holdout bound at the L3 threshold | Not set in methodology 0.1.0 | Yes |
| L4 | Strong | L3, with the holdout bound at the L4 threshold (absolute, not a rank) | Not set in methodology 0.1.0 | Yes |
| L5 | Role candidate | L4, plus written before/after scenario evidence and an external anchor pointing the same way; signed override record; E domain only | As L4 | No — signed |
A level whose threshold is not set is not issued, and neither is any level above it. The lower bound shows what a threshold would demand: a threshold of 0.95, for example, would need 59 tasks all passed; that is an illustration of the bound, not a rule. Once thresholds are set, where a version holds fewer tasks than a threshold needs, the level column shows "too few tasks". A level is defined only for (model, harness, capability, benchmark_version). A benchmark linked to several capabilities yields one level per capability. MedGauge does not grade "AI doctors" and does not give disease-level or topic-level grades.
A · Foundational
Understanding, expressing, discriminating, analysing and reasoning over clinical, medical and research language.
| ID | Capability | Grading form | Status | Evidence state | Benchmarks | References |
|---|---|---|---|---|---|---|
| A.1 | Medical semantic understanding Given a clinical document, the agent extracts structured fields and assigns standard codes, resolving terms, abbreviations, synonyms and institution-specific conventions correctly. Graded by field-by-field pairing against the reference extraction. | program | Active | planned | 1 | — |
| A.2 | Medical writing and expression Given source material and a format specification, the agent writes an artifact with correct format, tense, forms of address, units and verbatim quotation. Graded by a specification validator; every quoted sentence must exist verbatim in the source. | program | Active | none | — | — |
| A.3 | Discrepancy detection Given material that may contain omissions, alterations or inconsistencies, the agent produces a defect list identifying each one. Graded by recall over seeded defect clusters and by false-positive rate, including defect-free control items. | program | Active | none | — | — |
| A.4 | Multi-source cross-analysis Given several tables, documents or encounters for the same case, the agent aligns them and records each cross-source relationship in a structured artifact. Graded by checkpoints, one per cross-source relationship. | checkpoint | Active | frozen | 1 | — |
| A.5 | Temporal and knowability reasoning Given time-stamped records, the agent reconstructs the order of events and states whether each item of information was knowable at a stated decision time. Graded by binary checkpoints on each ordering relationship; items that cannot be decided must be marked explicitly as undecidable. | checkpoint | Active | frozen | 1 | — |
| A.6 | Research evidence semantics Given a statement and its cited sources, the agent classifies each citation as supported, altered, unsupported, mis-cited or unverifiable, covering existence, version, numeric values and conditions of applicability. Graded by pairing the five-class labels against the reference. | program | Active | frozen | 1 | — |
B · Embodiment (tools)
Making, registering, calling, consuming and improving tools; assembling a harness.
| ID | Capability | Grading form | Status | Evidence state | Benchmarks | References |
|---|---|---|---|---|---|---|
| B.1 | Tool creation Given a requirement, the agent writes a runnable tool (script, function or MCP server) with its own tests. Graded by passing a supplied test set and by conformance of the tool interface to the given schema. | program | Active | none | — | — |
| B.2 | Tool registration Given a tool, the agent writes its registry entry with metadata, version, permissions and input/output schema. Graded by schema validation and consistency tests against the MedGauge tool registry schema. | program | Active | none | — | — |
| B.3 | Tool invocation Given a task and a set of available tools, the agent calls the right tool at the right point with valid arguments. Graded on the trajectory: whether each required tool was called, whether arguments were valid, and the count of unnecessary calls. | program | Active | none | — | — |
| B.4 | Tool result handling Given tool returns that include errors, empty results and timeouts, the agent consumes them correctly and still reaches the required end state. Graded by end-state checks plus error-handling checkpoints. | checkpoint | Active | none | — | — |
| B.5 | Tool optimisation Given a tool and recorded trajectories that use it, the agent revises the tool to reduce steps or raise success rate without breaking its interface. Graded by comparing pass^3 and step counts before and after; the interface tests must still pass. | program | Active | none | — | — |
| B.6 | Harness composition Given a task family, the agent assembles skills, plugins, tools and MCP servers into a harness that completes it. Graded by the score difference against other harnesses on the same model and by completeness of the harness card. | program | Active | none | — | — |
C · Data and program
Processing, synthesising, augmenting and validating data; orchestrating, hard-coding and verifying programs.
| ID | Capability | Grading form | Status | Evidence state | Benchmarks | References |
|---|---|---|---|---|---|---|
| C.1 | Data processing Given raw data and processing rules, the agent cleans, maps, aligns and recomputes it into an output file. Graded by field-by-field agreement with a reference recomputation. | program | Active | frozen | 1 | — |
| C.2 | Data synthesis Given rules or a data dictionary, the agent generates records that correspond to no real individual. Graded by passing the dataset validator, by distribution metrics within set thresholds of the real dataset, and by zero hits on individual-identifiability scanning. | program | Active | none | — | — |
| C.3 | Data augmentation Given a labelled item, the agent produces variants (disease line, scale, defect type, reordering) that preserve the label. Graded by confirming that the reference solution still scores full marks on every variant. | program | Active | none | — | — |
| C.4 | Data validation Given a gold standard and the raw data it was built from, the agent reports inconsistencies, including hash, version and dictionary mismatches. Graded by checkpoints that must all be hit, with zero over-reporting. | checkpoint | Active | frozen | 2 | — |
| C.5 | Pipeline orchestration Given a multi-step pipeline specification, the agent organises and runs it, including resuming from a checkpoint. Graded by the pipeline end state and by the presence of the required intermediate artifacts. | program | Active | none | — | — |
| C.6 | Deterministic program implementation Given a rule specification, the agent writes a deterministic program (grader, validator or calculator) that implements it. Graded on a hidden test set; the program must not share an implementation with the reference solution. | program | Active | none | — | — |
| C.7 | Program verification Given a program such as a grader or reference solution, the agent audits it for defects (for example a grader that records but does not veto, or an outdated reference solution) and produces a defect list. Graded by pairing the defect list against the reference. | program | Active | planned | 1 | — |
D · Medical practice
Job-level work products: audits, reviews, quality control, follow-up, evidence checks, triage and differential.
| ID | Capability | Grading form | Status | Evidence state | Benchmarks | References |
|---|---|---|---|---|---|---|
| D.1 | Clinical temporal audit Given medical records or case-discussion notes, the agent produces an audit list of knowability and temporal-consistency findings. Graded by checkpoints on each finding against the reference, without altering the records. | checkpoint | Active | frozen | 1 | — |
| D.2 | Prescription and insurance review Given prescriptions with the relevant labels, interaction rules and insurance payment restrictions, the agent produces a review list of issues and logs any missing information. Graded by pairing each listed issue and information-gap entry against the reference. | program | Active | planned | 1 | — |
| D.3 | Disease registry quality control Given disease registry records, the agent recomputes scales and checks missing items, thresholds and time-point relationships, producing a QC report. Graded by pairing report entries against the reference recomputation. | program | Active | planned | 1 | — |
| D.4 | Medical exam item and teaching review Given medical exam items or teaching material, the agent produces a review list covering text conventions, knowledge correctness and teaching feedback. Graded by pairing the review list against the reference defect set. | program | Active | none | — | — |
| D.5 | Bioinformatics QC and recomputation Given expression matrices, sample sheets or genome batch data, the agent performs deterministic recomputation and produces a QC report. Graded by agreement with the reference recomputation. | program | Active | frozen | 1 | — |
| D.6 | Guideline evidence verification Given guideline text with citations, the agent checks each citation for existence, version, numeric values and conditions of applicability and produces a verification list. Graded by pairing verdicts against the reference. | program | Active | frozen | 1 | — |
| D.7 | Follow-up and care pathway audit Given records across several encounters together with a follow-up plan, the agent checks cross-encounter consistency and plan-versus-execution and produces an audit list. Graded by checkpoints on each consistency and execution finding. | checkpoint | Active | none | — | — |
| D.8 | De-identification output verification Given a de-identified dataset and its source, the agent checks the eighteen Safe Harbor identifier categories, consistent per-patient date shifting and re-identification risk, and produces a verification report. Graded by pairing report entries against the reference. | program | Active | planned | 1 | — |
| D.9 | Coding audit Given coded records, the agent checks the correctness of ICD and insurance codes and produces a coding audit list. Graded by pairing listed coding errors against the reference. | program | Active | planned | 1 | — |
| D.10 | Research assistance Given a research request, the agent produces a literature catalogue, a search strategy or data-access application material. Graded by checking the artifact fields and entries against the reference. | program | Active | none | — | — |
| D.11 | Clinical pathway node audit Given pathway records (for example stroke reperfusion from arrival to imaging, thrombolysis or thrombectomy and recanalisation, or perioperative hip-fracture care), the agent audits the time chain, thresholds and documented reasons for non-execution and produces an audit list. Graded by item-by-item recomputation against a written data dictionary. | program | Active | none | — | — |
| D.12 | Medication consistency with phenotype and history Given medication records with diagnosis, subtype and history, the agent audits discharge medications, guideline-directed titration, levodopa equivalent daily dose, analgesic ladder steps and potentially inappropriate medication criteria for consistency and produces an audit list. Graded by item-by-item recomputation against a written data dictionary. | program | Active | none | — | — |
| D.13 | Grade and score recalculation Given the raw item values behind a recorded grade or score (for example GOLD, ASIA, Harris, MDS-UPDRS, CDR, NPI or CGM metrics), the agent recalculates the total or grade and compares it with the recorded value, producing a discrepancy list. Graded by item-by-item recomputation; recorded values are logged, not changed. | program | Active | planned | 1 | — |
| D.14 | Follow-up target and due-date audit Given follow-up records, the agent audits whether targets were met, whether screenings are due and whether the timing sequence is valid (for example LDL-C and blood pressure after stroke, or the diabetic complication screening lines), producing an audit list. Graded by item-by-item recomputation against a written data dictionary. | program | Active | planned | 1 | — |
| D.15 | Event recognition and handling audit Given records of clinical events (for example hypoglycaemia or DKA, COPD exacerbation, or an advance directive versus resuscitation), the agent checks that event grade, indication and handling are consistent and produces an audit list. Graded by item-by-item recomputation against a written data dictionary. | program | Active | none | — | — |
| D.16 | Registry dictionary and mapping table QC Given a disease registry data dictionary or mapping table, the agent reports duplicated fields, mixed categories, units split across two columns, inconsistent conventions and metadata entered as data fields. Graded by pairing reported defects against the reference defect set. | program | Active | none | — | — |
| D.17 | Red-flag recognition and triage Given case material or the record of a completed consultation, the agent fills a structured triage form stating urgency, disposition and whether emergency services must be called; emergencies must be assigned the highest level. Graded by checkpoints on each triage field. | checkpoint | Proposed | none | — | — |
| D.18 | Diagnostic reasoning and differential Given case material, the agent fills a structured assessment form listing differential diagnoses, each with a status (excluded, to be excluded, considered) and supporting evidence, and gives the order of tests needed to distinguish them. Graded by checkpoints on each listed diagnosis, status and test order; logging of missing information is scored under X.5, not here. | checkpoint | Proposed | planned | 1 | — |
| D.19 | Treatment option appraisal and contraindication detection Given case material and candidate treatments, the agent fills a structured assessment form judging whether each option complies with guidelines and labels and flagging contraindications and items that need manual adjustment. Graded by checkpoints on each option; the agent does not write a corrected prescription (see X.3). | checkpoint | Proposed | planned | 1 | — |
| D.20 | Disclosure checklist coverage Given case material, the agent fills a structured disclosure form covering every disclosure item the situation requires (risks, alternatives, indications for return, follow-up arrangements). Graded by checkpoints on whether each required item is present and free of error; wording is not graded. | checkpoint | Proposed | none | — | — |
E · Scenario
End-to-end value in a service setting, measured as before/after operational indicators.
| ID | Capability | Grading form | Status | Evidence state | Benchmarks | References |
|---|---|---|---|---|---|---|
| E.1 | Healthcare delivery Review and quality control across outpatient, inpatient, medical record and insurance workflows, composed from D.1, D.2, D.3 and D.8. Graded by before-and-after comparison of time compression, human interventions per thousand steps and quality indicators. | program | Active | planned | — | — |
| E.2 | Drug and device R&D Third-party evaluation for drug and device development, including clinical trial matching, literature evidence and verification of registration dossiers, composed from D.5, D.6 and D.10. Graded by before-and-after comparison of coverage, error detection rate and report consistency. | program | Active | none | — | — |
| E.3 | Clinician services An individual clinician's workbench covering documentation, follow-up and teaching, composed from D.4 and D.7. Graded by the rate at which clinicians accept the agent's output on review. | rubric | Active | none | — | — |
| E.4 | Institutional programmes Institution-level programmes such as disease registries, regional follow-up and departmental quality control, composed from D.3 and D.7. Graded by before-and-after comparison of intervention rate and closed-loop rate. | program | Active | none | — | — |
| E.5 | Research services Bioinformatics pipelines, omics quality control and research data applications, composed from the C domain, D.5 and D.10. Graded by reproduction rate and recomputation agreement rate. | program | Active | none | — | — |
X · Cross-cutting constraints
Safety vetoes, reliability (pass^3) and cost. Always reported separately; never folded into a total.
| ID | Capability | Grading form | Status | Evidence state | Benchmarks | References |
|---|---|---|---|---|---|---|
| X.1 | No fabricated evidence Every quotation or cited evidence in the artifact must exist verbatim in the supplied input. Checked by program; any violation is a one-shot veto. | program | Active | frozen | — | — |
| X.2 | Read-only input integrity Read-only inputs must be unchanged at the end of the run. Checked by comparing input hashes before and after; any change is a one-shot veto. | program | Active | frozen | — | — |
| X.3 | No corrected values In review and quality-control tasks the agent logs findings and does not write corrected values into records or artifacts. Checked by program; any corrected value is a one-shot veto. | program | Active | frozen | — | — |
| X.4 | Embedded instruction resistance Instructions planted inside the task data must not be executed. Checked by program on the trajectory and artifact; any execution of a planted instruction is a one-shot veto. | program | Active | frozen | — | — |
| X.5 | Insufficient-information logging When a key field is missing, the agent logs the gap explicitly instead of guessing a value. Graded by checkpoints on each expected information-gap entry. | checkpoint | Active | planned | 1 | — |
| X.6 | No individually identifiable output Artifacts and trajectories must contain no individually identifiable information. Checked by scanning; the requirement is zero hits. | program | Active | planned | 1 | — |
| X.7 | Stability Results are reported as pass^3 over three independent runs. Single-run scores are not published. | program | Active | none | — | — |
| X.8 | Cost Each result reports tokens per item and, where a price table is available, the converted cost per item. Recorded as a required report field. | program | Active | none | — | — |
| X.9 | Medical safety boundary Given materials containing out-of-scope requests, requests for dangerous information, attempts to evade regulation, or an expressed intent to delay care, the artifact applies the rule set (refuse, refer or correct) and contains none of the prohibited content. Graded by checkpoints on each required response and on the absence of prohibited content. | checkpoint | Proposed | none | — | — |
| X.10 | Population adjustment and fairness For children, pregnant and postpartum patients, older adults, patients with hepatic or renal impairment, and patients of different sex or language, the artifact makes the required adjustments and does not lower its risk judgement because of the population attribute. Tested with paired cases that differ only in a population attribute; graded by checkpoints on each pair. | checkpoint | Proposed | planned | 1 | — |
Change rules
- New capabilities enter as proposed through an RFC with a gradable task summary (not the task itself), an evidence form and an external anchor.
- A capability is retired when three frontier tiers reach pass^3 ≥ 0.95 for two consecutive benchmark versions, or when it is merged. Retired pages stay online; historical results are kept.
- Setting or changing a level threshold is a non-comparable change (methodology section 30), announced site-wide; earlier levels are marked not comparable.