MedGaugeCapabilities / A · Foundational
A.3 Discrepancy detection Active
Given material that may contain omissions, alterations or inconsistencies, the agent produces a defect list identifying each one. Graded by recall over seeded defect clusters and by false-positive rate, including defect-free control items.
Evidence: none. No benchmark card lists this capability yet. No level is issued: methodology 0.1.0 sets no level thresholds.
- Domain
- A · Foundational
- Evidence form
- program
- Veto constraints
- —
- Depends on
- —
- Required by
- C.4C.7D.1D.16D.17D.2D.4D.6D.9
- Internal task families
- doc_audit, teach_review, evidence_check
- External anchors
- MEDEC, HealthBench
- Version
- 0.1.0 · 0.1.0
Definition in detail
Much of medical review work is finding what is wrong or missing in material someone else produced. An agent that over-reports wastes reviewer time; one that under-reports gives false assurance.
Inputs a task gives
- Documents or tables that may contain seeded omissions, alterations or internal contradictions
- The reference material or rules against which a discrepancy is defined
- Defect-free control items mixed with defective items
- A defect taxonomy listing the allowed defect types
Artifact required
A defect list with one entry per finding: location (document, field or line), defect_type (from the supplied taxonomy), evidence (verbatim excerpt) and description. An item with no defects yields an explicit empty list.
How it is graded
Seeded defects are grouped into clusters, so one underlying defect visible at several locations counts once. A program scores recall over clusters and the false-positive rate over entries that match no cluster. Control items must yield an empty list; any entry on a control item is a false positive. Evidence excerpts must exist verbatim in the input.
In scope
- Omissions, alterations and contradictions within the supplied material
- Classifying each finding with the supplied taxonomy
- Returning an empty list for defect-free items
Out of scope
- Writing a corrected version of the material - prohibited under X.3
- Domain review lists that add clinical or coding rules on top of detection - graded under D.1, D.2, D.4, D.6 and related D entries
- Defects in graders, validators or reference solutions - graded under C.7
- Inconsistencies between a gold standard and its raw data - graded under C.4
Typical failures
Expected failure patterns; not measured rates.
- Reporting acceptable stylistic or formatting variation as a defect
- Splitting one underlying defect into several entries, or merging distinct defects into one
- Declaring a control item defective on the assumption that every item contains an error
- Missing omissions, which leave no text to quote, while catching alterations
Benchmarks
No benchmark card lists this capability.
Grading references
No grading reference is mapped to this capability.