MedGaugeCapabilities / C · Data and program
C.4 Data validation Active
Given a gold standard and the raw data it was built from, the agent reports inconsistencies, including hash, version and dictionary mismatches. Graded by checkpoints that must all be hit, with zero over-reporting.
Evidence: frozen. A frozen benchmark version tests this capability. No site-verified result exists yet.
- Domain
- C · Data and program
- Evidence form
- checkpoint
- Veto constraints
- —
- Depends on
- A.3 Discrepancy detectionC.1 Data processing
- Required by
- D.16D.3D.5D.8E.5
- Internal task families
- traj_suite_audit, deid
- External anchors
- BixBench
- Version
- 0.1.0 · 0.1.0
Definition in detail
A benchmark or registry is only as trustworthy as the match between its gold standard and the raw data it came from. Hash, version and dictionary mismatches invalidate results without any visible error.
Inputs a task gives
- A gold standard
- The raw data the gold standard was derived from
- A manifest with file hashes and versions
- The data dictionary
Artifact required
An inconsistency report: item, inconsistency_type (hash, version, dictionary, value, count), expected, observed and evidence.
How it is graded
Each seeded inconsistency is a checkpoint and all checkpoints must be hit. Over-reporting must be zero: an entry that matches no reference inconsistency fails the item.
In scope
- Recomputing and comparing hashes
- Checking versions and dictionary conformance
- Comparing values and counts between gold standard and raw data
Out of scope
- Correcting the gold standard - prohibited under X.3
- Defects in graders or reference solutions - graded under C.7
- Domain QC of registry records and dictionaries - graded under D.3 and D.16
Typical failures
Expected failure patterns; not measured rates.
- Hashing a different byte representation than the manifest used
- Reporting a documented, expected version change as a defect
- Missing a dictionary mismatch because the values look plausible
Standards: std-croissant
Benchmarks
- Annotation QA Audit Planned
- Bioinformatics QC Active
Grading references
No grading reference is mapped to this capability.