MedGaugeCapabilities / C · Data and program
C.7 Program verification Active
Given a program such as a grader or reference solution, the agent audits it for defects (for example a grader that records but does not veto, or an outdated reference solution) and produces a defect list. Graded by pairing the defect list against the reference.
Evidence: planned. Only planned benchmark cards list this capability so far; no frozen benchmark version exists for it.
- Domain
- C · Data and program
- Evidence form
- program
- Veto constraints
- —
- Depends on
- A.3 Discrepancy detectionC.6 Deterministic program implementation
- Required by
- E.5
- Internal task families
- traj_suite_audit
- External anchors
- —
- Version
- 0.1.0 · 0.1.0
Definition in detail
A grader that records a violation but does not veto, or a reference solution built on an outdated rule, silently corrupts every score computed with it.
Inputs a task gives
- The program under audit, such as a grader, validator or reference solution
- Its rule specification
- Sample inputs and outputs
- Version history of the rules
Artifact required
A defect list: location, defect_type (for example records but does not veto, outdated rule, boundary error, unreachable branch), a triggering input, and expected versus observed behaviour.
How it is graded
A program pairs defect entries with the reference defect set, scoring recall and false positives. Triggering inputs may be executed to confirm that the stated behaviour occurs.
In scope
- Semantic defects where code runs but enforces the wrong rule
- Boundary, version and veto-logic defects
- Providing a triggering input for each defect
Out of scope
- Writing the fixed program - not part of this capability; X.3 applies in review tasks
- Inconsistencies in data rather than code - graded under C.4
Typical failures
Expected failure patterns; not measured rates.
- Reporting style issues as defects
- Missing a defect because the program runs without error
- Stating a defect without an input that triggers it
Benchmarks
- Annotation QA Audit Planned
Grading references
No grading reference is mapped to this capability.