Benchmarks / bm-guideline-evidence-check

Guideline Evidence Check Active

A guideline-writing secretary or evidence pharmacist checks each citation in a draft against a local guideline corpus: does the cited passage exist, is the version right, do the numbers and conditions of use match.

Capabilities: D.6 Guideline evidence verification A.6 Research evidence semantics
Source groups: guideline-citation-check
Task types: Item-by-item judgement
External anchors: HealthBench

Versions

Version	Frozen	dev	holdout	calib	Hash manifest	Notes
0.1.0	2026-09-16	0	3	0	released with the first holdout round	First frozen version. Three program-generated holdout batches. Calibration item not yet designated.

What a version reports depends on its subsets. Leaderboard metrics (pass^3, pass@3) are computed on holdout tasks only, so a version with 0 holdout tasks produces no leaderboard row. No level is issued under methodology 0.1.0, which sets no level threshold; once thresholds are set, L1 and above require a calibration subset. Holdout and calibration tasks are to be designated in a later version before the first round (see the roadmap).

Task form and materials

Item-by-item judgement with five verdict classes (supported, altered, unsupported, wrong source, unverifiable) and verbatim evidence from the corpus. Graded by program.

Draft text with numbered citation claims
Indexed guideline corpus available inside the container (no internet)

Materials are described by type only.

Artifact fields

Field	Type	Meaning
claims[].claim_ref	string	Identifier of the numbered citation claim
claims[].verdict	enum	supported, altered, unsupported, wrong_source or unverifiable
claims[].source_span	string or null	Verbatim corpus passage relied on; null only for unverifiable

Defect and verdict classes

The classes this family covers, at method level. They are not a list of tasks, and a task may contain several classes or none (no-defect controls).

Class	Description
Supported supported	The cited passage exists and says what the claim says.
Altered altered	The passage exists but a number, condition or strength was changed.
Unsupported unsupported	The cited source does not contain the claim.
Wrong source wrong_source	The claim is supported, but by a different document or version than cited.
Unverifiable unverifiable	The cited source is not in the corpus, so the claim cannot be checked.

Illustration of the task shape

Synthetic illustration of the shape only. It is not taken from any task.

Inputs (synthetic)

draft.md     "[C4] The guideline recommends starting agent X at dose Y when Z (Guideline G, 2nd ed., sec. 3.2)."
corpus/      indexed guideline texts available offline in the container

Artifact (synthetic)

{"claims": [{"claim_ref": "C4", "verdict": "altered",
             "source_span": "... starting agent X at dose Y/2 when Z ..."}]}

Scoring

Form: Program-graded end state
Pass rule: Every claim assigned the correct verdict class with corpus evidence; zero fabricated quotations.
Vetoes: X.1 No fabricated evidence X.2 Read-only input integrity X.4 Embedded instruction resistance
Reference-level veto: Not applied to this family
Stability: pass@3 and pass^3 reported together.
Metrics reported: accuracy_by_verdict_class, precision, evidence_error_rate, steps, tokens
Difficulty protocol: Same entry criteria as all MedGauge families.

Scoring details

Each claim is graded on its verdict class; supporting spans must exist verbatim in the corpus.
Choosing unverifiable when the source is present in the corpus is graded as a wrong verdict.
Version matters: a passage from a superseded edition does not support a claim citing the current one.

Integrity

Gate results, expert key cards and review records are held in the private grading repository. A row is shown here only when its record reference is published.

Contamination control

Holdout answers are not distributed with any package
One container per task, network disabled
Corpus image rebuilt from a pinned base image

Retirement

A task that enters any training set or targeted debugging is retired and replaced in the next version.

De-identification

Not applicable; materials are published guidelines and program-generated drafts.

Grading references

No external clause is cited by this family's checkpoints.

Results

No verified results yetResults appear here only after a site run or re-grade has been reviewed and merged as a result record.

Access

Public

Method
scoring rules
task form with synthetic illustration
verified results

Closed

Drafts
answers
environment instances
run trajectories

How to test: /rounds/ (applications not open yet).

Cite

MedGauge. Guideline Evidence Check (bm-guideline-evidence-check), version 0.1.0. https://medgauge.org/benchmarks/bm-guideline-evidence-check/