Methodology / Articles

Method · 2026-10-29

When guidelines disagree

Analysis Methodology 0.1.0 · what the grades mean

How grading references handle a clause on which sources from different jurisdictions point in different directions, without marking a correct answer wrong.

The problem

Guidance from different jurisdictions need not agree. In the abstract, two guidelines can set different thresholds for the same action, or different timing for the same precaution. A benchmark that picks one side would mark every answer that follows the other side wrong — not because the answer is unsafe, but because the benchmark is parochial. This article states MedGauge’s rule for that case; it makes no claim about which particular guidelines disagree.

The rule

A grading reference whose sources disagree lists both sources side by side, each with its locator and check date, and carries a jurisdiction statement: a task that tests it must state the applicable regional standard in its materials. Every task records the standard it is graded against in its region-and-standard label (T8); where sources conflict, the label names the version that applies. A task that states no region does not test the reference at all.

Why not average the two

Averaging produces a requirement that neither guideline makes. Choosing the stricter one sounds safe, but it marks guideline-conformant care as an error. Making the task state the region keeps the grade tied to a real, citable standard.

What it means for results

Such references are tested only in tasks that name a region. Results on those tasks therefore show whether a model follows the stated standard — which is the competence a deployed system needs — rather than which region’s habits it absorbed. Every such task also passes the conformant-alternative test with the other region’s standard, so that grading never marks a correct alternative wrong.

Related: Standards · Grading references