Isolation and the closed set

Closed set, open method. Anyone can read exactly how a task family is built and graded. No one outside the site can see the tasks themselves.

What is public and what is closed

Public

Methodology, scoring rules, task forms
Synthetic illustrations of artifacts
Schemas for every registry, including task metadata
Hash manifest of holdout answers for each frozen version, released with the first holdout round
Benchmark cards, capability registry, harness and tool registries
Grading references: standard, locator and the rule each checkpoint applies
Site-verified result records, with every run under an opaque task key
Difficulty class and task label distributions per family version
Voided task versions with their attribution

Closed

Task statements, case data, environment instances
Answers, reference solutions, verifiers
Run trajectories
Which grading references each task tests
Difficulty components and task labels of each task
Expert key cards, gate runs and review records
Per-checkpoint grading detail (public records carry only a hash)

Publishing a grading reference does not publish an answer. Tasks test whether a clause applies to an unseen case, when it is triggered, and what to log when information is missing. Tasks that only ask for a reference to be restated are confined to the calibration subset.

Contamination control

Holdout answers never ship with a package; a hash manifest takes their place, released with the first holdout round.
A public task set can overlap with training data, so that a score reflects recall rather than skill (a study of SWE-bench Verified); this is why the holdout is closed.
One container per task with networking disabled; the verifier is mounted only at grading time.
Task material is never used for training or targeted debugging. If it is, the task is retired.
Instructions planted in data test whether a model follows them (X.4).

Training isolation

Training data built for the same capability uses independent source material, stored and licensed separately from evaluation sets. Before release, a similarity check covers six dimensions: patient, data source, original document, task-statement template, answer, and environment state. The result and date are recorded on the dataset card.

De-identification

Removal of identifiers in the 18 categories listed by the HIPAA Safe Harbor method (HHS guidance), used as the identifier checklist.
Dates shifted by one consistent offset per patient, preserving intervals. Safe Harbor itself removes every date element except the year; because shifted dates are kept, the method as a whole is not Safe Harbor and rests on the risk assessment below.
Technical registration per GB/T 37964-2019 and re-identification risk assessment per GB/T 42460-2023.
De-identified is not anonymised. Institutional de-identified data is used only inside the sandbox and is never offered for download. Automated scans that find no identifiers are supporting evidence, not a conclusion.
No page on this site contains individually identifiable information, and submissions containing it are not accepted.