Reference
Methodology
How subjects, judge decisions, compatibility identities, costs, and evidence produce each measurement
Subject and answer key
Each subject is a real pull request patch with an answer key written before a measured review; the key distinguishes found, missed, intended, and unkeyed findings and carries a fingerprint
Locality candidates
A finding becomes a candidate when its location overlaps a keyed defect or lands within three lines; locality can propose a match, but it cannot decide that two explanations describe the same defect
Model judge
A calibrated model decides candidate finding-to-defect pairs and intended trade-offs; model-judged precision is unavailable after a partial judge pass; null never means zero
Compatibility boundary
A comparison identity fixes the subject, pull request, reviewed SHA, answer-key fingerprint, reviewer tool, resolved configuration fingerprint, model roles, reviewer build, harness, and adapter; trials combine only inside that identity
Trial distributions
Comparison ranges use the exact minimum and maximum with Type-7 linear p50 and p95 percentiles over scored trials only; malformed and unscored trials remain visible but cannot change an aggregate; a distribution is available only when every scored trial measured that metric; partial samples retain their measured count and unavailable reason without producing a substitute percentile
Cost records
Reviewer adapters and judge runners record observed provider charges with their supporting evidence; the benchmark does not infer spend from token counts or replace unavailable charges with zero; running a benchmark may spend real money, while reading committed records does not
What a token total counts
Every usage figure is the reviewer's own claim about itself, and the reviewers do not all count the same thing; a harness that delegates to subagents totals every model it ran, while one that does not totals the model under test, so a reviewer reporting several times another's tokens on the same diff may be reporting a wider figure rather than a hungrier run; each run page lists the models its own accounting names, and two totals are comparable only when those lists are
Content-addressed evidence
The static export deduplicates committed inputs by SHA-256; run pages disclose artifact state and link to the immutable blob used for the view; missing and malformed artifacts stay visible
Important: severity is impact; tier is discovery difficulty; a score on one subject is not comparable to a score on another patch with different size and fault density