Why this exists
Every accuracy number a testing tool publishes rests on an answer key somebody wrote by hand. Hand-planting a few dozen defects is doable once. Keeping that key honest, and redoing it after every change to the engine, is not something anyone actually does, which is why most of these numbers are quoted once and never re-measured.
So we manufacture the key instead. A mutation operator sabotages a page the
way real products break: an alt text lost, a link that starts returning 404, a label unhooked
from its field, a literal undefined shipped where a value should be. The act of
planting is the answer-key entry, so every catch is attributable to a rule and a
page, and every miss can be named rather than counted.
How a run works
- The product is never touched. The sabotage happens in a mirror proxy in front of a clean copy, so nothing is written to the fixture and a crashed run cannot leave a half-broken product behind.
- The engine runs blind. The sabotaged site goes in; the answer key does not.
- The scorer counts only what it can attribute. A plant counts when it was planted, served, and measured. Anything the clean twin also reports is excluded, because a defect the fixture already had is not one this run created.
The result
400 of 400 attributable plants caught across 21 defect families, on 2 fixture products, at seed 1.
| Defect family | Caught | Recall | 95% interval |
|---|---|---|---|
image-alt | 16 of 16 | 100% | 80.6% to 100% |
broken-image | 16 of 16 | 100% | 80.6% to 100% |
dead-link | 32 of 32 | 100% | 89.3% to 100% |
label | 16 of 16 | 100% | 80.6% to 100% |
select-name | 8 of 8 | 100% | 67.6% to 100% |
button-name | 28 of 28 | 100% | 87.9% to 100% |
link-name | 32 of 32 | 100% | 89.3% to 100% |
color-contrast | 32 of 32 | 100% | 89.3% to 100% |
heading-order | 12 of 12 | 100% | 75.8% to 100% |
html-has-lang | 32 of 32 | 100% | 89.3% to 100% |
asset-404 | 32 of 32 | 100% | 89.3% to 100% |
js-error | 32 of 32 | 100% | 89.3% to 100% |
no-canonical | 8 of 8 | 100% | 67.6% to 100% |
template-scaffold | 8 of 8 | 100% | 67.6% to 100% |
no-favicon | 8 of 8 | 100% | 67.6% to 100% |
no-sitemap | 8 of 8 | 100% | 67.6% to 100% |
source-maps-public | 8 of 8 | 100% | 67.6% to 100% |
no-404-page | 8 of 8 | 100% | 67.6% to 100% |
machine-text | 32 of 32 | 100% | 89.3% to 100% |
empty-view-source | 8 of 8 | 100% | 67.6% to 100% |
document-title | 24 of 24 | 100% | 86.2% to 100% |
| Overall | 400 of 400 | 100% | 99% to 100% |
What it missed
None on this run. That is a result on a fixed date and a fixed seed, not a promise about the next one, which is why both are printed above.
What this number does not mean
The honest part, and the reason to give the table above any weight at all.
- The table is not a claim about your site. These are fixtures we wrote, broken by operators we wrote, so the run measures whether the engine catches the defects it was designed to catch, on pages built to contain them. Your product will contain defects no operator here plants.
- A perfect score on a small sample is still a wide interval. Eight of eight reads as 100%, and the interval beside it says the true rate could be as low as 68%. That is why every row carries one. The intervals narrow as the sample grows, and nothing here narrows them by wishing.
- A family that is not listed is not measured. The table is the whole of what has an operator. Absence from it is not a pass.
- The model-driven checks are not in this table. This measures the deterministic layer, where a missing accessible name is a fact rather than a judgement. What a model finds is measured separately, and less well.
Reproduce it
The artifact carries the seed, so the same sabotage can be rebuilt. Nothing here needs an API key, and the run costs nothing.
node proving.cjs --fixture both --rounds 4 --seed 1
A different seed plants a different sabotage. If the number only holds at seed 1, it was never a number.