Why we plant defects
Every accuracy number a testing tool publishes rests on an answer key somebody wrote by hand. Hand-planting a few dozen defects is doable once. Keeping that key honest, and redoing it after every change to the engine, is rarely done, so most of these numbers are quoted once and never re-measured.
So we manufacture the key instead. A mutation operator sabotages a page the
way real products break: an alt text lost, a link that starts returning 404, a label unhooked
from its field, a literal undefined shipped where a value should be. The act of
planting is the answer-key entry, so every catch is attributable to a rule and a
page, and every miss can be named rather than counted.
How a run works
- The product is never touched. The sabotage happens in a mirror proxy in front of a clean copy, so nothing is written to the fixture and a crashed run cannot leave a half-broken product behind.
- The engine runs blind. The sabotaged site goes in; the answer key does not.
- The scorer counts only what it can attribute. A plant counts when it was planted, served, and measured. Anything the clean twin also reports is excluded, because a defect the fixture already had is not one this run created.
The result
400 of 400 attributable plants caught across 21 defect families, on 2 fixture products, at seed 1.
| Defect family | Caught | Recall | 95% interval |
|---|---|---|---|
image-alt | 16 of 16 | 100% | 80.6% to 100% |
broken-image | 16 of 16 | 100% | 80.6% to 100% |
dead-link | 32 of 32 | 100% | 89.3% to 100% |
label | 16 of 16 | 100% | 80.6% to 100% |
select-name | 8 of 8 | 100% | 67.6% to 100% |
button-name | 28 of 28 | 100% | 87.9% to 100% |
link-name | 32 of 32 | 100% | 89.3% to 100% |
color-contrast | 32 of 32 | 100% | 89.3% to 100% |
heading-order | 12 of 12 | 100% | 75.8% to 100% |
html-has-lang | 32 of 32 | 100% | 89.3% to 100% |
asset-404 | 32 of 32 | 100% | 89.3% to 100% |
js-error | 32 of 32 | 100% | 89.3% to 100% |
no-canonical | 8 of 8 | 100% | 67.6% to 100% |
template-scaffold | 8 of 8 | 100% | 67.6% to 100% |
no-favicon | 8 of 8 | 100% | 67.6% to 100% |
no-sitemap | 8 of 8 | 100% | 67.6% to 100% |
source-maps-public | 8 of 8 | 100% | 67.6% to 100% |
no-404-page | 8 of 8 | 100% | 67.6% to 100% |
machine-text | 32 of 32 | 100% | 89.3% to 100% |
empty-view-source | 8 of 8 | 100% | 67.6% to 100% |
document-title | 24 of 24 | 100% | 86.2% to 100% |
| Overall | 400 of 400 | 100% | 99% to 100% |
What it missed
None on this run. The result is for one date and one seed, both printed above, and the next run may differ.
What this number does not mean
- The table is not a claim about your site. These are fixtures we wrote, broken by operators we wrote, so the run measures whether the engine catches the defects it was designed to catch, on pages built to contain them. Your product will contain defects no operator here plants.
- A perfect score on a small sample is still a wide interval. Eight of eight reads as 100%, and the interval beside it says the true rate could be as low as 68%, so every row carries one. The intervals narrow only as the sample grows.
- A family that is not listed is not measured. The table lists every family that has an operator. Absence from it is not a pass.
- The model-driven checks are not in this table. This measures the deterministic layer, where a missing accessible name is a fact rather than a judgement. What a model finds is measured separately, and less well.
Base rates from real products
The test above answers whether a finding is real. This section answers a different question, one a founder can use before launch: how a product compares with real ones on the things a stranger judges in ten seconds. The comparison needs no traffic, revenue or account, because every figure is read off the pages as they stand.
- 193 products, read once each. The home page, plus the pricing and signup pages it links to. Read-only: no clicking, no typing, no forms, no repeat visits, and robots.txt honoured. 254 were listed: one was skipped because its robots.txt asked us to, 50 refused us and 10 never loaded, and all of those are recorded as what they were rather than as products with nothing wrong.
- Found by following links, never by guessing. A visitor reaches a pricing page by clicking one. Probing for a URL nobody linked is scanning a site rather than reading it, so a product whose home page has no pricing link has no pricing page read.
- Almost a quarter offer no way to reach a person. 77% have one. 81% have a headline that says more than the product’s own name. 87% of the pricing pages we read state a price. The middle of them ask 1 to 6 questions at signup.
- Every figure carries its interval and its sample. A rate over 193 products and a rate over 16 are not the same kind of statement, and a group too thin to quote returns nothing at all rather than a number that looks solid.
What it does not do: it measures how common something is and where a product sits, never what either is worth in money. Nothing in this corpus watched a checkout.
Reproduce it
The artifact carries the seed, so the same sabotage can be rebuilt. The command runs from a copy of the nth Labs repository, which is private: Run it yourself says how to ask for one. Nothing here needs an API key, and the run costs nothing.
node proving.cjs --fixture both --rounds 4 --seed 1
A different seed plants a different sabotage, so you can check that the result holds beyond seed 1.