A confirmed bug becomes a standing check

Generating tests is the easy part, and treating it as the product creates three problems that compound:

• Green by construction. A generated test is green the day it is written and green every day after. Nothing in that sequence is evidence it can detect anything.
• No way to tell protection from decoration. From the outside, a test that is guarding you looks exactly like one that was never capable of failing.
• Confidence with nothing behind it. A suite made mostly of the second kind reports everything passing while the product breaks underneath it, which is worse than having no suite, because it costs you the one thing a suite is supposed to buy.

So regression checks are built on a single rule, and every behaviour below follows from it.

Key features of regression checks

1. A guard starts from a confirmed defect

• A guard comes from a finding a run filed and you judged real, and that carries a repro a browser can re-execute.
• The probe is inverted rather than copied. A finding’s repro confirms a defect; the guard runs the same probe and passes when it is refuted.
• That makes a guard the assertion that a bug is absent, so it starts failing the moment a regression puts the bug back.

2. A guard must fail before it counts

• At the moment a guard is minted, the defect is still in the product, so the guard must fail on its first run. That failure is the proof.
• The proof is written onto the guard once and never overwritten. Re-recording it on every failing run would quietly turn proven in March into proven today and hide how long something has been red.
• A guard that has never failed is reported and never counted. Green is not evidence.

3. Six states, one of which counts as protection

Every re-check lands in one of six states on every run. Each row of the table below was produced by calling the same function the product calls, with the arguments a real run passes it. There are seven rows for six states because proven covers two different situations, and the table shows both.

Two of those rows do the work:

• Unproven is the ordinary failure of every generated suite: a check that passed on its first run and has therefore never demonstrated it can detect anything.
• Unsound is the case most tools do not look for. The ledger says the defect is open; the guard says it cannot reproduce it. Both cannot be true, so either the finding was wrong or the repro does not test what the finding claims. That guard is quarantined rather than counted.

4. What a run refuses to protect

Not every finding can become a check, and a run that seated fewer without saying so would be hiding useful information. Every refusal is named, with the reason:

The fourth row comes from a bug we shipped. A guard was minted for a finding whose repro no engine implemented, ran, reported that it could not be executed, and was then counted in the suite’s own total, dragging down a fraction it could never have contributed to. A guard that can only ever report nothing should not be counted as protection.

What a run tells you

The suite reports the arithmetic rather than a score. Nine guards is not nine pieces of protection, and the line the console prints does not round it up.

Both lines are the output of headlineOf() in lib/guard.cjs, called with the summary shapes a real run hands it.

The second line is the main purpose of the feature. A regression here means a defect you found, judged real and fixed is reproducing again in a real browser. No model is involved at any step, so the result does not depend on a model’s opinion.

What you have in month three

This is the part that compounds, and it is what a testing tool is worth once the first run is no longer new.

• The suite builds itself from your decisions. Every defect you confirm adds a check that runs on every later audit, in a real browser, with no API key and no per-run cost.
• It is shaped like your product. It is derived from your product’s own judged defects, so nobody can ship you a copy of it.
• The number at the top can be relied on. Every check counted in it had to be seen to fail before it was allowed to count.

The limit is the same as for everything else here. This protects what was found, confirmed and re-executable. It says nothing about the defects nobody has found yet, and no suite can.

Getting started with regression checks

Regression checks are on by default and need no configuration. The loop is four steps:

1. Run an audit. Findings land in the ledger with their repro attached.
2. Confirm what is real. Only a finding you judged can be protected, which is the point of judging it.
3. Watch the guard fail. It is minted while the defect is still there, so its first run is red on purpose. That is the proof being recorded.
4. Fix the defect. The guard goes green, moves to armed, and from then on it is the thing that tells you if the bug comes back.

From there it runs with every audit, and the report separates what is armed from what is merely present. Anything it could not seat is listed with the reason, so the gap between what you found and what you are protected against is visible rather than assumed.