A confirmed bug becomes a standing check

Every tool in this category can generate tests. That is not the hard part, and treating it as the product creates three problems that compound:

• Green by construction. A generated test is green the day it is written and green every day after. Nothing in that sequence is evidence it can detect anything.
• No way to tell protection from decoration. From the outside, a test that is guarding you looks exactly like one that was never capable of failing.
• Confidence with nothing behind it. A suite made mostly of the second kind reports everything passing while the product breaks underneath it, which is worse than having no suite, because it costs you the one thing a suite is supposed to buy.

So re-checks are built on a single rule, and every behaviour below follows from it.

Key features of re-checks

1. Minted from a defect, not from a prompt

• A guard comes from a finding that was actually found, actually judged real by you, and that carries a repro a browser can re-execute.
• The probe is inverted rather than copied. A finding’s repro confirms a defect; the guard runs the same probe and passes when it is refuted.
• That makes a guard the assertion that a bug is absent, so it starts failing the moment a regression puts the bug back.

2. Proven by failing, before it is counted

• At the moment a guard is minted, the defect is still in the product, so the guard must fail on its first run. That failure is the proof.
• The proof is written onto the guard once and never overwritten. Re-recording it on every failing run would quietly turn proven in March into proven today and hide how long something has been red.
• A guard that has never failed is reported and never counted. Green is not evidence.

3. Six answers, and only one of them is protection

Every re-check lands in one of six states on every run. The table below is not a description of that logic: each row was produced by calling the same function the product calls, with the arguments a real run passes it. There are seven rows for six states because proven covers two different situations, and it is worth seeing both.

Two of those rows do the work:

• Unproven is the ordinary failure of every generated suite: a check that passed on its first run and has therefore never demonstrated it can detect anything.
• Unsound is the case most tools do not look for. The ledger says the defect is open; the guard says it cannot reproduce it. Both cannot be true, so either the finding was wrong or the repro does not test what the finding claims. That guard is quarantined rather than counted.

4. It says what it refused to protect

Not every finding can become a check, and a run that quietly seated fewer would be hiding the most useful thing it knows. Every refusal is named, with the reason:

The fourth row is the one that was learned by shipping it. A guard was minted for a finding whose repro no engine implemented, ran, reported that it could not be executed, and was then counted in the suite’s own total, dragging down a fraction it could never have contributed to. A slot that can only ever report nothing is a slot pretending to be protection.

What a run tells you

The suite reports the arithmetic rather than a score. Nine guards is not nine pieces of protection, and the line the console prints refuses to round that up.

Both lines are the output of headlineOf() in lib/guard.cjs, called with the summary shapes a real run hands it.

The second line is what this whole feature exists for. A regression here is not a model’s opinion that something looks wrong. It is a defect you already found, already judged real, already fixed, reproducing again in a real browser. There is no model anywhere in that loop, so there is nothing to disagree with.

What you have in month three

This is the part that compounds, and it is the honest answer to what a testing tool is worth once the first run stops being novel.

• The suite builds itself from your decisions. Every defect you confirm adds a check that runs forever afterwards, in a real browser, with no API key and no per-run cost.
• It is shaped like your product. It is derived from your product’s own judged defects, so nobody can ship you a copy of it.
• The number at the top is spendable. Every check counted in it had to be seen to fail before it was allowed to count.

The limit is worth stating plainly, because it is the same limit as everything else here. This protects what was found, confirmed and re-executable. It says nothing about the defects nobody has found yet, and no suite that claims otherwise is telling you the truth.

Getting started with re-checks

Re-checks are on by default and need no configuration. The loop is four steps:

1. Run an audit. Findings land in the ledger with their repro attached.
2. Confirm what is real. Only a finding you judged can be protected, which is the point of judging it.
3. Watch the guard fail. It is minted while the defect is still there, so its first run is red on purpose. That is the proof being recorded.
4. Fix the defect. The guard goes green, moves to armed, and from then on it is the thing that tells you if the bug comes back.

From there it runs with every audit, and the report separates what is armed from what is merely present. Anything it could not seat is listed with the reason, so the gap between what you found and what you are protected against is visible rather than assumed.