nth Labs
Start testing
Product Pricing Security Blog Help About
Sign in Start testing
Blog

36 out of 36 is not 100%

You ran 36 tests and all 36 passed. The interval everybody reaches for says your success rate is 100%, plus or minus nothing. It also says a rate can be negative, and that 1 in 20 could be 105%. Here is why, and the one-line fix.

nth Labs 7 min read

Every builder eventually reports a rate. 36 of 36 checks passed. 19 of 20 users finished the flow. Both are samples, so neither number is the thing you wanted to know. Each is a guess at it, and how good a guess depends on how many times you looked.

The standard way to put error bars on a guess like that is the Wald interval. It is the one in the textbook and the first result for “confidence interval formula”.

p ± 1.96 × √( p(1-p) / n )

It is simple and it is famous. At the sample sizes a small team works with, it is also wrong, and wrong in a way you can see with your own eyes.

Four answers it should not be able to give

Where the textbook interval breaks Red above: Wald, the formula in the textbook. Below it: Wilson, with the observed rate marked. the axis runs past both ends on purpose, because two of these intervals do too 0%25%50%75%100% WaldWilson a perfect score 36/36100 to 10090 to 100every planted defect found 41/41100 to 10091 to 100one miss in twenty 19/2085 to 10576 to 99one hit in twenty 1/20-5 to 151 to 24nothing found at all 0/200 to 00 to 16our own judged layer 4/164 to 4610 to 50
Computed by scripts/wilsonfigs.cjs. The Wilson row calls the function the product actually publishes numbers with; Wald is written out in the script because nothing here implements it.
  • A perfect score gets no uncertainty at all. 36 of 36 comes out as 100% to 100%. Plug in p = 1 and the square root is zero, so the interval has no width. Thirty-six observations, and the formula claims certainty.
  • Rates go negative. 1 hit in 20 gives a lower bound of −4.6%.
  • Rates go over 100%. 19 of 20 gives an upper bound of 104.6%.
  • Finding nothing proves nothing is there. 0 of 20 comes out as 0% to 0%, which would let a tool that found no bugs report that your product has none.

None of those is an edge case you can steer around. They are where the numbers land whenever you measure something that mostly works.

A 95% interval that is right 4% of the time

The failures above are visible. The real problem is not.

An interval labelled 95% is making a promise. Run the experiment over and over and 95% of the intervals you build will contain the true value. That promise can be checked exactly. Fix the number of trials and the true rate, then add up the probability of every outcome whose interval would have covered it.

What a “95% confidence interval” is actually worth How often each method’s interval really contains the true rate, at 16 trials. The shaded band is the difference. exact binomial sums over all 17 outcomes · 399 sample points no simulation, so this figure needs no error bars of its own 0%20%40%60%80%100%0%20%40%60%80%100% 95% promised Wald, the textbook formulaWilson, what we publish 3.9% at a true rate of 99.8% never below 85.1% the true rate you are trying to measure
Exact binomial sums over all 17 possible outcomes at 16 trials, at 399 sample points across the true rate. No simulation, which is why this figure needs no error bars of its own.

The flat line is the promise. Wilson tracks it. Wald never reaches it at any true rate on this chart, and its best showing anywhere is 94.7%. At the edges it gives out completely: above a true rate of 99% its real coverage drops under 5%, so a “95% confidence interval” holds the truth one time in twenty-five.

The sawtooth is not noise. Counts are whole numbers, so as the true rate slides the set of outcomes that would cover it changes in jumps, and every method has that shape. What separates Wald is how far below the line its teeth sit, and that they stay there.

The fix is one line

The Wilson score interval inverts the test rather than approximating it, and the result stays inside the range a proportion is allowed to occupy:

Wilson, in full

centre = (p + z²/2n) / (1 + z²/n), half-width = z√( p(1-p)/n + z²/4n² ) / (1 + z²/n)

Four more terms than Wald, and all four failures go away. 36 of 36 becomes 90.4% to 100%. It still stops at 100%, where a rate has to stop, but it now carries the floor that thirty-six observations buy you. 0 of 20 becomes 0% to 16.1%, which is the right answer to “we looked and found nothing”.

The z²/2n in the numerator is doing the work. It pulls the centre away from the boundary in proportion to how little data you have. That is the correction a small sample needs, and it is the term Wald has no equivalent of.

How to read a rate somebody quotes you

  • A rate with no interval is not a measurement. “95% accurate” with no n behind it tells you nothing at all. Ask what the denominator was.
  • Be suspicious of a perfect score. 100% from a small sample is the one case where the common formula produces a confident lie, so it is the one most likely to be quoted at you.
  • A wide interval is the honest one. 4 out of 16 is 10% to 50%. That is an embarrassing thing to publish and it is what sixteen trials are worth.
  • If you report rates, switch the formula. It is a few lines and it removes a class of impossible answer permanently.

We publish our own detector numbers this way, which is how the widest bar in the first figure came to be on this page at all. Our judged layer is 4 of 16. The formula does not care which direction it embarrasses you in, and that is the whole reason to use one.