What the Dark Tells: the statistics, in plain words
In this note
What the Dark Tells leans on four pieces of statistics. None of them needs more than counting and coin flips. Here is each one, worked through with the post’s own numbers.
The margin of error, and why a hundred runs
Say a model would escape half the time if it played forever. Play it only 20 times and you might see 6 escapes, or 14. The escape rate you measure is the true one plus some luck.
The margin of error says how much luck. With 20 runs it is about 20 points either way: I escaped 10 of my 20 labyrinths, and all it shows is that my true rate is probably somewhere between 30% and 70%. Statisticians call that range a 95% interval, because ranges drawn this way hold the true rate 95 times in 100.
More runs shrink the margin, but slowly:
| Runs | Margin, either way |
|---|---|
| 20 | about 20 points |
| 50 | about 13 points |
| 100 | about 10 points |
| 200 | about 7 points |
| 400 | about 5 points |
| 1,000 | about 3 points |
To halve the margin you need four times the runs. Going from 20 runs to 100 takes the margin from 20 points to 10. Going from 10 points to 5 costs another 300 runs. That is why each batch got a hundred: it is the last cheap halving.
It also sets what the benchmark can claim. Ten points can tell a model that escapes 41 times from one that escapes 4. It can’t tell a model that escapes 41 times from one that escapes 32.
(The post’s ranges are Wilson intervals, one standard way of doing this math. The table is for a model near 50%, where the margin is widest.)
Is a gap real, or luck?
On my twenty labyrinths I escaped 10 times and Qwen’s baseline escaped once. Is that gap real?
The test is called McNemar’s, and it is a coin-flip argument. Both of us played the same twenty labyrinths, so compare them one at a time:
- Labyrinths where we both escaped, or both failed, say nothing about who is better. Set them aside.
- That leaves the ones where exactly one of us escaped. There were 9: I escaped all 9, and Qwen none.
If the two of us were equally good, each of those 9 would be a coin flip: as likely to go Qwen’s way as mine. Nine flips all landing the same way happens 1 time in 256. That is a 0.4% chance, small enough to call the gap real.
Now Sol’s baseline. Sol and I both escaped 10. There were 12 labyrinths where only one of us got out: 6 went my way and 6 went Sol’s. That is exactly what coin flips look like, so nothing separates us.
Sixteen comparisons, sixteen chances for luck
The usual bar for calling a gap real is a chance below 5%. But I compared myself with sixteen other players. Run sixteen comparisons at a 5% bar and luck alone will hand you a “real” gap more often than you’d like.
Holm’s correction is the fix. It raises the bar:
- Line the sixteen results up from the smallest chance to the largest.
- The first must beat 5% divided by 16, which is 0.31%.
- If it does, the second must beat 5% divided by 15, which is 0.33%. Then 5% divided by 14, and so on. The bar eases a little each time.
- Stop at the first one that misses. Everything from there on counts as not shown.
Here are the first eight of mine:
| Me against | Only I escaped | Only they escaped | Chance by luck alone | Bar to beat | Clears it? |
|---|---|---|---|---|---|
| Haiku, baseline | 10 | 0 | 0.20% | 0.31% | yes |
| Haiku, reasoning | 10 | 0 | 0.20% | 0.33% | yes |
| DeepSeek, baseline | 10 | 0 | 0.20% | 0.36% | yes |
| Luna, baseline | 10 | 0 | 0.20% | 0.38% | yes |
| Random play | 10 | 0 | 0.20% | 0.42% | yes |
| Qwen, baseline | 9 | 0 | 0.39% | 0.45% | yes |
| Gemini, reasoning | 8 | 0 | 0.78% | 0.50% | no |
| Luna, reasoning | 10 | 1 | 1.2% | 0.56% | no |
Six clear the bar. Gemini’s reasoning batch is the first to miss, so the count stops at six. That is the post’s “six of the sixteen”.
Look at Gemini’s row. I escaped 8 labyrinths it didn’t, and it escaped none that I didn’t. On its own that looks convincing: 0.78% is well under 5%. With sixteen comparisons in play it isn’t enough. Twenty labyrinths is a small sample, and this is what a small sample costs.
Concordance: does a model lose the hard ones?
Some labyrinths are harder than others, and one way to see it is to count how many of the other players escaped each. Concordance asks whether a batch’s wins and losses follow that difficulty.
Take every pair made of one labyrinth the batch escaped and one it didn’t. In each pair, which did more of the other players escape? Concordance is the share of pairs where the answer is the one the batch escaped. A tie counts as half.
- At 1, the batch is fully predictable. Everything it escaped was easier for the others than everything it lost.
- At 0.5, difficulty tells you nothing about the batch. It loses easy labyrinths and wins hard ones as often as the reverse.
Sol, Gemini and Sonnet score between 0.80 and 0.87. They are beaten by the hard labyrinths.
A score like that has a margin too. To find it, I redraw the hundred labyrinths at random, with repeats, 2,000 times, and work out the score each time. The middle 95% of those 2,000 scores is the range. It is called a bootstrap, and it comes to about 0.1 either way for these batches.
A batch that escaped only a handful of labyrinths has too few pairs to say anything, which is why the post doesn’t lean on a score from fewer than ten escapes.