Measuring your own system
Gates, negative controls, and when a test lies.
Before rewriting our research, we probed every read tool by content, not by HTTP status. Thirty-odd tools worked. Four defects surfaced, one of which was not a defect: we had reported a number without reading the design behind it.
For three and a half hours our agent layer recorded none of five local-model runs. A network sensor that knows nothing about agents recorded all five, to the minute. Self-report was not wrong. It was absent, and absence reads as 'nothing happened'.
Every figure on this site comes from a running system. The standard, the time it failed us by a factor of eighteen, and where the numbers come from.
Una gave the correct physics and cited the encyclopedia instead of the primary paper sitting next to it. Four hypotheses, each tested by changing one variable. She does not pick the most authoritative source. She cites the one whose wording she reused.
Two passing evaluations in one day, both blind to the change they were meant to judge. What they missed, and the rule we took from it.
A paid vision model listed ten apps that do not exist. Our local model counted fourteen icons where there were eleven. Both were wrong about the same 12-pixel detail, and chasing it found a real bug nobody would have seen.