How to Spot Bad Data in Under a Minute
Seven checks that catch most junk statistics before you repeat them: sample size, base rate, source chain, date drift, and the three framing tricks.
Most bad statistics don't survive sixty seconds of attention. The problem is that they're usually given zero seconds, because they arrive inside something we already agree with.
Here are the seven checks I run before repeating any number, in the order I run them. Together they take about a minute once they're habitual.
The seven checks
1. Where does it terminate?
Follow the citation. Then follow that one. Keep going until you reach either a primary source or nothing. The number of widely-circulated statistics that terminate in nothing — a press release citing a survey that was never published, or a slide with no attribution — is genuinely startling once you start checking.
This single check catches more bad data than the other six combined.
2. What's the sample?
Size, and more importantly composition. A survey of 2,000 people is meaningless if all 2,000 were customers of the company publishing it. Self-selected online polls tell you about people who answer online polls.
3. What's the base rate?
"Doubled your risk" is uninformative without knowing the starting risk. From 1 in 10 million to 2 in 10 million is a doubling and it is also nothing. Percentage changes without absolute numbers are the most common way true statements mislead.
4. When?
Numbers circulate long after they expire, and dates fall off in transit. If you can't establish when the data was collected — not published, collected — treat it as unusable. In fast-moving fields, three years is often ancient.
5. What was actually measured?
The proxy is rarely the thing. "Engagement" isn't interest. "Sessions" aren't people. "Reported symptoms" aren't diagnoses. When a headline claims something was measured, find the sentence describing the actual measurement — it's often meaningfully different from the claim.
6. Who benefits if I believe this?
Not disqualifying. Directional. It tells you which checks to run hardest. A vendor's own benchmark showing their product winning isn't a lie, but you should want to know what the competitors were configured like.
7. What's missing?
The hardest and most valuable one. Which comparison isn't shown? Which time period got cropped? If someone shows you three years of a five-year dataset, the missing two years are the story.
Three framing tricks that use true numbers
Relative when absolute is boring. "50% increase" sounds substantial. "From 2 to 3" often doesn't. Both are true; only one is informative, and the choice of which to lead with is itself a signal.
The truncated axis. A chart starting at 94% instead of 0% turns a rounding error into a cliff. Look at the axis before you look at the shape — a habit that takes two seconds and never stops being useful.
The composite index. Several unrelated things combined into one score with unexplained weights. The weights are where the argument is hiding, and they're almost never shown.
The most misleading statistics are usually true. That's precisely why the checks have to be about method rather than about accuracy.
Practising it without becoming exhausting
Two things made this sustainable for me.
First, only run the full check on numbers you're about to repeat. You cannot audit everything you read and trying will make you miserable. The obligation attaches at the point of transmission, not consumption.
Second, be equally suspicious of numbers you like. This is the hard part and the only part that really matters. Everyone scrutinises data that contradicts them. The check that improves your accuracy is the one you run on the statistic that flatters your existing position — and if you notice you're not running it, that's the signal.
The seven checks on one real claim
Worth walking through end to end. Take a claim I encountered while researching another piece: "73% of remote workers report higher productivity."
1. Where does it terminate? Article → a second article → a vendor blog post → a survey the vendor commissioned. Four hops to reach anything primary. Already a warning: laundering through intermediaries is how weak numbers acquire authority.
2. What's the sample? 1,100 respondents, recruited from the vendor's own user base — people who had already chosen remote-work software. Not a sample of remote workers; a sample of enthusiasts.
3. Base rate? Not applicable here, but the related question is: compared to what? The survey asked about productivity relative to an office, with no measurement of either.
4. When? Fieldwork in 2021. The claim was circulating in 2026 with no date attached. Five years, across a period when remote work changed enormously.
5. What was measured? Self-reported perceived productivity. Not output, not hours, not anything observable. That's a legitimate thing to measure and a completely different claim from "are more productive".
6. Who benefits? A company selling remote-work software. Not disqualifying, and it tells you to check the sample composition — which, as above, is where the problem was.
7. What's missing? The response rate, and the wording of the question. Both absent from the published summary.
Total time: about four minutes, most of it on the citation chain. The claim isn't a lie — it's a real survey with a real result. It just doesn't support the sentence it's being used to support, which is the usual situation.
Reading a chart in ten seconds
Charts compress the same tricks into a form that bypasses your scepticism, because a picture feels like evidence. Four things to look at, in this order, before you look at the shape:
- The y-axis start. If it isn't zero, the visual magnitude is not the real magnitude. Sometimes that's legitimate — small variations in a large number can be the story — but you need to know before your eyes decide.
- The x-axis range. Where does it start and stop, and is the cut-off doing work? A trend beginning at an unusual year is often a trend that reverses if you go back further.
- What's being counted. Absolute numbers, rates, percentage change, or an index? Percentage-change charts of small numbers produce dramatic shapes from nothing.
- Missing series. If one thing is plotted, ask what the obvious comparison would have been and why it isn't there.
Then look at the shape. Doing it in this order costs a few seconds and stops the picture from framing the question before you've read the axes — which is exactly what a well-designed misleading chart is for.
Questions I get about this
Found an error in this piece? Tell me — corrections get made in the open with a dated note, never a silent edit. The log is public.
If you want the long version: there's a course opening later this year — The Research Method, waitlist only for now. If this post already solved your problem, you don't need it.