Home › Statistics › Simpson's paradox: better in every group, worse overall
Simpson's paradox: better in every group, worse overall
A treatment can be better for every type of patient and still look worse overall.
The puzzle
A 1986 study compared two treatments for kidney stones. Treatment A had a higher success rate for small stones. It also had a higher success rate for large stones. But when the researchers added everyone together, Treatment B had the higher success rate.
How can something be better in every group but worse overall? No numbers were fudged. This is Simpson's paradox.
| Successful treatments | Treatment A | Treatment B |
|---|---|---|
| Small stones | 81 of 87 (93%) | 234 of 270 (87%) |
| Large stones | 192 of 263 (73%) | 55 of 80 (69%) |
| All patients | 273 of 350 (78%) | 289 of 350 (83%) |
Source: Charig and colleagues, British Medical Journal, 1986. A was open surgery; B was a less invasive keyhole procedure.
⚖️ Change who gets Treatment A
Keep each treatment's success rate within each group exactly as in the study, but change the mix of patients sent to Treatment A.
Treatment B's patients stay as in the study (270 small, 80 large). The flip happens once more than about 52% of A's patients have large stones.
Why it happens: the groups aren't the same size
Large stones are harder to treat, whatever the method. In the study, doctors tended to give the tough cases (large stones) to Treatment A, the more invasive surgery, and the easier cases to Treatment B. So three-quarters of A's patients had large stones, compared with under a quarter of B's.
The overall success rate is a weighted average of the two groups. A's average is dragged down because most of its patients sit in the harder group. B's average is pulled up because most of its patients sit in the easier group. The comparison is really "A on hard cases versus B on easy cases".
The hidden factor (stone size) is called a confounding variable, sometimes a "lurking" variable. It affects both who gets which treatment and how well it works.
So which treatment is better?
For this data, A: it does better for both kinds of patient, and the overall figure is misleading because of who was sent where. But there's no purely mathematical rule that says "always split the data". Whether the grouped or the combined figures tell the true story depends on what causes what. Splitting by a confounder (like stone size) is right; splitting by something the treatment itself changed can be wrong.
Other famous examples
- University admissions: in 1973 UC Berkeley's graduate school admitted a lower share of women than men overall, yet most individual departments admitted women at similar or higher rates. Women had mostly applied to the more competitive departments.
- Batting averages: one cricketer or baseballer can have a better average than another in each of two seasons but a worse average across both, when they batted very different numbers of times in each.
- Hospital survival rates: a top hospital can have worse overall survival than a small one simply because it takes the sickest patients.
Questions
What is Simpson's paradox in simple terms?
A trend that appears in several groups of data disappears or reverses when the groups are combined. It happens when the groups are different sizes and a hidden factor affects both the group sizes and the outcome.
How do you avoid being fooled by Simpson's paradox?
Ask whether something besides the thing you are comparing could affect both who ends up in each group and the result. If so, compare like with like within each group. Randomised trials avoid it by assigning treatments by chance.
Is Simpson's paradox named after The Simpsons?
No. It's named after statistician Edward Simpson, who described it in 1951, though others had noticed the effect earlier.