‽ Paradox Lab

Home › Statistics › Simpson's paradox: better in every group, worse overall

Simpson's paradox: better in every group, worse overall

A treatment can be better for every type of patient and still look worse overall.

The puzzle

A 1986 study compared two treatments for kidney stones. Treatment A had a higher success rate for small stones. It also had a higher success rate for large stones. But when the researchers added everyone together, Treatment B had the higher success rate.

How can something be better in every group but worse overall? No numbers were fudged. This is Simpson's paradox.

Successful treatmentsTreatment ATreatment B
Small stones81 of 87 (93%)234 of 270 (87%)
Large stones192 of 263 (73%)55 of 80 (69%)
All patients273 of 350 (78%)289 of 350 (83%)

Source: Charig and colleagues, British Medical Journal, 1986. A was open surgery; B was a less invasive keyhole procedure.

⚖️ Change who gets Treatment A

Keep each treatment's success rate within each group exactly as in the study, but change the mix of patients sent to Treatment A.

Treatment ATreatment B

Treatment B's patients stay as in the study (270 small, 80 large). The flip happens once more than about 52% of A's patients have large stones.

Why it happens: the groups aren't the same size

Large stones are harder to treat, whatever the method. In the study, doctors tended to give the tough cases (large stones) to Treatment A, the more invasive surgery, and the easier cases to Treatment B. So three-quarters of A's patients had large stones, compared with under a quarter of B's.

The overall success rate is a weighted average of the two groups. A's average is dragged down because most of its patients sit in the harder group. B's average is pulled up because most of its patients sit in the easier group. The comparison is really "A on hard cases versus B on easy cases".

The hidden factor (stone size) is called a confounding variable, sometimes a "lurking" variable. It affects both who gets which treatment and how well it works.

So which treatment is better?

For this data, A: it does better for both kinds of patient, and the overall figure is misleading because of who was sent where. But there's no purely mathematical rule that says "always split the data". Whether the grouped or the combined figures tell the true story depends on what causes what. Splitting by a confounder (like stone size) is right; splitting by something the treatment itself changed can be wrong.

Other famous examples

Questions

What is Simpson's paradox in simple terms?

A trend that appears in several groups of data disappears or reverses when the groups are combined. It happens when the groups are different sizes and a hidden factor affects both the group sizes and the outcome.

How do you avoid being fooled by Simpson's paradox?

Ask whether something besides the thing you are comparing could affect both who ends up in each group and the result. If so, compare like with like within each group. Randomised trials avoid it by assigning treatments by chance.

Is Simpson's paradox named after The Simpsons?

No. It's named after statistician Edward Simpson, who described it in 1951, though others had noticed the effect earlier.

Try another paradox

False positive paradox

A 99% accurate test can be wrong most times it says "positive". See 1,000 people.

Statistics

Birthday paradox

Just 23 people gives better than even odds. Fill random rooms and check.

Probability