Meta-analysis statistics, start to finish

A meta-analysis turns several studies' means into one pooled estimate through four statistical choices: how to weight each study, whether to assume one true value or many, how to measure disagreement between studies, and which interval to report. Each choice changes the headline figure or how far to trust it. This page walks all four on one small invented example.

What pooling is for

Each published study reports a mean and a measure of its uncertainty from a finite sample. Pooling combines those estimates into one with a narrower uncertainty than any of them alone. The gain is real, but it depends on a set of assumptions that are easy to forget once the pooled number appears in a headline.

The general idea is covered in what a meta-analysis is, and is not. This page is the mechanics underneath it, and it stays general: it does not retell any one study, and Veale 2015 has its own page.

The example used throughout

The three studies below are invented to show the arithmetic. They are illustrative only and do not describe any real paper. Each reports a mean erect length, an SD and a sample size.

The standard error is the SD divided by the square root of n. Its square is the variance of that study's mean, and it is the raw material for everything below.

Choice one: the weights

The standard weight is the inverse of the variance of each study's mean. A study that pins its mean down tightly gets a large weight; a noisy or small study gets a small one. Weighted mean: why big studies count more walks through the arithmetic and what it implies.

For the example, the weights are 15.4, 138.4 and 39.1, which add to 192.9. The weighted mean is 13.21 cm, pulled toward the large Study B, against a plain average of 13.00 cm. This is the fixed-effect answer, and its standard error is the square root of 1 over 192.9, about 0.072 cm. The Cochrane Handbook (Higgins et al., chapter 10) is the standard reference for these methods and describes both the fixed-effect and random-effects approaches.

Choice two: fixed or random effects

A fixed-effect model assumes every study is estimating exactly the same true mean, and that studies differ only by sampling error. That is plausible when the studies are near-replicates: the same method, the same kind of men.

A random-effects model assumes each study estimates its own true mean, and that those true means vary around an overall mean by a spread called tau-squared. Studies measured in different countries, with different instruments or different recruitment, are the case it exists for.

The random-effects weight adds tau-squared to each study's variance before inverting it. That one addition flattens the weights: the large study can no longer dominate, because the between-study variance puts a floor under every study's uncertainty.

Choice three: how much the studies disagree

Cochran's Q sums the squared distance of each study from the pooled mean, each scaled by its weight. In the example the fixed-effect pooled mean is 13.21, and the squared, weighted distances add to a Q of about 20.2 on 2 degrees of freedom. If the studies differed only by chance, Q would be expected to be near its degrees of freedom, so a Q of 20 against 2 is a strong signal that they differ for other reasons.

I-squared rescales that: it is (Q minus degrees of freedom) divided by Q, here about 90 percent. Higgins, Thompson, Deeks and Altman (2003), BMJ, introduced it as a way to describe the proportion of the variation in estimates that is due to heterogeneity rather than chance. It is a proportion, not a size: a high I-squared says the disagreement is real, and does not say how many centimetres it is.

Tau-squared supplies the size. The DerSimonian-Laird estimate is (Q minus degrees of freedom) divided by a scaling constant built from the weights, which gives about 0.22 in the example, so a between-study SD of about 0.46 cm. Heterogeneity: when pooled studies disagree covers what to look for when I-squared is high.

Random effects, worked

Add 0.216 to each squared standard error, then invert. The weights become 3.6, 4.5 and 4.1, shares of 29, 37 and 34 percent. The big study has lost its dominance, because its tiny sampling error is now small next to the between-study variance.

The random-effects pooled mean is 13.01 cm, close to the plain average, and its standard error is about 0.29 cm, nearly four times the fixed-effect figure. That is the honest price of admitting that the studies do not agree. With only three studies, tau-squared is itself badly estimated, so these figures are a demonstration, not a result.

Choice four: which interval

The confidence interval around a pooled mean says where the overall mean probably lies. It narrows as studies and samples accumulate, and with enough data it can become very tight. That is the point made in what 15,521 men buys you: a large sample pins the mean down and does nothing to narrow the spread of individuals.

A prediction interval answers a different question: what range of true study means would a new, similar study be expected to find. IntHout, Ioannidis, Rovers and Goeman (2016), BMJ Open, argued that meta-analyses should report it routinely. In their analysis of Cochrane reviews, among statistically significant random-effects results with heterogeneity, 72.4 percent had a 95 percent prediction interval that included a null or opposite effect. The two intervals can tell quite different stories about the same pooled figure.

In the example, a rough prediction interval is the pooled mean plus or minus 1.96 times the square root of tau-squared plus the squared standard error: about 11.9 to 14.1 cm. With three studies the proper interval uses a far larger multiplier, so this is a lower bound on how wide it should be. The comparison between the intervals, and how size papers use them, is in prediction interval versus confidence interval.

Neither is a standard deviation. A standard deviation describes the spread of individual men, a confidence interval describes the precision of a mean, and papers print both in neighbouring columns. Confidence interval versus standard deviation is the post to read when they get swapped.

Checks a careful analyst runs

A pooled figure is more believable when it survives a few stress tests. None of them proves the number right, and each exposes a way it could be wrong.

Leave one out. Drop each study in turn and recompute. In the example, dropping Study B moves the fixed-effect mean from 13.21 to about 12.7 cm, because B carried 72 percent of the weight. If one study can move the answer that far, the answer is really that study's answer.

Subgroups. Split the studies by something that plausibly matters, such as measurement state or whether the observer was a clinician, and pool within each group. If the subgroup means differ by more than their intervals allow, the pooled figure is an average of two different quantities. Method shift and a study mean shows how large that gap can be when a protocol changes.

Small-study pattern. Plot each study's mean against its precision. If the small, noisy studies lean one way and the large ones the other, the missing studies are probably not missing at random. A funnel-shaped scatter with a gap in one corner is the signature.

Sensitivity to the model. Report both fixed and random effects. When they agree, the model choice does not matter; when they disagree, as in the example, the gap itself is a finding about heterogeneity.

A note on sample sizes

The number of men and the number of studies do different jobs. The number of men narrows the within-study standard errors. The number of studies determines how well the between-study variance can be estimated, and three studies, or even a dozen, give a weak estimate of it.

A pooled sample of many thousands of men can therefore sit on top of a handful of studies, and the between-study spread is then barely known. That is one reason the prediction interval for a new study can be far wider than the confidence interval for the mean suggests.

What no statistic repairs

All of the above assumes the study means are measured on a comparable footing. Pooling does not correct a biased method, an unrepresentative sample, or a mix of states and instruments. It averages them in, with weights, and the interval on the result reflects only the disagreement and sampling error it can see.

Two further limits follow. Publication bias shrinks the visible spread in ways the model cannot detect without extra tools, and the reasons are in publication bias in the size literature. And a handful of studies cannot support a stable estimate of between-study variance, so small meta-analyses should be read with the random-effects interval and a wary eye.

Reading a pooled figure

A checklist, in order:

1. How many studies, and how many men in each state?
2. Fixed or random effects, and was the choice stated?
3. What is I-squared, and is a tau or prediction interval given?
4. Which studies carry the most weight, and would you trust their method?
5. Is the interval quoted a confidence interval or a prediction interval?

Where a figure about a photograph is concerned, that is a different kind of number. Rate Cock scores images, and its scores do not come from pooling measured lengths. How a scoring tool is built is covered at aipenis.com, and what a single score is made of is covered at penisrater.com. How a panel of human judges is weighed against one another is that site's question too. The statistics on this page apply to measured studies, where an honest pooled figure always comes with the spread between studies attached.