Small samples, wide intervals

A single study of forty or fifty men can report an average that sits a full centimetre away from the large pooled estimate, and nothing needs to have gone wrong for that to happen. Small samples swing. That is what a confidence interval is telling you, and it is worth reading rather than skipping to the headline number.

What makes a small sample swing

Any sample drawn from a population will, by chance alone, sometimes include more men from one end of the distribution than the true population share would predict. The smaller the sample, the more that random imbalance shows up in the sample mean, because there are fewer data points to average it back out. A sample of a few dozen is small enough that this effect is not a subtle footnote - it can move the reported mean by an amount that looks, to a casual reader, like a real difference between populations or protocols.

A worked comparison of interval width against sample size

Confidence interval width scales with the inverse of the square root of the sample size, holding the underlying spread constant. Quadrupling a sample roughly halves the width of its confidence interval - a study of 40 men and a study of 160 men, drawn from the same population with the same measurement care, will report similar means on average, but the 40-man study's interval around its own mean will be roughly twice as wide. This is arithmetic that follows from how a standard error is calculated, not a figure drawn from any specific paper, and it is worth working through in the abstract precisely so it does not get mistaken for a claim about any one study.

Why the interval is the part to read

A single-point mean invites treating it as a settled fact. The confidence interval attached to it is the study's own honest statement of how much that mean could plausibly move if the same protocol were repeated on a fresh sample of the same size. A small study with a wide interval that overlaps the large pooled estimate is not contradicting it - it is consistent with it, and reporting the point estimate alone, stripped of its interval, is what makes the two look like a disagreement.

Why a wide interval is not a weakness to hide

A small study reporting a wide confidence interval alongside its mean is doing exactly what good methodology requires - stating honestly how much its own estimate could move on a repeat sample. The failure is downstream, when that interval gets dropped in the retelling and the point estimate travels alone. A study is not less trustworthy for reporting a wide interval; a summary of that study is less trustworthy for hiding one.

Where this shows up in circulation

Older or smaller single studies still get cited individually, sometimes because they are easier to find summarised than the full meta-analysis, sometimes because their point estimate happens to support whatever the citing source wanted to say. A small study's mean, quoted without its interval, reads as precise in a way the underlying data do not support. Studies disagreeing with each other for reasons of method, population and instrument is one layer of this problem; sampling variability in small studies is a separate layer that would still produce disagreement even if every study used an identical protocol on an identical population.

What a large pooled sample buys you instead

The 15,521 men in Veale 2015 buy precision about where the population mean sits, not a narrower spread of individuals - that distinction matters here because the swing this post describes is about the mean's own precision, which a large sample corrects, not about how spread out individual men are, which no sample size changes. Reading the study itself in full is the way to see how its size and design address exactly this weakness in the smaller papers that came before it.

Sampling variability is a property of measured data specifically, and it does not have a direct equivalent on a rating platform, where each output is a single judgement rather than an estimate of a population mean. Rate Cock and services like it, including Penis Rater, Rate Penis and AI Penis, are not estimating a population parameter from a sample at all when they score an individual photograph, so the small-sample caution here is specific to studies that are trying to characterise a distribution - which is exactly what this site's percentile discussions rely on getting right.