Reading a size study critically

To judge a size study, ask seven questions in order: who was recruited, who held the ruler, which state and method, who was excluded, how the figure was rounded, what spread is reported, and what the number is being used to claim. A study that answers them plainly can support a figure. A story that answers none of them is a number with a headline.

Why a checklist and not a verdict

Size figures reach most people through a chain: a paper, then a press summary, then a chart, then a forum post quoting the chart. Every link drops detail. By the end, "mean erect length in a sample of volunteers who self-measured at home" has become "average is X".

A checklist fixes this because it is boring and repeatable. It does not ask you to decide whether a study is good. It asks you to write down what the study did, so that a figure can be compared with another figure only when they are comparable. That is the same move this site makes everywhere: replace an anecdote with a method.

1. Where did the sample come from?

Start with recruitment, because it decides who can appear in the data at all.

Clinic samples contain men who attended for other reasons, and clinic samples are not the general population. Volunteer samples contain men who chose to be measured, and the direction of that choice is a standing caveat, as volunteer bias in size studies explains. Online surveys contain whoever found the survey, and online surveys and the numbers they produce shows why their figures sit above measured ones.

Write down the recruitment route and the sample size. A small sample is not wrong, but small single studies swing by sampling alone, and its interval will say so if you read it.

2. Who held the ruler?

The second question is the one that moves figures most. Either a trained observer took the reading or the participant took it on themselves.

The gap between the two is not mainly dishonesty. It is angle, motive and landmark, which self-measured versus clinician-measured lays out. A study that mixes both without separating them is reporting an average of two different quantities.

A paper should state who measured and ideally whether more than one observer did. Where two observers measured the same men, inter-rater agreement in the measured studies is the figure that tells you how much of the spread is just observers differing.

3. Which state, which method?

A length is not a length until it has a state and a landmark attached. Flaccid, stretched and erect are three different quantities, and bone-pressed and non-bone-pressed are two different measurements of each. There is no single average length, because the word "average" names none of these.

Check each of these in the methods section:

• the state: flaccid, stretched or erect, and how erection was obtained
• the landmark: whether the ruler was pressed to bone
• the line: along the top, or somewhere else
• the instrument: ruler, tape or something else
• for girth, the site on the shaft

If any is missing, the figure is incomplete. A method change alone can shift a whole study's mean by more than its interval, which how a method change shifts a whole study mean shows.

4. Who was excluded?

Exclusions are correct for a study's purpose and still shape its numbers. A study of measurement excludes men whose condition would distort the reading, which trims the tails. Who studies exclude, and why it shifts the mean walks through the effect.

Look for an explicit list. A paper that excludes nobody and does not say so is hiding a decision, not avoiding one.

5. How was it rounded, and how precisely?

A reported mean is rounded to match its uncertainty. Treating it as exact over-reads it, and rounding in published tables covers this.

The same lens applies to your own conversions. A figure converted from centimetres to inches and quoted to two decimals carries the precision of the original, not of the decimals, as 15.24 cm is not more precise than 6 inches points out.

6. What spread is reported, and which kind?

Look next to the mean for a second number. It is an SD, an SE or a CI, and which one it is changes what the mean tells you.

A standard deviation describes individuals. A confidence interval bounds the mean. The error column in a published table and confidence interval versus standard deviation set out the difference, and the confusion between them is the most common misreading of a results table.

A large sample makes the interval around the mean small. It does nothing to the spread of men, which is exactly what what 15,521 men buys you argues.

7. What is the number being used to claim?

The last question is about the use, not the study. A figure from one population, state and method can support a statement about that population, state and method, and about nothing wider.

This is where press stories fail most often. A percentile chart is redrawn from one study or a self-report survey without saying which, as how press percentile charts get made describes. A country league table mixes self-report and clinical studies, then ranks them to a decimal, as country league tables shows. Neither the table nor the chart will tell you the number was never comparable.

The worked contrast: Veale et al. (2015)

The standard reference here is Veale et al. (2015) in BJU International, a systematic review and meta-analysis that constructed nomograms from up to 15,521 men. Running the checklist on it is useful because it passes some questions cleanly and fails others honestly.

• Sample: pooled clinical samples, with a minimum sample size per study.
• Who measured: a health professional, using a standard procedure.
• State and method: reported separately for flaccid, stretched and erect, though not every pooled study pressed to bone, as how many studies press to the bone records.
• Exclusions: men with a complaint of small size, erectile dysfunction or previous penile surgery, among others.
• Spread: reported as standard deviations, with nomograms drawn from them.
• Claim: a reference distribution, not a prediction for any individual.

It does not pass everything, and the authors say so. Veale 2015: the limitations the authors list covers heterogeneity, clinic-based samples and uneven erect data. A study that lists its own weak points is one you can weigh; one that does not has not necessarily got none.

The contrast is the lesson. A self-reported survey fails questions two and three on sight, because nobody checked who held the ruler or what state was meant. The meta-analysis does not become perfect by comparison. It becomes legible, which is the real difference.

Common traps when the checklist is applied

Four mistakes recur even for careful readers.

Reading "normal range" as approval. In a results table it means the middle of the sample, a description and not a judgement, as normal range is a statistical term first explains.

Trusting one striking study. Studies with unusual findings are likelier to be published and cited, which publication bias in the size literature covers, so a surprising single result deserves more suspicion, not less.

Assuming the pooled mean describes you. A meta-analysis is an average of its inputs and inherits their methods and biases, and what a meta-analysis is, and is not says what it can and cannot be. When the pooled studies disagree more than chance allows, heterogeneity is the term for it, and the pooled figure is an average of different things.

Comparing a self-measured number straight to a table. The two were taken under different conditions, and the correct move is to write your own method line first, as write the method next to the number describes, and only then look for a comparable row.

None of these needs new statistics. Each is a question you can answer by reading the methods section slowly.

Run it on a headline

Take a headline giving an average for some country. Ask each question and see where the story goes quiet.

Usually it is silent on who measured, which state and which method, and on what the spread was. Those silences are your answer. The number may still be true of something, but it is not yet a figure you can place yours beside.

If you want to compare your own reading, the route is a figure taken with the method the study used, written with its method line, placed against a reference whose population you have named. The same discipline applies to adjacent claims. When a story quotes a score from a photograph, as Rate Cock produces, it is reporting a rank among images, not a distance, so none of the seven questions above applies to it in the same way; how an AI scoring system works is a separate topic with a separate checklist. A tool that publishes a score distribution of photographs, or a site built around human judges, is describing opinion about pictures, and a length taken with a ruler is a different kind of claim.

Keep one written line per question when you do this, for example "recruited from clinics, health professional measured, erect and stretched reported separately, SD reported". Seven short lines make two studies comparable at a glance, and they also make plain when a study cannot be compared with another at all, which is a legitimate outcome. If a paper leaves a line blank, leave it blank in your note and say so, rather than filling it with a plausible guess.

The checklist is short enough to keep beside any paper. Seven questions, answered in writing, will remove most of the confusion before it reaches a chart.