Repeatability and agreement, start to finish

A measurement repeats when the same person with the same instrument gets nearly the same number again, and it agrees when a different person or a different instrument lands close to that number. Repeatability and agreement are separate properties, and neither tells you whether the number is right. This guide defines the vocabulary once, then shows how each property is measured and what the result lets you say.

Three words that get used as one

Forum threads use "consistent", "accurate" and "reliable" as if they were interchangeable. In measurement work they are not, and the first step to a usable number is keeping them apart.

Repeatability is the closeness of readings taken under the same conditions: one person, one instrument, one method line, one short stretch of time. It is the smallest spread you can get, because nothing is allowed to change except the act of taking the reading again.

Reproducibility is the closeness of readings when something is deliberately changed: another person, another instrument, another day, another room. It is always at least as wide as repeatability, because every extra source of variation can only add to the spread.

Agreement is the practical question behind reproducibility: when two readings of the same thing come from two sources, how far apart can they plausibly be? It is expressed as a difference, in centimetres, rather than as a correlation, and that choice matters more than it first appears.

None of the three says anything about accuracy, which is closeness to the true value. Five readings that all sit 3 mm long because of a bad zero are perfectly repeatable and perfectly wrong, a distinction taken apart in precision versus accuracy. The Joint Committee for Guides in Metrology's Guide to the expression of uncertainty in measurement (JCGM, 2008) is the standard international reference for treating these as separate contributions to the uncertainty of a result, and the same separation is used throughout this site.

Repeatability: one person, one instrument

The cleanest repeatability question is this: if I measure the same thing three times in a row, how far apart will the readings be? The answer is a property of your technique and your instrument together, not of the object.

The usual summaries are the range of a small set of readings, or a standard deviation if you have enough sessions to compute one. Three readings give a range you can trust only loosely. Five sessions of three readings give you something you can state with a straight face, which is why a repeatability test you can run this week uses that design and walks the arithmetic through by hand.

Two details decide whether the number you get is honest.

First, each reading has to be a fresh attempt: remove the instrument, reposition it, read again. If you leave the tape in place and re-read the same mark, you are measuring your eyesight, not your technique. The pull of the previous number is a real effect, covered in anchoring on the last number.

Second, conditions have to be held still. Same state, same posture, same time of day, same landmark. A repeatability figure collected under drifting conditions is quietly a reproducibility figure, and it will look worse than your technique deserves.

Reproducibility: change one thing at a time

Reproducibility is repeatability with a deliberate change introduced. The useful way to study it is to change exactly one factor and watch what the spread does.

Changing two things at once makes the result uninterpretable. If a friend measures with a different tape on a different day and the number moves, you cannot say which change did it.

The general lesson is that the spread you see grows as you let more things vary, and that the honest width of a reported number is the width under the conditions in which it will be used. A figure you intend to compare with someone else's has to carry reproducibility, not just repeatability.

Agreement between two readers

Two readers measure the same object with the same instrument under the same protocol. Each reader's own readings are repeatable to some degree; the question is whether the two sets sit on top of each other.

The wrong way to answer it is to correlate the two sets. A correlation coefficient measures whether the readings rise and fall together, not whether they match. Reader B could read a steady 5 mm longer than reader A on every occasion and the correlation would be nearly perfect, while the two would never produce the same number. Bland and Altman made this argument in a 1986 paper in The Lancet, and it remains the reason agreement is reported as a difference.

The right way is to look at the differences directly. For each pairing, subtract one reading from the other. Two numbers then summarise the lot: the mean difference, which is the systematic gap between the readers (the bias), and the spread of the differences, which says how far apart any single pair can land. The convention in Bland and Altman's paper is to report the mean difference plus and minus 1.96 times the standard deviation of the differences, which they call the limits of agreement. If the pair are statistically well behaved, about 95 per cent of future differences should fall inside that interval.

What this gives you is a statement a reader can use: "a second reader's number will typically sit within 0.4 cm of mine, with the second reader running 0.1 cm long on average". That is much more useful than "we agree well", and it is the form the plot in Bland-Altman agreement plots draws.

A worked sketch, with invented numbers only to show the steps. Six paired readings of the same object by two readers give differences of +0.2, +0.1, +0.3, 0.0, +0.2 and +0.1 cm. The mean difference is +0.15 cm. The standard deviation of the six differences is about 0.11 cm. The limits are then +0.15 plus or minus 1.96 times 0.11, roughly -0.07 to +0.37 cm. Six pairs is far too few to trust those limits, but the arithmetic is the whole method, and more pairs only tighten it.

Agreement between two instruments

The same logic applies when the thing that changes is the instrument rather than the person. One reader, two tapes, or a tape and a rigid ruler, on the same object.

The new complication is that the interesting output is the bias. If tape A reads 0.2 cm longer than tape B on average, that is a systematic difference, and it points at one of the two instruments, which is why a second instrument helps even though it cannot say which one is wrong. A third reference, a known length such as a printed scale or a card of known dimensions, is what breaks the tie.

Instrument disagreement also tends to depend on the reading. A tape that stretches reads more wrong the harder you pull, and a ruler with an offset end is wrong by the same amount everywhere. If you plot the differences against the size of the reading and see a slope, the disagreement is not a constant bias, and a single mean difference will mislead.

Intraclass correlation, briefly

When researchers report agreement or repeatability for a whole study, the figure they usually give is an intraclass correlation coefficient, or ICC. It is a ratio: the share of the total variation that comes from real differences between the people being measured, as opposed to differences between readers or occasions. A value near 1 means the readers add little noise relative to the differences between subjects; a value near 0 means the noise swamps them.

The detail that trips readers up is that "the ICC" is not one thing. Koo and Li, in a 2016 guideline in the Journal of Chiropractic Medicine, set out how the form of the coefficient has to be chosen to match the study design, including whether raters are a fixed set or a sample of possible raters, and whether a single reading or an average is the unit. Quoting an ICC without naming the form is close to quoting a length without naming the unit.

Two further cautions. The ICC depends on how different the subjects are from each other, so the same reader noise gives a higher ICC in a varied sample than a homogeneous one, and ICCs from different samples are not directly comparable. And a high ICC can sit alongside a large systematic gap between readers, because it is not built to catch constant bias. That is why the Bland-Altman difference approach and the ICC are normally read together, and intraclass correlation explained works through the pairing.

How studies handle this

Clinical size studies use trained observers, written protocols and a rigid ruler or tape precisely to push the reproducibility figure down. Some report inter-rater agreement; many do not. Inter-rater agreement in studies covers what the published figures look like, and measurement error inside the studies themselves covers why pooled numbers carry that error as a floor.

This matters for your own readings. The pooled reference in Veale et al. (2015) in BJU International is built from measurements taken by health professionals, not self-reported, and your own number carries your own repeatability on top of whatever method differences separate your setup from theirs. Setting your result beside that reference makes sense only once you know how wide your own error is.

A plan for testing your own

Here is the shortest sequence that covers all three properties.

1. Write the method line first: state, landmark, instrument, site, rounding. The method-line convention is the format.
2. Run the five-session repeatability test. That gives you a within-session and a between-session figure.
3. If a second person will ever take a reading, have them take a set under the same method line, blind to yours, and compute the differences.
4. If you own two instruments, read the same object with both and look at the gap.
5. State the result as a number with a range and the conditions it covers, not as a bare figure.

For how many readings each session needs, see how many times to measure. The full accounting of where error comes from, which this guide only touches, is in the error budget in full.

What agreement cannot tell you

A tight agreement figure says two readers or two tapes behave alike. It does not say either is correct. Two people can share the same misreading of a landmark and agree beautifully. Only a check against a known length, or against a protocol with a definition both readers follow, addresses that.

It also says nothing about whether the quantity is stable. If the thing measured changes from one hour to the next, agreement between readers taken an hour apart is mixed with real change, and no amount of statistics can separate the two after the fact.

Agreement figures are also local. They describe these readers, this instrument and this protocol. A different protocol can move them, which is the point of fixing the protocol before you measure.

Where other sites pick up

A reading error is a different kind of number from a score. If the question is how a photo-based tool states the accuracy of what it outputs, that tool's owner answers it. How a ten-point score is built is a separate construct from anything here, and how human judges vary is its own subject, with its own agreement statistics. The usual vocabulary of repeatability and agreement carries over to those cases, but the instrument is a person looking at a picture, which is a different problem, and the one a service like Rate Cock deals with in its own way.

For the measurement side, what carries over is simple. Report the spread alongside the number, say what was held fixed, and say what was allowed to vary. A figure that comes with its conditions and its range can be compared with another; a bare one can only be believed or not.