Honest statistics for small samples

Handloaders generate more measurement data than almost any hobby and analyze it worse. Three-shot groups become verdicts. Five-shot standard deviations get quoted to two decimals as if they were carved in stone. A wiggle in a ladder becomes a “flat spot” that is really just noise. None of this is stupidity — it is what happens when good instruments meet bad intuitions about small samples. This is the one part of the app that is deliberately not a black box: the statistics are standard, auditable, textbook method, shown in full, because the whole point of a referee is that you can check its calls. And the referee only ever evaluates what you measured — it never predicts.

The central error: a 5-shot SD is a guess dressed as a number

Suppose you fire five shots, run them over a chronograph, and the app reports a standard deviation of 12 fps. The natural reading is “this load has an SD of 12.” That reading is wrong — not because 12 is miscalculated, but because five shots barely constrain the true SD at all.

The sample standard deviation is the Bessel-corrected formula:

$s = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2}$

In words: take each shot’s velocity $x_i$, subtract the string’s average $\bar{x}$, square the differences (so overs and unders both count), add them up, divide by $n-1$ (one less than the number of shots — the Bessel correction, which keeps the estimate from being biased low), and take the square root. That gives $s$, the spread of your five numbers.

The trap is that $s$ is itself an estimate of the true underlying SD, and with only five shots it is a wildly imprecise one. The honest way to show this is a confidence interval on the SD — a range the true value plausibly lies in. For a 5-shot string, the 95% interval runs roughly

$[\,0.6\,s,\ \ 2.9\,s\,]$

Read that again. Your “SD of 12” is really “somewhere between about 7 and 35, with 95% confidence.” Two loads reading 12 and 18 from five shots each are statistically indistinguishable — the difference is almost certainly noise. Loadbook always shows the interval next to the SD precisely so you cannot fool yourself. Pretending a 5-shot SD is a precise number is the field’s central error, and the fix is simply to never hide the uncertainty.

What a confidence interval actually tells you

A confidence interval is not “the range my shots fell in” and not “where the next shot will land.” It is a statement about how well your sample pins down a true underlying quantity — the true mean, or the true SD — given the noise and the sample size. The mean of a string comes with a Student’s-$t$ confidence interval:

$\bar{x} \pm t_{1-\alpha/2,\,n-1}\,\frac{s}{\sqrt{n}}$

Here $\bar{x}$ is your measured average velocity, $s$ is the sample SD, and $n$ is the number of shots. The $\tfrac{s}{\sqrt{n}}$ part is the standard error — it shrinks as you shoot more, which is the whole reason more shots buy more certainty. The $t$ factor is a multiplier from Student’s $t$-distribution that widens the interval for small samples to honestly account for the fact that you also had to estimate $s$ from the same thin data. The single most important reflex to build: a wider interval means less certainty, not more. A tight interval is earned by data; it is never assumed.

Extreme spread is not a consistency metric

The most-quoted number at the range — extreme spread, the gap between your fastest and slowest shot — is the least trustworthy as a measure of consistency, for one blunt reason: it can only grow as you shoot more. It is an order statistic (it depends only on the two extreme values), and the more shots you take, the more chances you have to catch an outlier, so extreme spread creeps up with sample size all on its own. This makes it meaningless to compare across strings of different lengths — a 10-shot ES will tend to beat a 5-shot ES from the same load, purely from counting. The app treats ES as exactly what it is and never compares it across strings of different round counts. Standard deviation, with its confidence interval, is the honest consistency metric.

Comparing two loads

The real question is usually comparative: is load A genuinely faster, or genuinely more consistent, than load B — or do they just look different because of noise?

Are the means different? — Welch’s t-test

To compare average velocities, Loadbook uses Welch’s unequal-variances t-test, which — unlike the older Student’s version — does not assume the two loads share the same spread:

$t = \frac{\bar{x}_A - \bar{x}_B}{\sqrt{s_A^2/n_A + s_B^2/n_B}}$

The numerator is the difference in measured means; the denominator combines each load’s spread ($s_A$, $s_B$) and sample size ($n_A$, $n_B$) into the uncertainty of that difference. A big $t$ means the gap is large relative to the noise. It yields a p-value (the chance of seeing a gap this big if the loads were truly identical) and, more usefully, a confidence interval on the difference — which might tell you the two loads differ by “somewhere between 3 and 40 fps,” i.e. real but poorly pinned down.

Are the SDs different? — a bootstrap

Comparing consistency (the SDs) is harder, because the neat formulas for SD comparison behave badly at realistic sample sizes. So Loadbook uses a bootstrap: it resamples each string with replacement thousands of times, recomputes each SD every time, and directly measures how often one load comes out more consistent than the other. This gives an honest probability that one load is genuinely tighter — no fragile distributional assumptions, just resampling the data you actually have.

How many shots would settle it? — the power advisor

When two loads look different but the test can’t confirm it, the honest answer isn’t “trust your gut” — it’s “you don’t have enough shots yet.” Given the difference you actually observed, the sample-size advisor computes how many rounds per load it would take to resolve that difference at 80% power. It turns “they look different” into “you’d need N shots each to know.”

Drift over a string

Sometimes velocity isn’t random noise around a fixed mean — it trends: climbing as the barrel heats, or shifting with ambient temperature over a session. Loadbook runs a regression-based drift test of velocity against an ordering variable (round count, temperature), reported with a confidence interval — and, crucially, only when the data spans enough range to support the claim. A drift “detected” across a two-degree temperature swing is not evidence of anything, and the honest engine refuses to pretend otherwise.

The creed: measured beats fitted beats estimated

Everything above serves one ranking, the same one that runs through the physics articles:

• A quantity measured from your own firearm, with its honest interval, outranks everything.
• A quantity fitted to published reference data is next.
• A quantity estimated from general principles is last, and honest only when it’s labeled as an estimate and carries an interval wide enough to admit it.

Notice the deep symmetry with the ballistics side. The prediction engines wrap every number in an interval and a grade because a model that overstated its precision would be lying. The statistics engine wraps every number in a confidence interval because a measurement quoted without its uncertainty is lying in exactly the same way. Different machinery, one ethic: never claim more precision than the data earned.

And the boundary holds to the end. This engine evaluates only what you measured — it never predicts, never borrows a modeled velocity to fill a gap, never lets a prediction contaminate the evidence. It is the referee, not the cheerleader. Its job is to tell you, honestly, what your data proves and what it doesn’t — which, for a hobby drowning in three-shot certainties, is the most useful thing a tool can do.