The one-paragraph version: a benchmark number is worth publishing when a stranger can disagree with it — reproduce the setup, get a different value, and locate the reason in the config delta. That requires the number to carry its configuration, its workload, its failures and its spread. Everything below serves that single property, and every rule earned its place by burning us first.
This is the protocol behind our published numbers, written for anyone posting a local-LLM measurement to a forum, a README or a blog. The harness that implements it is public; none of it requires our tooling.
Ask the running server what it runs
Not your launch script — the server. llama.cpp answers on /props with its
effective settings; vLLM prints its resolved configuration at startup.
Defaults change between releases, and the flag you did not set is still part
of your config. The sharpest measurement error in
our own journal came from a parallel-slots
default that changed between versions and silently divided per-stream
throughput before the benchmark began. Your memory of the command line is
not evidence; the process’s own answer is.
Pin what you ran, byte-level
Runtime by image digest, not tag — tags move. Model by artifact hash, not name — repositories re-upload. Device firmware named explicitly — the DGX Spark discourse flipped its verdict on the same hardware after a firmware update, which means undated Spark numbers are now unattributable gossip. If a reader cannot tell which bytes produced your number, they cannot disagree with it, only argue about it.
Freeze the workload and name it
The largest spread in our archive comes not from hardware or flags but from workload shape: prompt length, output length, content class. On the same box and build, a short-output profile and a heavy-context profile produce numbers that differ by more than most hardware upgrades — reconciling published DGX Spark figures mostly meant discovering which unstated profile each one used. Fix the corpus, fix the sampling settings, and publish the profile next to the number. A tok/s figure without its workload is unfalsifiable.
Count the right tokens, from the right side
Three separate traps live here. Count output tokens from the API’s
usage.completion_tokens, never from streaming event count — under
speculative decoding one event can carry several tokens and the event count
silently inflates throughput. Say which side of the wire you measured:
client-side includes the network and the server’s queue, engine-side does
not, and the two disagree under load. And for reasoning models, decide
whether your first-token metric means the first token of anything or the
first token of the answer — the gap between those definitions is
sometimes the entire result.
Keep failures in the denominator
A request that returned HTTP 200 with an empty answer is not a lost sample; it is a result. Dropping failed, empty or malformed responses before averaging quietly turns a reliability problem into a speed bonus — the server that fails fastest wins. We grade every response with format, language and repetition gates and publish the share that produced nothing alongside the latency of the ones that did.
Repeat, and publish the spread
A single pass is an anecdote with decimals. Our floor for a quotable number is three repeated runs on one unit, and where it matters we repeat the pattern on a second commercially identical unit. On some hardware classes identical runs disagree by margins that dwarf the effects being argued about — single-decimal precision from any single pass, ours included, is optimism wearing a lab coat.
Quiet the host, and prove it
Nothing else runs on the box during a measurement — and “nothing else” is a claim, so record it. Our manifests carry a host inventory snapshot before and after every cell; a GPU neighbor appearing mid-run invalidates the cell. Check the server log for silent CPU fallback while you are at it: a model that quietly left the GPU produces beautifully reproducible numbers about the wrong device.
Publish so you can be disagreed with
Raw run records, the exact config, the workload identity and the number — together, at a stable address. That is the entire design of our claim pages: value, scope, limitations and runs share a permalink. And when someone’s number disagrees with yours, treat it as data: the config delta behind a disagreement is usually more informative than either figure. Ours have a standing submission path.
The checklist
Before posting a number, it should answer: what device and firmware, what runtime and digest, what model and hash, what quant, what workload shape and sampling, what concurrency, which side of the wire, how many repeats, what got excluded and why. If any answer is “not sure”, the number is not ready — and if all answers are present, even a number that later proves wrong is useful, because it can be found wrong. That is the whole difference between a measurement and a mood.