Watch any local-LLM forum long enough and you will see the ritual. Someone posts the speed of a model on a popular box. Someone else replies that theirs is nearly twice as fast, or half as fast, same model, same box. Accusations follow — wrong build, broken install, shilling. Almost every time, both numbers are real. Both were measured. They describe different systems that happen to share a product name.
A benchmark number is a configuration, compressed down to one value. The compression is lossy, and everything the compression dropped is where the disagreement lives.
The default you didn’t set is still your config
The sharpest specimen in our own journal: a benchmark that showed a 2.6-fold gap between two deployments of the same image on neighboring nodes. Days of tuning theories died to one request against the running server: a runtime default for parallel slots had changed between versions, and per-stream throughput was being divided before measurement ever began. The full story is in the Luddites essay; the moral survives compression better than the number did. You did not set the flag, so you did not mention it when you posted the number. The flag was still there.
llama.cpp, vLLM and their peers ship dozens of such defaults, and the defaults move between releases. Two people running “the same command” a month apart are not running the same command.
A benchmark without a firmware hash is gossip
The DGX Spark discourse this year flipped from “overpriced and throttling” to “the update changed everything” — the same device, praised and buried by the same community, months apart. Which means most Spark numbers circulating today were taken on stacks that no longer exist, and comparing them across that firmware boundary is archaeology, not benchmarking.
We hit the same wall from the measurement side: published speeds for the model we serve on our own Spark pair disagreed with each other so much that we wrote a reconciliation page instead of another number. Most of the spread dissolved once each figure was pinned to its workload profile and runtime configuration. One kernel-selection flag, which the automatic mode deliberately skips, moves the same hardware between performance tiers — and that flag has never once appeared in a forum post quoting the resulting number.
On our Strix Halo box the story repeats at a smaller scale: choosing the wrong backend costs about as much as doubling the weight precision. Version, firmware, backend, kernel flag — none of them fit inside a single number, and all of them are the number.
What a number must carry
Our claim registry has a fixed set of fields, and the set was not designed top-down. Each field earned its place as a documented way two honest measurements diverged: exact device and firmware, runtime with build and image digest, model artifact with hash, quantization, workload identity with prompt and output shape, concurrency, sampling settings, which side of the wire measured, and how the values were aggregated. Strip any one of them and there exists a real pair of contradicting numbers whose contradiction that field explains.
That is the test worth applying to any published figure, ours included: not “is it accurate” but “does it carry enough config to be disagreed with?” A number you cannot disagree with — because you cannot tell what it describes — is not evidence. It is mood.
How to read a forum number
Treat it as an interval, not a point, until the configuration surfaces. If you get to ask one question, ask for the three fields that move results the most in our experience: backend, concurrency, and context depth. Ask what the server actually ran, not what the poster intended — the running process will tell you, and it out-argues everyone’s memory.
And when you publish, invert it. Pin everything, name everything, and put the config next to the number where quoting one without the other takes effort. That is the entire design of our claim pages: the value, its scope, its limitations and its raw runs share a permalink, so the number cannot wander off alone and start rumors.
Numbers do not travel. Configs travel. Quote configs.