# How to benchmark a local LLM so the numbers survive scrutiny

> Most published local-LLM speeds cannot even be disagreed with: they carry no configuration. A practical protocol from a lab that publishes raw runs — what to pin, what to ask the running server, which tokens to count, and why failures belong in the denominator.

- Published: 2026-08-20
- Evidence level: lab_single_run
- Funding: Self-funded internal research
- Canonical: https://agmind.ai/reports/how-to-benchmark-local-llm/

The one-paragraph version: a benchmark number is worth publishing when a
stranger can disagree with it — reproduce the setup, get a different value,
and locate the reason in the config delta. That requires the number to carry
its configuration, its workload, its failures and its spread. Everything
below serves that single property, and every rule earned its place by
burning us first.

This is the protocol behind our published numbers, written for anyone
posting a local-LLM measurement to a forum, a README or a blog. The harness
that implements it [is public](https://github.com/botAGI/agmind-bench); none
of it requires our tooling.

## Ask the running server what it runs

Not your launch script — the server. llama.cpp answers on `/props` with its
effective settings; vLLM prints its resolved configuration at startup.
Defaults change between releases, and the flag you did not set is still part
of your config. The sharpest measurement error in
[our own journal](https://agmind.ai/essays/luddites-were-right/) came from a parallel-slots
default that changed between versions and silently divided per-stream
throughput before the benchmark began. Your memory of the command line is
not evidence; the process's own answer is.

## Pin what you ran, byte-level

Runtime by image digest, not tag — tags move. Model by artifact hash, not
name — repositories re-upload. Device firmware named explicitly — the DGX
Spark discourse flipped its verdict on the same hardware after a firmware
update, which means undated Spark numbers are now
[unattributable gossip](https://agmind.ai/essays/benchmark-number-config-in-disguise/). If a
reader cannot tell which bytes produced your number, they cannot disagree
with it, only argue about it.

## Freeze the workload and name it

The largest spread in our archive comes not from hardware or flags but from
workload shape: prompt length, output length, content class. On the same
box and build, a short-output profile and a heavy-context profile produce
numbers that differ by more than most hardware upgrades —
[reconciling published DGX Spark figures](https://agmind.ai/reports/dspark-speed-numbers-disagree/)
mostly meant discovering which unstated profile each one used. Fix the
corpus, fix the sampling settings, and publish the profile next to the
number. A tok/s figure without its workload is unfalsifiable.

## Count the right tokens, from the right side

Three separate traps live here. Count output tokens from the API's
`usage.completion_tokens`, never from streaming event count — under
speculative decoding one event can carry several tokens and the event count
silently inflates throughput. Say which side of the wire you measured:
client-side includes the network and the server's queue, engine-side does
not, and the two disagree under load. And for reasoning models, decide
whether your first-token metric means the first token of *anything* or the
first token of the *answer* — the gap between those definitions is
[sometimes the entire result](https://agmind.ai/reports/ttft-thinking-model-strix-halo/).

## Keep failures in the denominator

A request that returned HTTP 200 with an empty answer is not a lost sample;
it is a result. Dropping failed, empty or malformed responses before
averaging quietly turns a reliability problem into a speed bonus — the
server that fails fastest wins. We grade every response with format,
language and repetition gates and publish
[the share that produced nothing](https://agmind.ai/reports/answerless-http-200/) alongside
the latency of the ones that did.

## Repeat, and publish the spread

A single pass is an anecdote with decimals. Our floor for a quotable number
is three repeated runs on one unit, and where it matters we repeat the
pattern on a second commercially identical unit. On some hardware classes
identical runs disagree by margins that dwarf the effects being argued
about — single-decimal precision from any single pass, ours included, is
optimism wearing a lab coat.

## Quiet the host, and prove it

Nothing else runs on the box during a measurement — and "nothing else"
is a claim, so record it. Our manifests carry a host inventory snapshot
before and after every cell; a GPU neighbor appearing mid-run invalidates
the cell. Check the server log for silent CPU fallback while you are at it:
a model that quietly left the GPU produces beautifully reproducible numbers
about the wrong device.

## Publish so you can be disagreed with

Raw run records, the exact config, the workload identity and the number —
together, at a stable address. That is the entire design of
[our claim pages](https://agmind.ai/claims/): value, scope, limitations and runs share a
permalink. And when someone's number disagrees with yours, treat it as
data: the config delta behind a disagreement is usually more informative
than either figure. Ours have a
[standing submission path](https://github.com/botAGI/agmind-lab/issues/new).

## The checklist

Before posting a number, it should answer: what device and firmware, what
runtime and digest, what model and hash, what quant, what workload shape
and sampling, what concurrency, which side of the wire, how many repeats,
what got excluded and why. If any answer is "not sure", the number is not
ready — and if all answers are present, even a number that later proves
wrong is useful, because it can be *found* wrong. That is the whole
difference between a measurement and a mood.

---

Machine-readable claim registry: https://agmind.ai/claims.json · llms.txt: https://agmind.ai/llms.txt