Reference
Measurement glossary
The vocabulary this lab measures with — terms either coined here or used in a stricter sense than common usage. Every definition is self-contained and links to the methodology section or evidence that grounds it.
- Useful context depth
- The context length at which a system still passes quality gates — retrieval, honesty, format — not the length it can merely allocate memory for. A model may accept a 128K window yet stop finding facts, or slow beyond usability, far earlier. Configured, loadable, completed, quality-preserving and recommended depth are five different numbers; vendor sheets usually quote the first, users experience the last.
- Grounding: /workloads/long-context-v1/
- Evidence level
- A label carried by every published number stating how much measurement stands behind it: lab_single_run (one valid run), lab_repeated (three repeats on one unit), lab_unit_replicated (reproduced on a second identical unit), external_reproduced (reproduced outside the lab), plus legacy, vendor_disclosed and estimated. A number below three runs is published only with its lower level stated next to it — never presented as repeated.
- Grounding: /methodology/#repeats
- Claim registry
- The only source of numbers on this site. Every published figure is re-derived in CI from raw per-request run records by a versioned DuckDB query; a hand-typed number cannot exist, and a published number cannot silently drift from its evidence. Each claim has a permanent page and a machine-readable JSON twin.
- Grounding: /claims/
- Controlled comparison
- An A-versus-B claim where both sides were measured on the same physical unit, under the same frozen workload revision and corpus, with exactly one variable changed — backend, artifact, flag. Comparisons that cross systems or versions without this control are labeled diagnostic and cannot back verdicts.
- Grounding: /compare/
- TTFA — time to first answer token
- Client-side time until the first token of the ACTUAL answer arrives — as opposed to TTFT, which fires on the first token of anything, including a reasoning stream the user never asked to read. On reasoning models the two diverge by orders of magnitude, and TTFT alone makes an assistant look instant while the user stares at a spinner.
- Grounding: /reports/ttft-thinking-model-strix-halo/
- Answerless request
- A request that returns HTTP 200 with an empty answer because the reasoning pass consumed the entire token budget before producing any user-visible text. Invisible to availability monitoring — the server reports success — and only caught by counting answer tokens. A failure mode this lab measures and keeps in the denominator.
- Grounding: /claims/strix.qwen36.interactive2.c1.answerless-1k/
- Grounded decline gate
- The honesty check for unanswerable controls: a question whose answer is absent from the document while a similar distractor fact is present. Every code-shaped value in the answer must be quotable from the source document — citing the in-document distractor while declining is the ideal answer; a value that appears nowhere in the source is the fabrication the control exists to catch.
- Grounding: /errata/
- Host quiescence
- The requirement that nothing else on the measurement host competes for the GPU or memory bandwidth during a run. The harness snapshots the container inventory at the start AND end of every run and records both in the manifest; a GPU-capable neighbor at either point invalidates the cell. A shared host is not a measurement host.
- Grounding: /methodology/
- Evidence bundle
- The sealed directory a run leaves behind: manifest with full cell identity, per-request records including failures, per-request quality-gate outcomes, harness-computed aggregates, the server log, and checksums over all of it. Sealed bundles are never edited retroactively; a rerun gets a new run ID. All published bundles are public.
- Grounding: /data/
- Prompt-cache session effect
- The gap between the first and second question over the same pasted document. With the server prompt cache on, the second question reuses the document prefill and answers near-instantly; with it off, every question pays the full prefill again. The effect requires a byte-identical prefix — an edited document restarts from zero.
- Grounding: /compare/prompt-cache-on-vs-off-doc-session/
- Needle ladder
- A long-context probe that varies document LENGTH while holding the needle position fixed (~50% depth): one synthetic fact per document, retrieval verified by exact containment, paired with unanswerable controls. It measures whether retrieval survives depth — deliberately the easiest honest probe, so a failure means something and a pass claims little.
- Grounding: /workloads/long-context-v1/
- Truth reset
- The founding policy of this site: no number appears in copy by hand. Every figure renders from the claim registry with its evidence level, scope and limitations attached; historical numbers quoted from published sources are explicitly marked as quotes; forbidden claim language (certification wording, unscoped production-ready, hallucination-free promises) is blocked by CI.
- Grounding: /methodology/
- Goodput (vs throughput)
- Requests per second that completed AND met the agreed latency objective — as opposed to throughput, which counts everything the server pushed out regardless of whether a user would have waited that long. Capacity stated without an SLO is throughput theater; failed requests stay in the denominator here.
- Grounding: /workloads/team-serving-v1/