Methodology v1

1. The qualification cell is the unit of scope

Every result belongs to a qualification cell: one exact combination of commercial device, firmware and BIOS, OS and kernel, GPU driver stack, runtime build, model file identified by hash, quantization, context and KV settings, launch command, and workload revision. The full system fingerprint is captured before measurement.

Changing any one critical layer creates a new cell. Results never silently carry over to a different driver, runtime build, model revision or device variant. A public configuration card describes the tested cell — not every version of the device.

2. Preregistration before headline runs

Before any headline run, the following are fixed in a preregistration document:

• the primary question and the systems with their critical versions;
• the workload revision and the model selection rule;
• the cells, the number of repetitions, and the headline metrics;
• quality gates and exclusion/invalidation rules;
• planned comparisons and the conditions for extending scope.

After preregistration, the headline metric, exclusion rules and configuration cells cannot change without a new revision. Exploratory runs are allowed, but they are labeled separately and never substitute for a preregistered result.

3. Run lifecycle and invalidation

Every run moves through a fixed lifecycle:

A run becomes invalid — kept in the journal, excluded from headline claims — when, among other conditions:

• the runtime or model hash is unknown or does not match the preregistered revision;
• a silent device fallback occurred — for example, a workload intended for the GPU quietly executed on the CPU;
• warmup traffic leaked into the measured interval, or a cold start was mixed into warm serving latency;
• the operator changed the system configuration during the run;
• the target output geometry was not held, telemetry clock drift made time series incomparable, or the benchmark client exhausted its own resources;
• the number of failed requests cannot be determined.

Failed requests never disappear from the denominator. A request that timed out or errored is counted against the run even if it produced no timing data. Per-cell request accounting — scheduled, started, completed valid, functional failures, timeouts, transport errors, server errors, invalid outputs, cancelled — is published alongside the result. After an OOM or crash, a run is not continued as valid; recovery requires a new run identifier.

4. Repeats and spread

A headline cell requires at least three independent measured runs. Individual run values are published, and the headline number is the median unless the workload methodology defines otherwise.

If run-to-run spread is wider than the internal quality-control threshold, additional runs are performed or the cause is identified before the number is published. That threshold is an internal QC rule of AGmind, not an industry standard.

Tail percentiles are reported only when the count of valid observations meets internal minimums, with the sample size shown next to the value. With a smaller sample, the raw distribution, median and range are shown instead, with an explicit insufficient-sample marker. Timeouts and errors remain visible in every distribution.

5. Claim types: controlled, system, diagnostic

• Controlled — exactly one factor varies between conditions. Relative statements about that factor are allowed.
• System — complete configurations are compared as they ship. Only side-by-side observed values are allowed, with every known difference disclosed; the result is not attributed to any single component.
• Diagnostic — microbenchmarks and probes. Their results are never extrapolated to user-facing capacity.

6. Quality floor

Every performance recommendation requires pre-declared quality or functional gates — for example: server/API contract, JSON and schema validity, output language, repetition and collapse detection, task correctness, retrieval and citation support, function execution, abstention on unanswerable inputs. If no gate was evaluated, the result carries an explicit label:

7. Long context: five distinct levels

“Supports long context” is not a single property. The methodology distinguishes:

1. Configured context — what the runtime is set to accept.
2. Loadable context — what actually loads without failure.
3. Successfully completed context — what completes requests end to end.
4. Quality-preserving context — where output quality holds against the short-context control.
5. Recommended context — what AGmind recommends under a specific workload and SLO.

The minimum long-context test set includes a short-context control, position-balanced tasks, multi-hop and aggregation tasks, an unanswerable control, repetition/collapse detection, and a latency/memory curve across context lengths.

8. Endurance minimum

A public endurance result requires a sustained active-load session of at least the fixed minimum duration defined by the methodology, followed by a separate drain phase, on a fixed workload and cell. Throughput, latency and errors are reported per time bucket, alongside temperature, clocks, utilization and power source. Throttling events, memory growth, and any restart, crash or recovery are recorded in the run journal.

9. Unit-to-unit replication

When the lab has two identical commercial units, headline anchor points are repeated on the second unit. This does not make the sample representative of an entire production batch, but it separates run-to-run repeatability from the possibility of one anomalous unit.

10. Verdict vocabulary

Verdicts come from a fixed vocabulary. A verdict always refers to a specific version, workload and acceptance criteria — never to a device in general.

11. Corrections, superseded and retracted claims

Corrections are published, not hidden. Every correction carries a date, a reason and the list of affected claims. The affected claim is marked superseded or retracted; the old report remains accessible, or is archived with an explicit redirect. Corrected figures are recomputed from the underlying data, never edited by hand. Material errors are listed on the errata page, and headline pages are updated from the claim registry.

12. What this methodology does not claim

• conformance with MLPerf or any other external benchmark program;
• statistical representativeness of the entire market of devices, or of a production batch;
• suitability for any workload that was not tested;
• product safety or regulatory-compliance conclusions of any kind;
• long-term support of future firmware, driver, runtime or model versions;
• causality, whenever more than one factor changed between conditions.