Time-to-first-token does not describe a thinking model

Evidence level: lab_repeated. Every number below is re-derived from the raw per-request records by the query named in the claim registry — none of them is typed into this page. Depth is diagnostic: three measured runs per operating setting at concurrency 1, on one unit. This is the pipeline-proof study that precedes the flagship, not the flagship itself. Unit-replicated numbers follow with the frozen preregistration.

The short version

A model that reasons before answering breaks the metric everyone quotes.

On the tested configuration, the conventional time-to-first-token looks excellent — a fifth of a second. But that first token is a reasoning token. The first token of the actual answer arrives roughly a hundred times later, and under a common token budget most requests never produce an answer at all: the caller gets HTTP 200 and an empty string.

None of this is visible in a benchmark that reports tokens per second.

What was measured

Timing is client-side: the clock starts when the request leaves and stops on the tokens as they arrive. Server-reported timings would have hidden exactly the effect this report is about. Every request carries a unique salt and the prompt cache is off, so nothing is answered from a previous request’s work.

Three operating settings were run, changing one thing at a time:

1. reasoning enabled, 1024-token budget;
2. reasoning enabled, 4096-token budget;
3. reasoning disabled via the chat template, 1024-token budget.

Results

The conventional first-token figure is identical in all three settings — around 221msrepeated. It cannot tell apart a configuration that answers immediately from one that never answers.

With reasoning enabled, the answer starts far later. At the 4096-token budget the first answer token arrives at a median of 23453msrepeated. The user watches tokens stream for that entire time — they are the model thinking out loud, not the reply.

At the smaller budget, most requests produced no answer at all. With reasoning enabled and a 1024-token budget, 75.0% of requestsrepeated of requests returned a successful HTTP response with an empty answer: the reasoning pass had consumed the whole budget before the answer began. An application checking status codes would have recorded these as successes.

Disabling reasoning collapses the gap. With the reasoning block switched off, the first answer token arrives at 220msrepeated — the answer starts with the stream. Median end-to-end time dropped by roughly five times against the 4096-token reasoning cell, and the answers still passed the language and repetition gates.

One gate failed in every setting. Half the answers exceeded the word budget declared per prompt. That is a property of the model’s verbosity, not of the reasoning setting, and it is why this configuration would not receive an unqualified pass on this workload.

What this means operationally

• A first-token number without a definition is not usable. Ask which token: the first one streamed, or the first one belonging to the answer. On a reasoning model these are different questions with a hundredfold difference in the answer.
• A token budget is a correctness setting, not just a cost setting. Below the budget the reasoning pass needs, the endpoint returns success with nothing in it. Anything monitoring only status codes will not notice.
• Empty-answer responses belong in your error rate. In this study they are counted as failures and stay in the denominator of every latency statistic.

What this does not establish

Not a capacity claim: concurrency 1 only. Not a quality verdict: the gates cover format, language and repetition, not correctness. Not a device claim: one unit, one image digest, one artifact. Repeated but not replicated: three measured runs per setting on the same unit, which is why the evidence level is lab_repeated and no operating envelope is recommended here. Whether disabling reasoning costs answer quality is untested — it plausibly does, and that is a separate qualification.

Evidence

Each run produced a bundle: run manifest with the full identity, per-request records including failures, per-request gate outcomes, the runtime log and a checksum file. The numbers in this report are re-derived from those records by SQL queries stored beside the claims; the build fails if a rendered value stops matching its evidence. How that works is described on the evidence and data page.