Controlled comparison
A 2k prompt vs a 32k prompt: what long context costs before the first token
Why is starting with a long document so much slower on a local llama.cpp server, and by how much?
The wait before the first token grows with document length: the model reads everything before saying anything, and the prefill bill scales with what it reads. Retrieval quality held at both depths on this workload. The escape hatch is the prompt cache — a repeated prefix pays the bill once, which is why the second question over the same document comes back almost immediately.
What was held fixed
Same unit, backend, artifact, quant and campaign; the only change is the document length in the prompt. Reasoning off, single stream.
| Metric | 2k-token document | 32k-token document |
|---|---|---|
| Time to first token, median | 1907msmedian over valid 2k-band requests · lab_repeated · evidence | 33965msmedian over valid 32k-band requests · lab_repeated · evidence |
Limits
Measured at 2k and 32k on one model and quant — no claim beyond 32k, and the curve between the two points is not characterized. Prefill-heavy workloads on other backends may scale differently.
Answering this question somewhere? Paste the table
Markdown, renders on GitHub / Reddit / forums. Attribution line included — the data is CC BY 4.0, take it.