The measured answer up front: with the llama.cpp prompt cache doing its job, the second question over the same document comes back almost immediately, while the first one pays the whole prefill. With the cache off, or broken by your pipeline, every question pays it. The numbers and the one condition the cache needs are below.
Everyone who has pasted a long file into a local assistant knows the rhythm. The first answer takes ages. Every answer after that comes back almost before you finish typing. Forum threads explain this with hand-waving about caching and leave it there. We measured the actual gap on one box, with the runtime pinned, three repeated runs, and the raw records published.
What was measured
One Beelink GTR9 Pro with a Ryzen AI Max+ 395 and 128 GB of unified memory, running llama.cpp server on the Vulkan backend, build pinned by image digest. The model is Qwen3.6-35B-A3B Q4_K_M pinned by artifact hash. The workload is a frozen document session: load an English document, ask a question, then ask a second question over the same document. Two document sizes, 8k and 32k tokens. Reasoning is off. The only toggle under study is the server prompt cache.
How much does the prompt cache actually save?
The first question over the 32k document costs 33940msrepeated. That is the prefill bill: the model reads the whole document before it can say anything, and no cache in the world helps a document it sees for the first time.
The second question over the same document, with the prompt cache doing its job, costs 860msrepeated. The document is already digested; the server only processes what changed.
Now the control cell. Same second question with the cache off: 33728msrepeated. The full prefill, paid again, for a document the server had just read. This is the configuration a surprising number of local setups quietly run in.
On the smaller 8k document the cached second question comes back in 660msrepeated, so the cached path barely notices document size. The uncached path scales with it.
When does the llama.cpp prompt cache actually hit?
The cache matches by prefix, byte for byte, within a live server session. Anything that touches the bytes before your new question throws the savings away: a system prompt that embeds the current time, a RAG pipeline that re-ranks and reshuffles retrieved chunks between turns, a proxy that rewrites history, a server restart. The document must arrive identical, in the same position, every turn.
This is why “my follow-ups are still slow” reports are usually not a hardware problem. The box is fine. Something in the middle is rewriting the prefix, and every question pays the first-question price.
What this changes in practice
For a personal document assistant on this class of hardware, the workflow that works is: pay the fill once, then stay in the session and ask everything you have. The workflow that hurts is bouncing between documents, or any pipeline that rebuilds context per request. Before buying faster hardware to fix slow follow-ups, check what your stack does to the prefix. The fix may be free.
What this does not say
- Nothing here is about answer quality. The session workload gates format, language and repetition; it does not grade comprehension.
- One model, one quant, one backend, one box. The shape of the gap is what transfers; exact figures do not.
- This is llama.cpp server behavior. Other runtimes cache differently, and we have not measured them here.
The side-by-side matrix lives on the comparison page, the tested-configuration verdict on the doc-session card. Every figure above has a permanent page with scope, limitations and raw runs: the claim registry.