# Why chatting with a document is slow once, then instant: the llama.cpp prompt cache, measured

> A document Q&A session on a Ryzen AI Max+ 395 box: what the first question over a 32k-token file costs, what the second one costs with the prompt cache on and off, and the one condition the cache needs before it saves you anything.

- Published: 2026-08-16
- Evidence level: lab_repeated
- Funding: Self-funded internal research
- Canonical: https://agmind.ai/reports/llamacpp-prompt-cache-strix-halo/

The measured answer up front: with the llama.cpp prompt cache doing its job,
the second question over the same document comes back almost immediately,
while the first one pays the whole prefill. With the cache off, or broken by
your pipeline, every question pays it. The numbers and the one condition the
cache needs are below.

Everyone who has pasted a long file into a local assistant knows the rhythm.
The first answer takes ages. Every answer after that comes back almost before
you finish typing. Forum threads explain this with hand-waving about caching
and leave it there. We measured the actual gap on one box, with the runtime
pinned, three repeated runs, and the raw records published.

## What was measured

One Beelink GTR9 Pro with a Ryzen AI Max+ 395 and 128 GB of unified memory,
running llama.cpp server on the Vulkan backend, build pinned by image digest.
The model is Qwen3.6-35B-A3B Q4_K_M pinned by artifact hash. The workload is a
frozen document session: load an English document, ask a question, then ask a
second question over the same document. Two document sizes, 8k and 32k tokens.
Reasoning is off. The only toggle under study is the server prompt cache.

## How much does the prompt cache actually save?

The first question over the 32k document costs
**33940 ms** ([strix.qwen36.docsession.c1.ttft-q1-32k](https://agmind.ai/claims/strix.qwen36.docsession.c1.ttft-q1-32k/)). That is the
prefill bill: the model reads the whole document before it can say anything,
and no cache in the world helps a document it sees for the first time.

The second question over the same document, with the prompt cache doing its
job, costs **860 ms** ([strix.qwen36.docsession.c1.ttft-q2-32k-cache](https://agmind.ai/claims/strix.qwen36.docsession.c1.ttft-q2-32k-cache/)).
The document is already digested; the server only processes what changed.

Now the control cell. Same second question with the cache off:
**33728 ms** ([strix.qwen36.docsession.c1.ttft-q2-32k-nocache](https://agmind.ai/claims/strix.qwen36.docsession.c1.ttft-q2-32k-nocache/)). The full
prefill, paid again, for a document the server had just read. This is the
configuration a surprising number of local setups quietly run in.

On the smaller 8k document the cached second question comes back in
**660 ms** ([strix.qwen36.docsession.c1.ttft-q2-8k-cache](https://agmind.ai/claims/strix.qwen36.docsession.c1.ttft-q2-8k-cache/)), so the
cached path barely notices document size. The uncached path scales with it.

## When does the llama.cpp prompt cache actually hit?

The cache matches by prefix, byte for byte, within a live server session.
Anything that touches the bytes before your new question throws the savings
away: a system prompt that embeds the current time, a RAG pipeline that
re-ranks and reshuffles retrieved chunks between turns, a proxy that rewrites
history, a server restart. The document must arrive identical, in the same
position, every turn.

This is why "my follow-ups are still slow" reports are usually not a hardware
problem. The box is fine. Something in the middle is rewriting the prefix, and
every question pays the first-question price.

## What this changes in practice

For a personal document assistant on this class of hardware, the workflow that
works is: pay the fill once, then stay in the session and ask everything you
have. The workflow that hurts is bouncing between documents, or any pipeline
that rebuilds context per request. Before buying faster hardware to fix slow
follow-ups, check what your stack does to the prefix. The fix may be free.

## What this does not say

- **Nothing here is about answer quality.** The session workload gates
  format, language and repetition; it does not grade comprehension.
- **One model, one quant, one backend, one box.** The shape of the gap is
  what transfers; exact figures do not.
- **This is llama.cpp server behavior.** Other runtimes cache differently,
  and we have not measured them here.

The side-by-side matrix lives on
[the comparison page](https://agmind.ai/compare/prompt-cache-on-vs-off-doc-session/), the
tested-configuration verdict on
[the doc-session card](https://agmind.ai/tested-configurations/strix-halo-qwen36-llamacpp-vulkan-docsession/).
Every figure above has a permanent page with scope, limitations and raw runs:
[the claim registry](https://agmind.ai/claims/).

---

Machine-readable claim registry: https://agmind.ai/claims.json · llms.txt: https://agmind.ai/llms.txt