Strix Halo × llama.cpp Vulkan × Qwen3.6-35B-A3B — document session cache

The question this card answers

Our pasted-document card established that a 32k-token document costs tens of seconds before the first token. This card answers the follow-up that decides whether document work is usable at all: does the second question over the same document pay that price again?

Verdict: PASS WITH LIMITS. With the server prompt cache enabled, it does not — by more than an order of magnitude. The limits define exactly when that holds.

Results

The first question over a 32k-token document pays the prefill regardless: 33940msrepeated to the first token. The second question over the same document:

• prompt cache disabled — 33728msrepeated: the full prefill is paid again, every question;
• prompt cache enabled — 860msrepeated: the document’s computation is reused and only the new question is processed. On the 8k document the cached second question starts in 660msrepeated.

Answers passed the format, language and repetition gates in both cells — the cache changed the wait, not the output discipline.

Operating guidance implied by the evidence

For document work on this configuration, run the server with the prompt cache enabled and keep one session per document: the half-minute cost is paid once at load, and the conversation is fluid after that. An application that reconstructs the prompt from scratch per question — or prepends a changing system message, breaking the shared prefix — silently returns to paying the full prefill every time.

The limits in “pass with limits”

• Byte-identical prefix only. The reuse works because the document prefix is identical between questions; any edit to the document or to text before it invalidates the cache.
• Single user, single session. Cache behaviour under concurrent sessions, memory pressure from multiple cached documents, and cache isolation between users were not measured and no claim is made.
• Our standard latency cells keep the cache off deliberately, so that no request is answered from another request’s work; this card measures the deliberate opposite and says so in its cell identity.

Revalidation triggers

Runtime image change (cache implementation), model artifact change, workload revision change, driver/kernel change. Harness and corpus: agmind-bench.