The short, measured answer for this box and these models: one model family found what was in the document and admitted what was not, without exception. The other found almost everything — and when a question had no answer in the document, it sometimes failed not by inventing one, but by saying nothing at all. Both behaviors matter, and the second one is invisible unless you measure for it on purpose.
“Will it hallucinate over my documents?” is the first question anyone asks before trusting a local model with real files, and it is almost always answered with vibes or with leaderboard scores measured on other people’s clouds. Our long-context workload asks it directly, on consumer hardware, with a control group.
What the test actually is
A document goes into the prompt — 2k to 32k tokens, English and Russian — carrying a specific retrievable fact, alongside distractor material chosen to look similar. Two kinds of questions follow. Needle questions have an answer in the document, and the gate checks the model returned that exact fact. Control questions are the point: they ask for something the document does not contain, and the gate passes the model only if it says so. Answering a control question with a confident value is the failure everyone fears; the gate also demands that any code-like value in an answer be literally quotable from the source, so a fabricated one cannot slip through as a near-miss. Failed requests stay in the denominator.
Did the models find what was there?
Qwen3.6-35B-A3B retrieved the needle at 100.0% of requestsrepeated across the depth ladder. gemma-4-26B-A4B came close at 95.8% of requestsrepeated. Retrieval over tens of thousands of tokens is largely a solved problem for this model class on this box — which is exactly why the control group is where the interesting result lives.
Did they admit what was not there?
Qwen3.6 declined every impossible question: 100.0% of requestsunit_replicated on the unanswerable controls. gemma-4 passed 75.0% of requestsrepeated of them — and the failures were not what we expected.
What did the failures look like?
Not fabrication. When gemma-4 failed a control question, the dominant pattern was an empty answer: the model’s reasoning pass consumed the token budget deliberating and produced nothing addressed to the user. HTTP 200, no invented value, no answer at all. That is a materially different risk from hallucination — a pipeline gets a blank instead of a lie — and it connects to a failure mode we measure separately: the answerless request. A model that cannot say “it’s not there” cheaply may instead say nothing expensively.
Does this mean these models never fabricate?
No, and the scope matters. The controls are synthetic: code-shaped facts in constructed documents, a design that makes grading mechanical and fabrication detectable. Adversarial domains, ambiguous questions, contradictory sources — none of that is probed here, and a clean control sheet on this workload is not a general immunity certificate for any model. What the measurement establishes is narrower and still useful: on documents of this shape, at these depths, one shipped configuration declined every impossible question and the other’s failures were silences, not inventions.
How to run this check yourself
The workload contract, corpora and grading gates — including the word-boundary decline markers and the groundedness check — are in the public harness. Point it at your own server and your own documents; if your numbers disagree with ours, we want them.
Every figure above has a permanent page with scope, limitations and raw runs: the claim registry.