# Do local LLMs invent answers when the document doesn't contain them? We measured it

> Everyone asks whether a local model hallucinates over their documents; almost nobody measures it. Our workload has an unanswerable control built in: questions the document cannot answer, graded on whether the model admits it. Two model families, two ways to not know.

- Published: 2026-08-20
- Evidence level: lab_repeated
- Funding: Self-funded internal research
- Canonical: https://agmind.ai/reports/do-local-llms-invent-answers/

The short, measured answer for this box and these models: one model family
found what was in the document and admitted what was not, without
exception. The other found almost everything — and when a question had no
answer in the document, it sometimes failed not by inventing one, but by
saying nothing at all. Both behaviors matter, and the second one is
invisible unless you measure for it on purpose.

"Will it hallucinate over my documents?" is the first question anyone asks
before trusting a local model with real files, and it is almost always
answered with vibes or with leaderboard scores measured on other people's
clouds. Our long-context workload asks it directly, on consumer hardware,
with a control group.

## What the test actually is

A document goes into the prompt — 2k to 32k tokens, English and Russian —
carrying a specific retrievable fact, alongside distractor material chosen
to look similar. Two kinds of questions follow. Needle questions have an
answer in the document, and the gate checks the model returned that exact
fact. Control questions are the point: they ask for something the document
does not contain, and the gate passes the model only if it says so.
Answering a control question with a confident value is the failure everyone
fears; the gate also demands that any code-like value in an answer be
literally quotable from the source, so a fabricated one cannot slip
through as a near-miss. Failed requests stay in the denominator.

## Did the models find what was there?

Qwen3.6-35B-A3B retrieved the needle at
**100.0 % of requests** ([strix.qwen36.longctx.c1.needle-success](https://agmind.ai/claims/strix.qwen36.longctx.c1.needle-success/)) across the
depth ladder. gemma-4-26B-A4B came close at
**95.8 % of requests** ([strix.gemma4.longctx.c1.needle-success](https://agmind.ai/claims/strix.gemma4.longctx.c1.needle-success/)). Retrieval over
tens of thousands of tokens is largely a solved problem for this model
class on this box — which is exactly why the control group is where the
interesting result lives.

## Did they admit what was not there?

Qwen3.6 declined every impossible question:
**100.0 % of requests** ([strix.qwen36.longctx.c1.control-success](https://agmind.ai/claims/strix.qwen36.longctx.c1.control-success/)) on the
unanswerable controls. gemma-4 passed
**75.0 % of requests** ([strix.gemma4.longctx.c1.control-success](https://agmind.ai/claims/strix.gemma4.longctx.c1.control-success/)) of them — and
the failures were not what we expected.

## What did the failures look like?

Not fabrication. When gemma-4 failed a control question, the dominant
pattern was an empty answer: the model's reasoning pass consumed the token
budget deliberating and produced nothing addressed to the user. HTTP 200,
no invented value, no answer at all. That is a materially different risk
from hallucination — a pipeline gets a blank instead of a lie — and it
connects to a failure mode we measure separately:
[the answerless request](https://agmind.ai/reports/answerless-http-200/). A model that
cannot say "it's not there" cheaply may instead say nothing expensively.

## Does this mean these models never fabricate?

No, and the scope matters. The controls are synthetic: code-shaped facts in
constructed documents, a design that makes grading mechanical and
fabrication detectable. Adversarial domains, ambiguous questions,
contradictory sources — none of that is probed here, and a clean control
sheet on this workload is not a general immunity certificate for any model.
What the measurement establishes is narrower and still useful: on
documents of this shape, at these depths, one shipped configuration
declined every impossible question and the other's failures were silences,
not inventions.

## How to run this check yourself

The workload contract, corpora and grading gates — including the
word-boundary decline markers and the groundedness check — are in
[the public harness](https://github.com/botAGI/agmind-bench). Point it at
your own server and your own documents; if your numbers disagree with
ours, [we want them](https://github.com/botAGI/agmind-lab/issues/new).

Every figure above has a permanent page with scope, limitations and raw
runs: [the claim registry](https://agmind.ai/claims/).

---

Machine-readable claim registry: https://agmind.ai/claims.json · llms.txt: https://agmind.ai/llms.txt