# Can a local LLM reliably return strict JSON? Measured across quants, backends and models

> Strict-JSON output is the load-bearing wall of local automation, and its reliability is usually asserted, not measured. On one box: quantization did not move it, the backend did not move it, reasoning mode did not move it — switching the model family did.

- Published: 2026-08-20
- Evidence level: lab_repeated
- Funding: Self-funded internal research
- Canonical: https://agmind.ai/reports/local-llm-json-reliability/

The short, measured answer: on this box, strict-JSON reliability was not a
function of quantization, backend or reasoning mode — every one of those
swaps left task success unchanged. It was a function of the model. One
family delivered a clean sheet across every configuration we ran; the other
dropped answers on the same tasks. If your pipeline depends on parseable
output, benchmark the model, not the settings.

The workload behind every number: deterministic short automation tasks with
closed label sets, graded end to end — the output must parse as JSON and
match ground truth per key. Failed requests stay in the denominator. One
Ryzen AI Max+ 395 box, llama.cpp pinned by digest, three repeated runs per
cell.

## Does quantization break JSON output?

No — not between these two quants, on this model. Q4_K_M passed
**100.0 % of requests** ([strix.qwen36.structured.c1.task-success-nothink](https://agmind.ai/claims/strix.qwen36.structured.c1.task-success-nothink/)) and
Q8_0 passed **100.0 % of requests** ([strix.qwen36q8.vulkan.c1.task-success](https://agmind.ai/claims/strix.qwen36q8.vulkan.c1.task-success/)) on
the same tasks. The popular fear that a smaller quant is where your JSON
starts breaking found no support here — and since the smaller quant also
[decodes faster](https://agmind.ai/reports/q8-0-vs-q4-k-m-strix-halo/), reliability is no
reason to pay for Q8_0 on this workload.

## Does the backend matter?

No. The ROCm build passed
**100.0 % of requests** ([strix.qwen36q4.rocm.c1.task-success](https://agmind.ai/claims/strix.qwen36q4.rocm.c1.task-success/)) on the same tasks
as the Vulkan build's clean sheet. Backends move
[speed](https://agmind.ai/compare/llamacpp-vulkan-vs-rocm-strix-halo/), not correctness —
in this configuration they are execution paths, not different minds.

## Does thinking mode help?

Not on tasks with a hard right answer. Reasoning on scored
**100.0 % of requests** ([strix.qwen36.structured.c1.task-success-think4k](https://agmind.ai/claims/strix.qwen36.structured.c1.task-success-think4k/)) against
the same clean sheet with reasoning off — at
[an order-of-magnitude time cost](https://agmind.ai/reports/should-you-disable-thinking-local-llm/).
For structured automation the thinking block is pure latency.

## The variable that actually moved: the model

Same box, same runtime discipline, same workload contract: gemma-4-26B-A4B
passed **93.8 % of requests** ([strix.gemma4.structured.c1.task-success](https://agmind.ai/claims/strix.gemma4.structured.c1.task-success/)) where
Qwen3.6-35B-A3B held a perfect sheet across every configuration above. Not
a catastrophic failure — most requests parse fine — but for an unattended
pipeline the gap between "every response parses" and "most responses
parse" is the gap between a cron job and a pager. The full model-swap
picture, including where gemma wins elsewhere, is in
[the swap report](https://agmind.ai/reports/model-quant-backend-strix-halo/).

## How fast is a reliable JSON answer?

With reasoning off, end-to-end to a complete, verified answer:
**844 ms** ([strix.qwen36.structured.c1.e2e-nothink](https://agmind.ai/claims/strix.qwen36.structured.c1.e2e-nothink/)). Fast enough
that the model stops being the bottleneck of most automations — the
[concurrency behavior](https://agmind.ai/reports/concurrency-capacity-strix-halo/) matters
more once the queue grows.

## What this does not say

- **No grammar-constrained decoding was used.** These are prompt-level
  results; llama.cpp's grammar enforcement can mechanically guarantee
  parseability (though not per-key correctness) and was deliberately off —
  we measured the models, not the straitjacket.
- **Closed label sets, short tasks.** Long free-form JSON documents with
  nested structure are a harder problem this workload does not probe.
- **Two model families, one box.** The finding that the model is the
  moving part is exactly why we do not extrapolate it to models we have
  not run.

Every number above has a permanent page with scope, limitations and raw
runs: [the claim registry](https://agmind.ai/claims/).

---

Machine-readable claim registry: https://agmind.ai/claims.json · llms.txt: https://agmind.ai/llms.txt