The short, measured answer: on this box, strict-JSON reliability was not a function of quantization, backend or reasoning mode — every one of those swaps left task success unchanged. It was a function of the model. One family delivered a clean sheet across every configuration we ran; the other dropped answers on the same tasks. If your pipeline depends on parseable output, benchmark the model, not the settings.
The workload behind every number: deterministic short automation tasks with closed label sets, graded end to end — the output must parse as JSON and match ground truth per key. Failed requests stay in the denominator. One Ryzen AI Max+ 395 box, llama.cpp pinned by digest, three repeated runs per cell.
Does quantization break JSON output?
No — not between these two quants, on this model. Q4_K_M passed 100.0% of requestsrepeated and Q8_0 passed 100.0% of requestsrepeated on the same tasks. The popular fear that a smaller quant is where your JSON starts breaking found no support here — and since the smaller quant also decodes faster, reliability is no reason to pay for Q8_0 on this workload.
Does the backend matter?
No. The ROCm build passed 100.0% of requestsrepeated on the same tasks as the Vulkan build’s clean sheet. Backends move speed, not correctness — in this configuration they are execution paths, not different minds.
Does thinking mode help?
Not on tasks with a hard right answer. Reasoning on scored 100.0% of requestsrepeated against the same clean sheet with reasoning off — at an order-of-magnitude time cost. For structured automation the thinking block is pure latency.
The variable that actually moved: the model
Same box, same runtime discipline, same workload contract: gemma-4-26B-A4B passed 93.8% of requestsrepeated where Qwen3.6-35B-A3B held a perfect sheet across every configuration above. Not a catastrophic failure — most requests parse fine — but for an unattended pipeline the gap between “every response parses” and “most responses parse” is the gap between a cron job and a pager. The full model-swap picture, including where gemma wins elsewhere, is in the swap report.
How fast is a reliable JSON answer?
With reasoning off, end-to-end to a complete, verified answer: 844msrepeated. Fast enough that the model stops being the bottleneck of most automations — the concurrency behavior matters more once the queue grows.
What this does not say
- No grammar-constrained decoding was used. These are prompt-level results; llama.cpp’s grammar enforcement can mechanically guarantee parseability (though not per-key correctness) and was deliberately off — we measured the models, not the straitjacket.
- Closed label sets, short tasks. Long free-form JSON documents with nested structure are a harder problem this workload does not probe.
- Two model families, one box. The finding that the model is the moving part is exactly why we do not extrapolate it to models we have not run.
Every number above has a permanent page with scope, limitations and raw runs: the claim registry.