The short answer from a controlled cell: the bigger quant bought nothing our quality gates could detect, and it decodes slower. On this box, for these workloads, Q4_K_M wins on both counts. The long answer, with the numbers and their limits, follows.
Every local LLM community relitigates this weekly. Someone downloads a model, sees six quant files, asks which one, and receives a wall of confident answers backed by perplexity tables from different hardware, different models, or nothing. What the thread almost never contains is the same model measured at two quants on the same machine with everything else frozen. That is this page.
The controlled cell
One Beelink GTR9 Pro, Ryzen AI Max+ 395 with 128 GB of unified memory. llama.cpp server, Vulkan backend, build pinned by image digest. Qwen3.6-35B-A3B from the same artifact family, once at Q4_K_M and once at Q8_0, each pinned by hash. Same workloads, same gates, three repeated runs each, reasoning off. The quantization is the only variable.
Is Q8_0 slower than Q4_K_M?
Decode pace is where you feel a quant all day, so start there. Q4_K_M runs at 15.9ms/tokenrepeated. Q8_0 on the same backend runs at 18.7ms/tokenrepeated.
The bigger weights push more bytes through the same memory bus for every token, and on a unified-memory box the bus is the ceiling. You pay it on every token of every answer, forever. Both paces are usable for a single reader; the gap is real and permanent.
An aside the backend curious will enjoy: Q8_0 on Vulkan lands at the same pace as Q4_K_M on ROCm, 18.7ms/tokenrepeated — picking the wrong backend costs about as much as doubling your weights. That story continues in the backend comparison.
Does Q8_0 improve quality over Q4_K_M?
On our gates: nothing detectable. Strict-JSON automation, the workload where degraded weights show up first as malformed output and wrong labels, passed at 100.0% of requestsrepeated on Q4_K_M and at 100.0% of requestsrepeated on Q8_0. Identical. Format, language and repetition gates on the interactive corpus: no difference either.
Read that claim as narrowly as it is written. Our gates check whether output parses, matches ground truth on closed label sets, and stays in the right language without loops. They are not a reasoning benchmark, not a knowledge probe, and they say nothing about the subtle quality differences long-form writing might surface. What they establish is that on tasks with a hard right answer, this model at Q4_K_M gave Q8_0 nothing to win.
Which quant should you use: Q8_0 or Q4_K_M?
Decide by what you are short of. Short of memory: Q4_K_M, and this is most people. The Q8_0 artifact is substantially heavier on disk and in memory, which on smaller boxes decides the question before quality gets a vote, and even on 128 GB it can be the difference between fitting a second model alongside or not. Short of patience: Q4_K_M again, and that is the uncomfortable finding. The default advice of “run the biggest quant that fits” spends your decode pace on a quality difference our hard-answer gates could not detect. If your workload is long-form prose where nuance might actually differ, run your own corpus through both — the harness is public.
What this does not say
- One model family. MoE models with small active sets are widely held to quantize gracefully; a dense model may repay Q8_0 differently. We measured this model, and claim only it.
- No perplexity, no accuracy suites. Deliberately: those numbers travel badly across hardware and tokenizers. Task gates with ground truth are cruder and honester.
- One box, one build. The pace gap shape should hold on any bandwidth-limited unified-memory machine; exact figures are this box only.
The full matrix with both quants and both backends is on the comparison page, the deep dive on quant-times-backend interaction in the quant and backend report. Every number above has a permanent page with scope and raw runs: the claim registry.