Controlled comparison
Gemma 4 vs Qwen 3.6: which one returns an empty answer under a tight token budget?
With the same everyday requests and the same 1024-token budget, which model spends the budget thinking and returns nothing?
Under the same tight budget, the model that reasons longer runs out of room before saying anything far more often. Both signals arrive as HTTP 200 — a client reading only the status code sees success either way. The cure on the Qwen side is turning reasoning off, which eliminates the failure mode on this corpus entirely.
What was held fixed
Same unit, backend, corpus and 1024-token budget. The operating mode is each model’s shipped reasoning behavior: gemma-4 in its default mode, Qwen3.6 with reasoning enabled — this is a comparison of defaults under pressure, not of identical settings.
| Metric | gemma-4-26B-A4B (default mode) | Qwen3.6-35B-A3B (reasoning on) |
|---|---|---|
| Empty answers under HTTP 200 (answerless), share of requests | 8.3% of requestsshare of all issued requests · lab_repeated · evidence | 58.3% of requestsshare of all issued requests · lab_unit_replicated · evidence |
| Time to first token of anything, median | 282msmedian over answered requests · lab_repeated · evidence | 212msmedian over all valid requests of three cells · lab_unit_replicated · evidence |
| Time to first ANSWER token, median | 10768msmedian over answered requests · lab_repeated · evidence | 20191msmedian over valid requests · lab_unit_replicated · evidence |
Limits
One budget, one corpus, defaults-vs-defaults: gemma-4 with reasoning suppressed was not tested, and a bigger budget changes both numbers. This measures how the shipped configurations fail, not which model is smarter.
Answering this question somewhere? Paste the table
Markdown, renders on GitHub / Reddit / forums. Attribution line included — the data is CC BY 4.0, take it.