The question this card answers
Does strict JSON output stay reliable for everyday automation tasks — extract fields from a message, classify a review, produce a structured plan — on this exact configuration?
Verdict: PASS WITH LIMITS. Every answer in every repeat parsed as strict JSON and matched the ground truth per key, in both reasoning modes. The limits are real and listed below.
What was measured
Sixteen deterministic tasks, EN and RU: extraction with an exact expected object (names, dates, negative temperatures, booleans — including Cyrillic values behind Latin keys), classification over closed label sets, and generation with a structural contract. The gate strips code fences, parses strictly, and compares required keys against the ground truth. A parse failure and a wrong value are distinguishable in the records, giving parse and task success separately.
Results
Task success with reasoning disabled: 100.0% of requestsrepeated — and with reasoning enabled at a 4096-token budget it is the same: 100.0% of requestsrepeated.
The difference is time. A complete valid answer takes a median of 844msrepeated without reasoning versus 14677msrepeated with it — on this corpus the reasoning pass bought no measurable correctness and cost roughly seventeenfold in wall-clock time.
The limits in “pass with limits”
- No tool execution. These are single-turn JSON answers; the deterministic-sandbox agent loop is a future workload revision. Agent-readiness claims remain invalid per the workload definition.
- Short, deterministic tasks with closed label sets. Free-form schemas, long documents and adversarial inputs are untested.
- One unit, concurrency 1, one runtime build, one artifact.
- At a 1024-token budget with reasoning enabled this configuration returns empty answers on most requests — see the everyday assistant card. The passing envelope assumes reasoning off or a 4096-token budget.
Revalidation triggers
Runtime image change, model artifact change, workload revision change (especially the sandboxed agent loop), driver/kernel change. Harness and corpus are public: agmind-bench.