The question this card answers
You paste a long document into the chat and ask about it. Does the system find the answer — and does it admit when the answer is not there — on this exact configuration, and what does it cost you to wait?
Verdict: PASS WITH LIMITS. Quality held at every measured depth; the practical constraint is latency, and it is quantified below.
What was measured
Public-domain prose (Dickens in English, Dostoevsky in Russian) cut into a context ladder of four rungs, nominally 2k/8k/16k/32k tokens. Those labels are targets, not measurements: the corpus was cut by a characters-per-token estimate that holds for English (measured 1.8k–30.0k) but overestimates Russian by about a third (measured 1.4k–21.1k). Per-item measured counts ship beside the corpus; the depth numbers below are English-only and so are unaffected. Each document carries one synthetic fact — a unique gate code — inserted mid-document. Retrieval questions ask for that code; control questions ask for a code that is not in the document while a similar distractor fact is present — the honest answer is “it is not stated”, and fabricating one fails the gate.
Results
Needle retrieval: 100.0% of requestsrepeated across the whole ladder, both languages, all repeats. Controls: 100.0% of requestsunit_replicated — the model declined to invent the missing code every time, distractor notwithstanding.
The cost is the wait. With a 2k-token document the first token arrives at a median of 1907msrepeated; with a 32k-token document, at 33965msrepeated — the growth is roughly linear in document length. Pasting a long report turns an instant assistant into a half-minute pause before anything appears, and nothing on screen distinguishes that pause from a hang.
The limits in “pass with limits”
- Retrieval, not comprehension. A synthetic needle at a fixed ~50% depth is the easiest honest probe; summarization, multi-hop reasoning and position sweeps are future revisions.
- Measured to 30k tokens only (the English 32k rung). No claim is made beyond that — a larger configurable window is not evidence of a larger useful one.
- One unit, single stream, reasoning disabled, one runtime build, one artifact; prompt cache off, so every request pays full prefill.
Revalidation triggers
Runtime image change (prefill speed is runtime-sensitive), model artifact change, workload revision change, driver/kernel change. Harness and corpus are public: agmind-bench.