Method
Workload library
Fixed, versioned workloads with frozen corpora, hashes and acceptance checks. A qualification result is only meaningful relative to a specific workload revision — a material change creates a new revision, never a silent update.
Released workloads have frozen, hash-pinned corpora and measured runs behind them — they back the published claims. Draft workloads have defined identity but unfrozen corpora, and are never used for public headline claims.
- interactive-assistant-v1 released
Is single-user streaming chat responsive on this system?
Client-side TTFT, inter-token latency and end-to-end time over a frozen 16-item EN+RU corpus in three length bands; format, language and repetition gates with per-item expectations.
- interactive-assistant-v2 released
Is single-user streaming chat responsive on this system for everyday human tasks?
Same contract as v1 — client-side TTFT, inter-token latency, end-to-end time; format, language and repetition gates — over a human-task corpus: everyday requests (messages, emails, explanations, plans) instead of self-referential benchmarking questions. 16 items, EN+RU, three length bands.
- team-serving-v1 Draft
What request rate does the system sustain within a stated SLO?
Planned contract (identity not frozen yet): concurrency sweep (closed-loop) plus frozen-trace open-loop mode; goodput vs throughput; failed requests stay in the denominator; output is an operating envelope.
- long-context-v1 released
How deep is the useful context on this system, not the configurable one?
Needle retrieval across a nominal 2k/8k/16k/32k context ladder — 12 items, EN (Dickens) + RU (Dostoevsky); measured prompt tokens run below nominal (EN within ~6%, RU at ~65%) and ship beside the corpus — plus unanswerable controls with a present distractor; format, language, repetition, needle and control gates; quality, TTFT and end-to-end time at each depth.
- structured-agent-v1 released
Does strict JSON output stay reliable for everyday automation tasks on this system?
16 deterministic items, EN+RU, three task types — extraction with exact ground truth, classification with closed label sets, generation under a structural contract. Strict JSON parse plus per-key match separate parse success from task success; the repetition gate approximates loop detection. No tool-execution sandbox in this revision.
- endurance-30m-v1 released
Does this system degrade under sustained load — and when?
Cycles the interactive-assistant-v2 human corpus closed-loop at concurrency 4 for 180 minutes; per-request start offsets give 5-minute windows with checkpoints at 30/60/180 min; windowed inter-token latency, TTFT and request success; format, language and repetition gates. The “30m” in the id is historical — the first checkpoint.
- rag-pipeline-ru-v0 Internal
Where does a Russian-language RAG pipeline lose quality, component by component?
Per-component evaluation: OCR CER/WER → splitter boundary F1 → Recall@k/nDCG → reranker delta → generation support/abstention/citations. Internal until 6 release gates pass.
- doc-session-v1 released
Does the second question over the same pasted document pay the full prefill again on this system?
Client-side time to first token for question pairs sharing a byte-identical document prefix (1 warmup item + 4 documents, 8k/32k tokens, EN+RU), with the server prompt cache off (baseline) and on; format/language/repetition gates on the answers.