The Cyrillic tax: what Russian text actually costs in tokens, measured

Every context-window plan starts with an assumption about how many characters fit in a token. Ours was wrong for Russian — publicly, in a frozen corpus — and the measured correction is more useful than the original numbers, so we published it as data instead of quietly rebuilding the corpus.

What we assumed and what we measured

The long-context corpus was cut to nominal rungs of 2k/8k/16k/32k tokens using a characters-per-token estimate: about 4 chars/token for English prose and 2.2 for Russian. The English assumption held; the Russian one did not. Per-item measured counts, read from the runtime’s own prompt-eval accounting and published in tokens-measured.json

English landed within about 6% of nominal. Russian landed at about two-thirds — this tokenizer (Qwen3.6’s, bundled in the GGUF) spends roughly 3.3 characters per token on literary Russian, not the 2.2 the corpus builder assumed .

What this means in practice

• The same document budget buys different depths per language. A “32k context” plan filled with Russian prose at our assumed density actually occupied about 21k tokens — the window was one-third emptier than the label said. Inverted: genuinely filling 32k tokens takes roughly half again as much Russian text as the estimate predicted.
• Chars-per-token is tokenizer-specific and language-specific. A ratio measured on English does not transfer to Cyrillic, and a ratio measured on one tokenizer does not transfer to another. If your RAG chunking, context budgeting or pricing math uses a single chars/token constant across languages, it is wrong in at least one of them.
• Measure, don’t estimate: the runtime tells you. Every llama.cpp server response carries prompt-token accounting; one pass over your own corpus replaces the folklore constant with ground truth. That is all tokens-measured.json is.

How this surfaced (and what it did to our own labels)

The mislabel was caught during a review of our long-context work: the corpus’s approx_tokens field disagreed with the server’s own accounting in our published run logs. The corpus file itself is frozen by hash and unchanged; the measured counts now ship beside it, the correction is logged on the errata page, and published depth claims are English-only and therefore unaffected. The affected Russian rungs are labeled with measured values everywhere they appear.

The deeper background — why Cyrillic tokenizes expensively, across several tokenizers, with the economics — is in our Russian long-read on Habr.

Limits

• One tokenizer measured here (Qwen3.6 GGUF); the Habr piece surveys others, and ratios differ by tokenizer family. The 3.3 chars/token figure is literary prose — code-mixed or technical Russian tokenizes differently.
• These are prompt-token counts, not quality claims. What depth does to retrieval and honesty is the long-context workload’s job — see useful context depth.