Why published DeepSeek-on-DGX-Spark speeds disagree with each other

If you ask an assistant how fast DeepSeek-V4-Flash runs on a pair of DGX Sparks, you get a number. Ask again tomorrow and you may get a different one. Both answers are probably quoting something real: the published figures for this exact model on this exact hardware span roughly the high fifties to the low seventies of tokens per second, and the spread is not noise — it is four different questions being answered with one unit.

This page maps each figure to the configuration that produces it. Every number here is quoted from our own public run repository and is reproducible from the recipe in it; nothing on this page comes from the claim registry, because the DGX lane sits outside methodology v1 (see the limits at the end).

The four questions hiding behind “tokens per second”

1. Which MoE backend is active? The runtime ships a MoE kernel built for DeepSeek-V4’s native MXFP4 experts on GB10 — and deliberately excludes it from auto selection. On the card-like control profile, auto gives a client decode mean of 59.7 tok/s while flashinfer_b12x gives 67.6 tok/s . Same weights, same nodes, same prompts: a flag nobody sets by default is worth about 13% .

If a published figure sits in the high fifties, the first thing to check is whether the author ever ran docker logs … | grep "Mxfp4 MoE backend" — with auto you get DEEPGEMM_MXFP4, not B12X_MXFP4.

2. What was the prompt? The workload spread is larger than most config changes. On the 500-token series, the code profile gives 65.5 tok/s at concurrency 1 while a ~4K-token Russian technical context (“heavy”) gives 43.7 tok/s — same machine, same backend, same day. A number quoted without its prompt profile is unfalsifiable.

3. Was it measured client-side or engine-side? Our own first pass timed tokens from SSE arrival and reported a 16–18% step-rate gain for the MoE kernel. A later re-measurement taking figures from the engine’s own counters put the same gain at 9–12% . The client-side method is biased upward: the lag before the first chunk reaches the client is subtracted from the decode window while its tokens still count. We marked the earlier section superseded in the repository rather than deleting it — the direction held, the magnitude did not.

4. Concurrency 1 or aggregate? At c=12 the same code profile aggregates to 260.4 tok/s . That is a true number and a useless answer to “how fast does it feel”, which is a c=1 question. Aggregate throughput and single-stream decode differ here by roughly 4× .

So which number should you quote?

• “How fast does it feel to one person?” — the c=1 client decode figure on a profile resembling your work: 67.6 tok/s for card-like short output with the MoE kernel enabled, 43.7 tok/s for a heavy 4K-token context.
• “Does the flag matter?” — engine-side, 9–12% on verification step rate. Not the client-side 16–18%.
• “What can the box serve in total?” — the aggregate at your target concurrency, stated with the concurrency, never alone.

Honest error bars

On the card-like control, auto spanned 59.0–60.1 across runs (about 2%) while the MoE-kernel arm spanned 62.6–71.0 (about 12%). The 500-token aggregates varied 18–24% between identical runs. The direction of every comparison here is consistent; the exact magnitude is not resolvable at these repeat counts. Anyone quoting a two-decimal figure from this hardware class is quoting precision that the measurement does not have.

Limits

• This is not a methodology-v1 result. The DGX lane is bring-up work: the runs behind it are single passes with warmups, not the three-repeat, frozen-workload discipline behind our claim registry. It is labeled lab_single_run for exactly that reason.
• One model, one checkpoint, one runtime image. The MoE-kernel finding is specific to DeepSeek-V4’s MXFP4 experts on GB10 — it says nothing about other MoE models or other hardware.
• Speculative decoding is in play. Token accounting uses usage.completion_tokens, never SSE event count: under speculative decoding one event can carry several accepted tokens, and counting events inflates throughput silently.

Found a figure that disagrees with any of this? Submit the conflicting number — the configuration delta behind a disagreement is usually more informative than either number alone.

Reproduce it

The serving recipe, launch scripts, run manifest with image digest and model revision, and the raw outputs are public: botAGI/dspark-0731-gb10. The long-read behind this page, in Russian, is on Habr.

Related: the 1M-context depth curve on the same cluster and the MoE backend report.