This is the runbook behind our published DeepSeek-V4-Flash-0731 numbers on a pair of DGX Sparks (GB10, TP=2 over 200G RoCE) — every step the way we actually run it, with the traps that cost us time marked as traps. The authoritative, versioned copy lives in the public repository with the run manifest and raw outputs; this page is the annotated tour.
The five steps
1. Weights on BOTH nodes — ~156 GiB on disk per node . We
benchmarked the community FP8 checkpoint pinned by revision; the official
checkpoint is noted in the repo but is not what our numbers describe. Pin
the revision — main moves.
2. Runtime image by digest, on both nodes. Tags move; the digest in the repo does not. Every published number names the digest it ran under — that is what makes a disagreement diagnosable later.
3. HEAD FIRST, then the peer. Rank 0 hosts the TCPStore; a peer that
connects before the head is listening hangs on a broken pipe with no useful
error. Launch the head, wait until ss -tln shows the store port listening,
only then launch the peer. This ordering bug reads obvious written down and
cost us real hours live.
4. Ready when /v1/models answers. Weight load takes ≈6–7 minutes
variables at the top of the launch script.
5. Verify the MoE backend actually took effect. The engine ships a
kernel purpose-built for this model’s MXFP4 experts on GB10 and deliberately
excludes it from auto:
docker logs vllm_dsv4 2>&1 | grep "Mxfp4 MoE backend"
# want: Using 'B12X_MXFP4' not: Using 'DEEPGEMM_MXFP4'Skipping this check is the single most common reason a published Spark
number lands in the high fifties instead of the high sixties: auto gave us
59.7 tok/s client decode where flashinfer_b12x gave 67.6 ,
a 9–12% verification-step-rate gain measured engine-side .
The numbers to expect
Final configuration, medians over repeated requests, from the public results :
| Cell | tok/s |
|---|---|
| card-like short output, c=1 | 67.6 |
| code profile, c=1 | 65.5 |
| heavy 4K-context prose, c=1 | 43.7 |
| code profile, c=12 aggregate | 260.4 |
The profile spread is larger than most configuration changes — quote a number without its profile and it is unfalsifiable. Why the figures circulating in forums disagree with each other is its own page.
Count tokens from usage.completion_tokens, never from SSE event count.
Under speculative decoding one event can carry several accepted tokens;
counting events silently inflates throughput.
The monitoring trap
On GB10 unified memory, dcgm-exporter does not work at all and NVML
returns N/A on a large share of queries — your Grafana will be green and
empty. We parse nvidia-smi with a custom textfile collector into
node-exporter metrics; the write-up is
on Habr, in Russian. Budget for
this: a serving box you cannot observe is a box you cannot operate.
Honesty section
- Everything here is
lab_single_run: single passes with warmups, one pair of units, outside our three-repeat methodology. Error bars on this hardware class are wide — identical aggregate runs varied 18–24% — so treat single-decimal precision from any source, including ours, as optimistic. - Our own first measurement pass was client-side-biased and we marked it superseded in the repo rather than deleting it; the engine-side series is the one to quote.
- Found a number that disagrees? Submit it — the configuration delta behind a disagreement is usually more informative than either figure.
Related: the 1M-context depth curve · the MoE backend report · is the Spark worth it at all.