Deploy a large MoE on DGX Spark with vLLM: single node and a two-node pair

This is the tutorial version of our serving recipe — the same steps, ordered for someone doing it the first time, with the traps we paid for marked before you reach them. It was written on a real pair of GB10 units serving a 284B-parameter MoE; a single Spark follows the same path minus the fabric section.

You need: one or two DGX Spark units on current firmware, Docker with NVIDIA runtime, and a lot of disk — frontier-class weights are measured in the hundreds of gigabytes per node.

Step 1 — pin everything before you download anything

Three identities decide whether your deployment is debuggable later: the model revision, the container digest, and the firmware version. Write all three down first. main moves, tags move, and the Spark’s firmware updates have historically changed behavior enough to flip community sentiment on the whole device. A deployment you cannot name precisely is a deployment you cannot ask anyone to help debug.

# record these three lines somewhere permanent:
# model:    <repo> @ <revision>
# image:    vllm/vllm-openai@sha256:<digest>
# firmware: (from your system info)

Step 2 — weights on every node

For a pair: the full weights go on BOTH nodes, identical revision. Budget real time for this and verify sizes match when done. For a single node, pick a model that actually fits one device — the frontier class does not, which is why the pair exists.

Step 3 — single node first, even if you own two

Bring up one node alone before touching tensor parallelism. You want to learn the container’s behavior — startup time, memory pattern, log shape — without the fabric as a second variable:

docker run -d --name vllm --gpus all \
  -v /var/lib/llm/models:/models -p 8000:8000 \
  vllm/vllm-openai@sha256:YOUR_DIGEST \
  --model /models/YOUR-MODEL --max-model-len 32768

Readiness is an endpoint, not a feeling: the server is up when curl localhost:8000/v1/models answers. Weight loading on this class of model takes minutes — watch docker logs -f vllm and do not restart it out of impatience halfway through.

Step 4 — for a pair: head first, then peer, in that order

The rank-0 node hosts the coordination store. A peer that starts before the head is listening dies on a broken pipe with no useful error — this ordering bug reads obvious written down and cost us real hours live. The sequence that works: launch the head, wait until the store port shows in ss -tln, only then launch the peer. Fabric settings (head and peer IPs, HCA, GID index) belong in environment variables at the top of your launch script, not scattered through it.

Step 5 — verify the kernel selection took effect

For MoE models, vLLM’s automatic backend selection can deliberately skip the kernel purpose-built for your model’s quantization. Grep the logs for the MoE backend line and confirm which one loaded — the difference is not cosmetic. If your model family ships a specialized backend, set it explicitly and check the log again. In our experience this one check separates a published Spark number from a disappointed forum post.

Step 6 — the smoke test

Same discipline as on any box: send a real chat completion, verify content is non-empty and finish_reason is stop, and take token counts from usage.completion_tokens — under speculative decoding, streaming-event counts lie.

Step 7 — monitoring, before you need it

Stock GPU observability half-works on GB10 unified memory: dcgm-exporter does not function and NVML answers N/A on much of what you would alert on. Set up the textfile-collector workaround now — you do not want to discover the observability gap during your first incident.

Where to go next

• What to expect from it: the measured buyer answer, context curve and concurrency story are in the Spark reports.
• 256K contexts and beyond: the configuration hunt and the broken-path warnings are their own story.
• Verify your deployment: the harness is public; the numbers to compare against live in the benchmark repository.