The one-paragraph version: on GB10, dcgm-exporter does not work and NVML answers N/A on much of what operators alert on. The reliable path is parsing nvidia-smi into node-exporter through a textfile collector, with a staleness alert on the collector itself. The details, and why the failure hides so well, are below.
There is a specific moment every DGX Spark operator reaches. The model is served, vLLM answers, the Grafana board is wired up the way it has been wired on every CUDA box for years. And the board is green. Green, empty, and lying: half the panels show no data, the other half show dashes, and nothing anywhere says why.
What actually breaks on GB10
Two separate failures stack on this hardware, and they hide each other.
dcgm-exporter does not function on GB10. The standard container that feeds GPU metrics to Prometheus on datacenter NVIDIA hardware does not work on the Spark’s unified-memory design. It does not crash loudly either. You get an exporter that runs and a dashboard with nothing behind it.
NVML answers N/A on a large share of queries. The library underneath
nvidia-smi and most Python monitoring snippets returns N/A for many of the
fields operators alert on, memory figures first among them. Unified memory
breaks the assumptions those counters were built on: there is no separate
VRAM pool to report, so tooling that expects one reports nothing.
The combination is nasty precisely because each half looks like the other half’s fault. The exporter is silent, so you check the library; the library answers N/A, so you assume the exporter would too and stop digging. Meanwhile the box serves a 284B-parameter MoE in production with no working GPU telemetry at all.
What works: nvidia-smi, parsed, through a textfile collector
The plain nvidia-smi text output on the Spark is more honest than the
libraries underneath it. Fields that respond, respond with real values. So
the collector we run is deliberately boring:
- A systemd timer runs
nvidia-smion a short interval. - A small script parses only the fields that actually answer on GB10 and
writes them as Prometheus metrics into a
.promfile. - node-exporter’s textfile collector picks the file up with the rest of the host metrics.
- One extra metric carries the collector’s own last-run timestamp, and the alert fires on staleness. A monitoring pipe that can die silently is the exact disease this page is about; the cure has to apply to the cure.
No daemons beyond what the host already runs, no privileged sidecars, and every metric that reaches the dashboard is one a human verified responds on this hardware. The full write-up with the parsing details is published on Habr, in Russian; the recipe it belongs to is on this site.
Why we bother saying this out loud
Because the failure mode is invisible by construction. A missing metric does not page anyone. Teams discover their Spark telemetry is fiction the first time something goes wrong and the dashboard has nothing to say about it — which is the most expensive possible moment for the discovery.
Our position, stated in the buyer’s answer on this hardware, stands: a serving box you cannot observe is a box you cannot operate. The Spark is operable. It just is not observable with the stock stack, and nothing in the stock stack tells you that.
Scope
This describes the GB10 units in our lab, on the firmware and driver stack current at the time of our bring-up, serving through vLLM. Monitoring behavior can change with driver releases; the collector approach survives such changes better than library bindings do, which is part of why we chose it. Nothing here is a statement about datacenter Blackwell, where dcgm-exporter is the correct answer.