# GPU monitoring on DGX Spark: when dcgm-exporter is dead and NVML answers N/A

> The standard GPU observability stack half-works on GB10 unified memory: dcgm-exporter does not run and NVML returns N/A on a large share of queries, so Grafana stays green and empty. The collector we actually serve vLLM with, and how to build it.

- Published: 2026-08-16
- Evidence level: lab_single_run
- Funding: Self-funded internal research
- Canonical: https://agmind.ai/reports/dgx-spark-gpu-monitoring/

The one-paragraph version: on GB10, dcgm-exporter does not work and NVML
answers N/A on much of what operators alert on. The reliable path is parsing
nvidia-smi into node-exporter through a textfile collector, with a staleness
alert on the collector itself. The details, and why the failure hides so
well, are below.

There is a specific moment every DGX Spark operator reaches. The model is
served, vLLM answers, the Grafana board is wired up the way it has been wired
on every CUDA box for years. And the board is green. Green, empty, and
lying: half the panels show no data, the other half show dashes, and nothing
anywhere says why.

## What actually breaks on GB10

Two separate failures stack on this hardware, and they hide each other.

**dcgm-exporter does not function on GB10.** The standard container that
feeds GPU metrics to Prometheus on datacenter NVIDIA hardware does not work
on the Spark's unified-memory design. It does not crash loudly either. You
get an exporter that runs and a dashboard with nothing behind it.

**NVML answers N/A on a large share of queries.** The library underneath
`nvidia-smi` and most Python monitoring snippets returns N/A for many of the
fields operators alert on, memory figures first among them. Unified memory
breaks the assumptions those counters were built on: there is no separate
VRAM pool to report, so tooling that expects one reports nothing.

The combination is nasty precisely because each half looks like the other
half's fault. The exporter is silent, so you check the library; the library
answers N/A, so you assume the exporter would too and stop digging. Meanwhile
the box serves a 284B-parameter MoE in production with no working GPU
telemetry at all.

## What works: nvidia-smi, parsed, through a textfile collector

The plain `nvidia-smi` text output on the Spark is more honest than the
libraries underneath it. Fields that respond, respond with real values. So
the collector we run is deliberately boring:

1. A systemd timer runs `nvidia-smi` on a short interval.
2. A small script parses only the fields that actually answer on GB10 and
   writes them as Prometheus metrics into a `.prom` file.
3. node-exporter's textfile collector picks the file up with the rest of the
   host metrics.
4. One extra metric carries the collector's own last-run timestamp, and the
   alert fires on staleness. A monitoring pipe that can die silently is the
   exact disease this page is about; the cure has to apply to the cure.

No daemons beyond what the host already runs, no privileged sidecars, and
every metric that reaches the dashboard is one a human verified responds on
this hardware. The full write-up with the parsing details is published
[on Habr, in Russian](https://habr.com/ru/articles/1030802/); the recipe it
belongs to is [on this site](https://agmind.ai/reports/deepseek-v4-flash-0731-dgx-spark-recipe/).

## Why we bother saying this out loud

Because the failure mode is invisible by construction. A missing metric does
not page anyone. Teams discover their Spark telemetry is fiction the first
time something goes wrong and the dashboard has nothing to say about it —
which is the most expensive possible moment for the discovery.

Our position, stated in the
[buyer's answer on this hardware](https://agmind.ai/reports/dgx-spark-worth-it/), stands: a
serving box you cannot observe is a box you cannot operate. The Spark is
operable. It just is not observable with the stock stack, and nothing in the
stock stack tells you that.

## Scope

This describes the GB10 units in our lab, on the firmware and driver stack
current at the time of our bring-up, serving through vLLM. Monitoring
behavior can change with driver releases; the collector approach survives
such changes better than library bindings do, which is part of why we chose
it. Nothing here is a statement about datacenter Blackwell, where
dcgm-exporter is the correct answer.

---

Machine-readable claim registry: https://agmind.ai/claims.json · llms.txt: https://agmind.ai/llms.txt