dcgm-exporter is dead and NVML answers N/A

What you see

vLLM serves, the Grafana board is wired the way it is on every CUDA box, and the board is green. Panels show no data or dashes, and nothing says why. nvidia-smi and the Python snippets that sit on NVML return

N/A

for many of the fields operators alert on, memory figures first. The exporter runs. There is just nothing behind it.

Where we saw it

DGX Spark, the GB10 units in our lab, serving through vLLM on the firmware and driver stack current at the time of our bring-up. The report is dated 2026-08-16 and names no driver, firmware or exporter version; neither does this card. The box was running the serving recipe linked under Evidence.

Cause

Unified memory. GB10 has no separate VRAM pool, and the counters NVML exposes were built on the assumption that one exists, so tooling that expects a pool reports nothing. dcgm-exporter does not work on the Spark’s unified-memory design either; the report records that as a separate failure and gives no mechanism for it. It does not crash loudly: it runs, and the dashboard behind it stays empty.

The two failures hide each other. The exporter is silent, so you check the library; the library answers N/A, so you assume the exporter would too and stop digging. A missing metric pages no one.

Fix

This is a workaround: the stock stack stays broken and we route around it. Plain nvidia-smi text output on the Spark is more honest than the libraries underneath it. Fields that respond, respond with real values. The collector we run:

1. A systemd timer runs nvidia-smi on a short interval.
2. A small script parses only the fields that actually answer on GB10 and writes them as Prometheus metrics into a .prom file.
3. node-exporter’s textfile collector picks the file up with the rest of the host metrics.
4. One extra metric carries the collector’s own last-run timestamp, and the alert fires on staleness.

No daemons beyond what the host already runs, no privileged sidecars. Every metric on the dashboard is one a human verified responds on this hardware. The report on this site gives the design; the parsing details are in the Habr write-up linked under Evidence.

Two checks before trusting any panel on a Spark: does this field answer on this box, and when did the collector last run.

Still open upstream?

Not tracked by us. The report notes only that monitoring behavior can change with driver releases and that the collector survives such changes better than library bindings do. Nothing here applies to datacenter Blackwell, where dcgm-exporter is the correct answer.

Evidence

• GPU monitoring on DGX Spark: when dcgm-exporter is dead and NVML answers N/A, the source report
• Serving recipe the collector belongs to
• Full write-up with the parsing details, on Habr, in Russian