In mid-August we went to check two freshly deployed test models on one of the
lab boxes. Routine look: docker ps, logs, a couple of smoke requests. Within
an hour the box had produced both classic monitoring lies, one in each
direction, and the pair was too clean not to write down.
Red but fine
Both containers showed unhealthy. Restart-worthy, page-someone unhealthy.
The models behind them were serving requests the whole time.
The cause took one docker inspect to find. The image ships a default
healthcheck that probes localhost:8080/health. The servers had been started
on ports 8090 and 8091. The probe was knocking on a door in an empty
corridor, concluding nobody lives in the building, and stamping the verdict
into the status column — while the actual servers answered every real
request one door over.
Nothing was ill. The measurement was. And the status column does not have a color for “the probe is misconfigured”; it only has a color for “dead”.
Green but broken
Same box, same afternoon. One of the models is a heavy reasoning model. We sent it a trivial question at temperature zero with an ordinary token budget, the kind any client application would use. It answered HTTP 200.
Inside the 200: an empty content field. The model had spent the entire
budget thinking — the reasoning channel was full of text, mid-sentence when
the budget ran out — and never emitted a single answer token. Every
transport-level signal reported success. Any client that reads content and
trusts the status code would have shown the user a blank message, logged a
successful request, and moved on.
We had measured exactly this failure mode on a frozen workload and published it as the answerless HTTP 200 report. It was strange to greet it in the wild the same week: not provoked, not constructed, just sitting there behind a green status on a healthy box.
The same disease
The two lies point in opposite directions, and that is what makes the pair instructive. One system reported failure for success; the other reported success for failure. Neither component malfunctioned. Both probes answered precisely the question they were configured to ask — “does anything answer on 8080?”, “did the HTTP request complete?” — and neither question was the one we thought we were asking, which was: is this model doing its job?
A status light is a claim about a measurement. If the probe’s configuration does not match the system it claims to describe — wrong port, wrong metric, wrong budget — you are monitoring a different system, one that happens to share a hostname with yours. The dashboard is not lying maliciously. It is answering exactly, in the direction you pointed it.
Both lies also share a survival strategy: nobody reads past the status column. Red gets restarted, which changes nothing, because the restarted container inherits the same probe. Green gets ignored, which is worse, because ignoring it is the point of green. In both directions the failure is invisible precisely where people look, and obvious one layer deeper — in the logs, in the response body — where nobody looks while the column has a color.
What this changed in our practice
The lab’s measurement rules exist because of specimens like these, and the pair sharpened three of them.
Grade the content, never the transport. Our gates read the answer text — whether anything was said, in the right language, without loops — and our first-token metric distinguishes the first answer token from the first token of anything, thinking included. The gap between those two definitions is where the green-but-broken case hides; the TTFA report is about that gap.
Probe what you serve, not what the image assumes. Our harness health checks target the exact endpoint and port a cell is configured to use, and fail fast when it does not answer — a dead container should cost seconds, not a timeout budget.
And when the telemetry itself is the suspect — on hardware where the standard GPU stack reports nothing at all — the collector carries its own staleness alert. The cure has to apply to the cure, or the monitoring pipe becomes one more system that can fail green.
The box, for the record, was fine all along. It was the questions that were broken — and the questions were ours. Point the probe at the thing you actually operate, and read one layer past the color at least once per deployment. The dashboard will answer whatever you ask. Ask the right thing.