While the API was running with its broker deleted, and after we had restored its databases, /health said this:
{ "status": "ok", "telemetryHealthy": true }
And the API's log, at the same moment, said this, several times a second:
Authentication failed: password is incorrect, or there is no user with such name.
type: "AUTHENTICATION_FAILED", code: "516"
Every ClickHouse query was failing. The health check was green. Docker Swarm, which decides whether a container is alive by asking that endpoint, agreed the service was fine.
Where the user went
ClickHouse had been restored onto its surviving data volume, and the sentrinel database was there, intact, with four million rows. The sentrinel user was not.
The container image creates a user from CLICKHOUSE_USER and CLICKHOUSE_PASSWORD at start-up — by writing it into /etc/clickhouse-server/users.d/, which is inside the container's filesystem and not inside the data volume. When the old container was removed, the user definition went with it. The data survived; the credential to read it did not.
The fix was to recreate the user with SQL instead of the environment variable. A SQL-created user is stored in the data volume's access/ directory and outlives the container. ClickHouse even reports the difference — storage: users_xml for the one that dies with the container, local_directory for the one that does not — which is a good thing to check the next time a restore "works".
Why the check said ok
The health check asked ClickHouse whether it was reachable. It was. It answered every query — with an authentication error, promptly and reliably. Reachable is not the same as usable, and a check that only asks the first question will report a database it cannot log into as healthy.
The uncomfortable part is that this check had already been through one lesson. It originally asked only the telemetry store, and reported healthy while Postgres was unreachable and every signed-in request was failing. So it was extended to check the control plane too. It just had not been extended to notice a store that answers "no".
Degraded is not down
There is a design choice in here worth stating, because the obvious fix is wrong.
The obvious fix is: if ClickHouse is broken, fail the health check. But the orchestrator's response to a failed health check is to kill and restart the container, and restarting the API cannot fix ClickHouse. It would take down authentication, the control plane, and the ingest path that was still successfully buffering — trading a broken half for a broken whole, on a loop.
So /health distinguishes. A dead control plane — no Postgres, no schema — returns 503 and lets the orchestrator act, because a restart is a plausible fix. A dead telemetry store returns 200 with "status": "degraded" and says which half, because that is a human's problem and the human needs the API up to fix it.
The failure here was that the second case was not being detected at all.
What it reports now
Two additions. The store check now runs a real query rather than a ping, so a store that answers with an error is degraded, not healthy. And the ingest pipeline reports which path is carrying traffic:
{
"status": "degraded",
"telemetryHealthy": false,
"ingest": { "policy": "auto", "using": "redpanda", "redpanda": "up", "direct": "down", "directError": "Authentication failed …" }
}
That last block is the one we wanted during those five days. A batch can be arriving safely while one of the two paths is dead, and that is precisely the state that goes unnoticed until the survivor fails too.
The general lesson
A health check answers exactly the question it asks. "Can I connect?" is a different question from "does this work?", and the gap between them is where an outage sits invisibly for days. Ask the second question, and make the answer distinguish between restart me and tell someone.