Sentrinel's ingest has two ways to accept a batch. It can write straight to ClickHouse, or it can append to a Redpanda log that a consumer drains into ClickHouse later. The log exists for one reason: when the store is down, the batch should wait somewhere rather than be refused and discarded.
In production, the Redpanda container was deleted — along with, it turned out, the databases — and nothing paged. The plugin kept posting. The API kept accepting. And for five days the ingest path did something worse than fail.
What "fallback" was doing
The old rule was simple: if the log is enabled, publish to it; if that fails, write the store directly. Reasonable. Here is what it cost with the broker gone:
Every single ingest call tried the broker first. The producer's retry ladder is deliberately patient — a send must not give up on a blip — so each call walked the whole ladder before falling through to the store it could have used immediately. Fifty-eight blocked calls a minute, each waiting seconds for a hostname that no longer resolved, to accomplish exactly what it would have accomplished in milliseconds by going straight to ClickHouse.
A component that was merely absent had become a latency problem for the component that was fine. And because the fallback "worked", the graphs stayed green.
What it does now
Three changes, and the third is the one that matters.
It fails over in both directions. Broker down: write the store. Store down: write the log, and let the consumer drain it when the store returns — which is the case the log was built for and the direction the old code never had. Both down: refuse the batch, so the plugin keeps it and retries. A fast 5xx and a slow one have the same outcome, and the fast one costs nobody anything.
The log is on by default. Not by a setting — by the broker answering. The API probes for one at boot, prefers the log the moment it answers, and starts the consumer at that moment too, even if the broker only appears an hour later. Durability now follows from running a broker rather than from also remembering to set a variable. SENTRINEL_INGEST_PIPELINE still exists, but it only pins the preference; it never turns failover off, because a pinned preference that refused the surviving path would be dropping telemetry to honour a config value.
It is a circuit breaker, not a retry. A failed path is marked down and skipped for a cooldown. The first batch to hit a dead path pays for the discovery; nothing after it does. What notices the recovery is a background probe, never a customer's request.
Two details that came out of testing, not design
The first version still cost seconds on a broker-less box, and the test suite told us: it went from fifteen seconds to forty-four. The cause was that we discovered an absent broker by publishing to it, which means walking the retry ladder. So the log is now probed before it is published to — a probe with no retries answers immediately, and a publish is only attempted after a probe has just said yes.
The second: a path inside its cooldown was still being kept as a last resort, so when the store failed we tried the known-dead broker anyway and paid the full timeout to learn nothing. It is now left out entirely. Trying something you watched fail a moment ago is not resilience.
It is visible now
/health reports which half is carrying traffic:
{ "ingest": { "policy": "auto", "using": "direct", "redpanda": "down", "direct": "up" } }
A pipeline running on one leg is a state you should be able to see, not infer from a latency graph a week later. That the health check said "healthy" for those five days is its own post.
What was actually lost
Nothing in the databases — the volumes survived and everything was restored. What was lost was five days of telemetry that arrived while nothing was running to receive it. The log would have held it. That is the argument for running one, and now it is also the default.
Details and every setting in the pipeline reference.