"What did we spend?" — custom metrics in Sentrinel

Someone evaluating Sentrinel wrote us a review that was almost entirely right, and the part that was right cost us the deal:

Sentrinel answers "why was this slow and what failed." It will not answer "what did we spend."

They had read the SDK properly. They found traceSpan, traced, sentrinelFetch, structured logging with context, deploy markers — and no counter, no gauge, no recordMetric. Their conclusion followed: wrap your DeepSeek call in traced() and you get latency, failures and the retry chain, tied to the user request that caused it. What you do not get is a chart of tokens.

They were right. So we built it.

The gap, precisely

A span is a shape: a name, a start, a duration, a status, and some attributes. It is very good at "this took 900 ms and failed twice, here is the trace it belonged to."

It is not a number you asked to be summed. You can put tokens: 1240 in a span attribute — many people do — and you still cannot chart tokens per model per customer this month, because nothing in the system was told that attribute is a quantity, or that summing it means anything.

Logs have the same problem in a friendlier costume. {module, model, tokensIn, costUsd} is searchable and human-readable, and it is still text. Grepping a month of logs to add up a column is not a dashboard.

The missing primitive was small and specific: a way to say "this is a number, add it up, and let me slice it."

What it looks like

import { count, gauge, histogram } from "@sentrinel/plugin";

// After the model call returns:
count("llm.tokens", usage.input_tokens,  { model: "deepseek-chat", direction: "input"  });
count("llm.tokens", usage.output_tokens, { model: "deepseek-chat", direction: "output" });
count("llm.cost_usd", priceOf(usage),    { model: "deepseek-chat" });

// Distributions, when the average hides the problem:
histogram("llm.cost_per_call_usd", priceOf(usage), { model: "deepseek-chat" });

// Levels that go up and down:
gauge("queue.depth", await queue.size());

Then on the Metrics page: pick llm.cost_usd, split by model, and read the total. That is the whole feature.

Three functions, because there are three genuinely different questions:

The part that took the thought

The naive version of this is a network write per increment. That is fine for count("signups") and catastrophic for count("llm.tokens", 1240) called on every completion, in every worker, forever.

So an increment does not touch the network. It mutates a map:

name + kind + canonical(labels)  →  { count, sum, min, max, last, samples }

and the whole map is folded into rows once per flush window. In our own test, 2,001 calls became 6 rows. A counter called ten thousand times a second costs one row every thirty seconds.

This is the same trade the request collector already made, and it is what makes count() safe to call in a loop — which matters, because a metrics API you have to be careful with is one people stop using.

Three decisions fell out of that design, and each of them is the sort of thing you only notice when it is wrong.

Labels are canonicalised before they become a key. {model, tier} and {tier, model} are the same series. Without that, two call sites that happened to write the keys in a different order would chart as two unrelated lines, and nothing would tell you why your total was split in half.

Every aggregate travels on every row — sum, last, count, min, max, p50, p95, p99 — regardless of which function you called. Storing only what a counter "needs" would save a few bytes and mean re-instrumenting, redeploying, and waiting a week to ask "what was the largest single increment" of data you already collected.

Percentiles use a reservoir. A histogram called a million times cannot hold a million floats, but keeping only the first 512 would make p95 a statement about your first few seconds. Past the cap, samples are replaced with decreasing probability, so the estimate stays representative of the whole window.

Cardinality, and refusing to be the outage

count("llm.tokens", n, { model, tier });      // good
count("llm.tokens", n, { userId: user.id });  // a series per user

The second one is the classic way to turn a metrics system into an incident. We cap a process at 2,000 live series and drop new ones past that, along with 12 label keys and 128 characters per value.

Dropping data is not a nice thing for a monitoring SDK to do. Growing without bound inside someone else's process is a much worse one. A monitoring tool that causes the outage it was installed to report has failed at the only thing it was there for, so when those two lose, memory wins and the drop count is reported.

NaN and Infinity are refused for the same reason. One poisoned value would make every chart drawn from that sum useless, and it would look like data.

What this is not

It is not a replacement for tracing, and the reviewer had this exactly right: they are complementary, not a choice.

Tracing tells you why — which call was slow, what it retried, which user request it belonged to, what failed downstream. Metrics tell you how much — tokens, dollars, depth, throughput. The DeepSeek call that took 4 seconds and the $41 you spent on that model today are two questions about the same event, and no single primitive answers both well.

What changed is that Sentrinel now answers the second one instead of pointing at your logs.

Since this was written

Two things that make the metrics more useful than they were on the day.

The LLM page and counters are one story now. A span carrying the standard gen_ai.* attributes is priced per call; a counter fed from the same usage object answers spend per model, per plan, per month. Which to reach for, and when, is Two questions about one LLM call.

An agent can read the result. With an AI agent key, the MCP server and CLI give a coding agent your issues, logs and traces — and the request behind a spike, so "why did llm.cost_usd jump at 14:00" can be handed to the agent rather than assembled by hand. Your coding agent can read production now.

Getting it

It ships in @sentrinel/plugin. Custom metrics need TELEMETRY_STORE=clickhouse — ingest answers 501 on the Postgres store rather than accepting them and quietly dropping them, because an integration that looks connected and is silently empty is the worst failure mode available.

Full API, cardinality rules and retention in the plugin reference.