LLM Observability: What Logs Miss

A traditional log line records that a request happened: the endpoint, the status, the duration. For an LLM system that is the least interesting part of the story. The interesting parts are invisible to the log: what the model was asked, what it answered, whether the answer was any good, and whether the answer today is worse than the answer last week.

LLM systems fail in ways that logs cannot see. The model drifts — same prompt, different quality, no error. The prompt changes and nobody notices the regression. The cost climbs and the invoice explains it a month late. The failures are slow, statistical, and silent.

Observability is the practice of making them visible: capturing the full request — prompt, response, tokens, latency, cost — and watching the aggregates move. It is the difference between finding out about a problem from a user and finding out from a dashboard.

The unit of LLM observability is the trace: the complete record of one request's journey — the prompt as sent, the response as received, the token counts, the latency breakdown, the cost, and the metadata that says which model, which prompt version, which feature. A trace is a log line that remembered everything.

Traces compound into the questions that matter. Which prompt version is live? What did this user actually receive? How much did this feature cost this week? The answers exist only if the traces were captured — retroactively, the data is gone.

And the capture is cheap relative to the model call it records. A trace costs a database write; the request it describes costs tokens. The asymmetry is the entire argument: for a rounding error of the request's cost, the request becomes inspectable forever.

Four numbers, watched over time. Cost per completed task: the all-in spend divided by the accepted answers, the metric that catches every silent regression at once. Latency at the percentiles that matter: the first-token and total distributions, because the average lies and the tail is what users feel. Error and retry rates: the failure taxonomy from the resilience discipline, because a rising retry rate is a falling margin. And quality: the eval scores run continuously against live traffic, because the model can get cheaper and worse at the same time.

The dashboard's job is not to show these numbers; it is to show their movement. A number is a fact; a trend is a signal. The teams that catch problems early are watching the slopes, not the values.

And the dashboard should be boring. A dashboard that demands attention every day is noise; the good one sits quiet until a slope bends, and then it is the only thing anyone looks at.

Drift is the failure mode that observability exists for: the system gets worse gradually, and no single request looks wrong. The model changes upstream and the answers shift. The prompt is edited and the tone moves. The traffic mix changes and the evals stop representing the users. Each change is small; the sum is a product that quietly stopped working.

The detection is the continuous eval: a slice of live traffic scored against the same rubric every day, plotted over time. When the curve bends, something drifted — and the trace data says what changed, because the traces record the model version and prompt version of every request.

And the discipline is the response: drift findings get the same treatment as incidents. A drift alert that is acknowledged and ignored is a dashboard pretending to be a safety system. The alert exists to trigger the investigation, not to decorate the screen.

The alerting rule is the same as everywhere else in operations: alert on the things that require action, and silence the things that do not. An alert on every slow request is noise; an alert on the slow-request rate crossing a threshold is a signal. The threshold is the product decision, and it is written down.

The LLM-specific alerts are the ones logs cannot produce: cost per task crossing its budget, the eval score falling below its floor, the retry rate climbing past its ceiling. These are the alerts that catch the failures this post opened with — the silent ones.

And every alert needs an owner and a runbook. An alert that pages nobody is a decoration; an alert that pages someone who does not know what to do is a punishment. The runbook is the difference between observability and anxiety.

For a prototype with no users, the full stack is ceremony — the traces have nothing to record and the dashboards have nothing to watch. The trigger is the same as everywhere else on this blog: the moment someone else depends on the system, the visibility stops being optional.

The overreaction to avoid is the opposite one: instrumenting everything and watching nothing. A trace pipeline that captures every request into a database nobody queries is a cost with a dashboard attached. The stack earns its keep when the numbers are reviewed on a schedule and the alerts have owners.

And the honest sequence: capture the traces first, build the dashboard second, add the alerts third. Each step is only worth building once the previous one is being used. Observability is a habit, not a purchase.