Why AI Observability Alone Won't Catch a Bad Answer

AI observability is the practice of tracing, logging, and monitoring LLM and agent systems in production. It's how you see why an output was wrong, not just that the request returned a 200. It differs from classic APM in one way that matters: a confident, fluent, wrong answer passes every health check you already have.

I've watched teams ship a support bot with clean dashboards, sane token counts, and zero errors in the trace. Then the bot invented a refund policy out of thin air, and a customer believed it. The trace looked perfect. The answer was wrong. Nothing in the observability stack caught it, because nothing in the stack was built to.

That gap, between "the system ran fine" and "the system was right," is what this piece is about. AI observability tells you what happened inside an LLM call. It does not tell you whether the output was correct. Treating the two as the same problem is the most common failure I see in production LLM stacks.

Key takeaways

Before the detail, the claims this piece will defend:

• Observability answers "what happened," not "was it right." A trace with clean spans and a 200 status looks identical whether the model answered correctly or confidently wrong.
• Traditional APM was not built for probabilistic output. It watches latency, errors, and uptime; it has no concept of a wrong-but-healthy response.
• OpenTelemetry's GenAI semantic conventions are the vendor-neutral instrumentation layer. gen_ai.* span attributes now standardize model name, token counts, and tool calls across providers.
• Agent systems need a span per tool call, not per request. A single agent run can produce dozens of spans across tools, retries, and sub-agent hand-offs.
• Observability plus evals closes the loop. Tracing catches what broke; evaluation catches whether it was right. Skipping either one leaves a gap that ships.

What is AI observability? Tracing vs. logging vs. monitoring

AI observability is the practice of instrumenting LLM and agent systems so an engineer can reconstruct what happened after the fact. That means exactly what the model saw, decided, and returned on any given request. It has three distinct layers, and conflating them is where most teams get confused.

Tracing captures the structure of a request: the parent span for the API call, child spans for retrieval, tool calls, and sub-agent invocations, each with a start time, a duration, and an outcome. Logging captures the content: the actual prompt, the actual completion, the retrieved documents, the tool arguments. Monitoring aggregates both into dashboards and alerts: token spend per hour, p95 latency, error rate by model version.

You need all three, and none of them alone tells you if the model was right. A trace shows a clean two-second round trip. A log shows a coherent, well-formatted answer. A monitor shows normal token spend. All three can stay green while the answer is fabricated.

Why traditional APM doesn't catch LLM failures

Application performance monitoring was built for deterministic systems. A database query either returns the right row or throws an error. A REST endpoint either returns 200 with the expected schema or fails loudly. APM tools watch for exactly that: exceptions, timeouts, non-2xx status codes, schema violations.

An LLM call almost never fails that way. It returns 200. The schema validates. The tokens are within budget. And the content can still be wrong: a hallucinated citation, a policy the model invented, a tool call with plausible but incorrect arguments. None of that trips a single APM alert, because APM was never built to ask "was this true," only "did this run."

Teams that bolt an LLM feature onto an existing APM stack get a false sense of coverage. The dashboards stay full and green. The incidents still happen, and they surface through a support ticket or a churned customer, not through an alert.

Traces, logs, and evals: why observability alone isn't enough

Observability and evaluation solve two different problems, and the industry blurs them constantly. Observability answers "what happened." Evaluation answers "was it right." You need both, and skipping either one leaves a gap.

A trace with clean spans, sane token counts, and a 200 status looks identical whether the model gave the correct answer or a fluent, wrong one. Teams that stop at tracing and logging, without wiring in evals, still ship confidently incorrect answers, because nothing in their stack asks the correctness question. I've made this argument at length in evals that predict production, not vanity metrics: the eval harness is what tells you whether a model deserves to ship. Observability is what tells you why it broke once it already has.

The two systems should share data. Your eval harness should sample from the same traces your observability stack captures. That way the hardest production cases feed the frozen eval set instead of disappearing into a log nobody reads.

Instrumenting LLM calls with OpenTelemetry's GenAI semantic conventions

Until recently, every LLM provider and every observability vendor rolled its own tracing schema. That fragmentation forced teams to write and maintain a translation layer every time they switched tools.

OpenTelemetry's GenAI semantic conventions fix this by standardizing the span attributes for LLM and agent calls: gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, finish reason, and tool-call structure, all under a common gen_ai.* namespace. Datadog, AWS, Azure, and Google Cloud have already adopted the convention, per OpenTelemetry's own write-up. That gives teams a vendor-neutral instrumentation layer instead of a one-off SDK per provider.

Here is a stripped-down example of what an instrumented span looks like. The values are illustrative, shaped to show the attribute structure, not pulled from a specific live system.

Core chat and embedding attributes are stable enough to build a production dashboard on today. Agent and tool-orchestration conventions are still settling as the spec matures through 2026, so expect some churn if you instrument multi-agent systems early.

Tracing multi-agent and tool-calling systems

A single-turn LLM call produces one span. An agent produces a tree. Each reasoning step, each tool call, each retry, and each sub-agent hand-off should get its own span, nested under the parent run, what practitioners call span-per-tick tracing.

Picture an agent that retries a flaky search API three times before giving up and answering from memory instead. Without span-per-tick tracing, you see one long request and a plausible answer. With it, you see the three failed tool calls, the fallback decision, and the moment the agent quietly stopped using the tool it was supposed to rely on. That's the difference between debugging from a guess and debugging from evidence.

The trace tree also shows where the token spend goes. A single agent run can burn 10 times the tokens of the final answer once you count every retry, every retrieved document, and every intermediate reasoning step. If you are only logging the last message, you are blind to the other 90% of the run.

Choosing a stack: cost, sampling, and the overhead trade-off

By early 2026 the agent-observability market had consolidated to a handful of production-grade platforms: LangSmith, Langfuse, Arize Phoenix, Helicone, Datadog LLM Observability, and Honeycomb LLM Observability. That's down from a far more fragmented field two years earlier, per digitalapplied.com's 2026 platform comparison. Pick based on where your traffic already lives, not a feature checklist. The differences that matter most are pricing model and how deep the free tier goes.

Prices shift, so verify current numbers before you budget (source: Braintrust's 2026 LLM cost-tracking comparison). The trade-off underneath the pricing is the real decision: full-fidelity logging, every prompt and every completion, is expensive at scale and a compliance liability if your prompts carry PII. Most teams end up sampling.

Sampling is where the honest cost shows up. You cannot log everything, so you log a slice, and the one trace you need during an incident is often the one your sampling policy skipped. The fix is targeted, not "log more": sample low-confidence outputs and flagged sessions at 100%, and sample routine, high-confidence traffic much lower. Designing that policy deliberately, instead of backfilling it after the first incident, is core to the observability and deployment automation work I see holding up in production.

From observability to guardrails: closing the loop in production

Observability is not the end state. It is the sensor layer that feeds two things: your eval harness and your guardrails. A trace that surfaces a hallucinated policy should route straight into your next eval set, not just into a postmortem doc nobody revisits.

Teams that close this loop treat every production incident as a new eval case, frozen into the next version of the harness. Every recurring failure mode becomes a candidate for a guardrail: a validation check, a retrieval constraint, a tool-permission boundary that stops the failure from reaching a user at all next time.

This is where engineering and revenue meet directly. Every incident your observability stack catches before a customer does is a support ticket you didn't have to eat, a churn risk you didn't create, and trust you didn't have to rebuild. Observability that only produces dashboards nobody acts on is a cost center. Observability wired into evals and guardrails is a margin decision.

My book Observability for AI Systems goes deeper on the trace-log-replay architecture: how to make any answer replayable before you need to explain it to a customer, not after.

FAQ

What's the difference between AI observability and LLM evaluation?

Observability tells you what happened: which model ran, what it retrieved, what tools it called, how long it took. Evaluation tells you whether the output was correct against a rubric or reference. You need observability to debug a failure and evals to decide if a model deserves to ship in the first place.

Do I need OpenTelemetry to trace LLM calls, or is a dedicated tool like Langfuse better?

They aren't competitors. OpenTelemetry's GenAI semantic conventions give you a vendor-neutral instrumentation layer; Langfuse, Arize Phoenix, and similar tools are where those traces land, get visualized, and get scored. Most production stacks in 2026 instrument with OpenTelemetry and ship the traces to a dedicated LLM observability platform.

How much does LLM observability cost at production scale?

It scales with trace volume, not seats. Free tiers cover early-stage traffic (Langfuse's Hobby plan gives 50,000 units a month). Production teams typically land in the low hundreds of dollars monthly once they cross 100,000 spans, plus whatever it costs to store full-fidelity logs if you aren't sampling. Budget for growth, not your current volume.

Can observability catch an AI agent that's hallucinating but sounds confident?

Observability alone, no. A trace with clean spans and a 200 status looks the same whether a tool-call result was real or fabricated. Catching a confident hallucination requires an eval or guardrail layer that checks the claim against a source of truth, not just that the call completed.

If you are instrumenting an LLM or agent system, you want the trace, log, and eval layers wired together from day one. Not bolted on after your first incident. That's the kind of production observability and monitoring build I'd point you toward. Build the sensor layer first. The guardrails and evals it feeds are what stop the next wrong answer from reaching a customer.