LLM API Observability: Logging, Tracing, and Alerting

Production AI APIs fail in ways traditional HTTP monitoring won't catch. A single prompt template change doubles your token consumption. A model node's P99 latency jumps from 2s to 18s with no obvious cause. Your billing statement at month-end is three times over budget. Every one of these problems traces back to the same gap: no real observability on your LLM API calls.

Standard HTTP monitoring falls short because AI API calls carry dimensions that HTTP calls don't — prompt content, token usage, model version, sampling parameters. Any one of them can be the root cause. This guide walks through a complete design, from structured logs to production alerts.

Structured Logging: What to Actually Capture

The most common mistake is logging only the HTTP status code. A proper AI API log entry needs these fields in a structured format:

{
  "trace_id": "abc-123",
  "timestamp": "2026-06-15T10:23:01Z",
  "model": "claude-opus-4.8",
  "provider": "anthropic",
  "prompt_tokens": 842,
  "completion_tokens": 317,
  "total_tokens": 1159,
  "latency_ms": 2340,
  "status": "success",
  "finish_reason": "stop",
  "prompt_hash": "sha256:e3b0c4...",
  "user_id": "u_9912",
  "feature": "doc_summarizer"
}

A few design decisions here are worth spelling out.

Never log raw prompt content. Prompts often contain user PII. Dumping them into Elasticsearch or CloudWatch is a compliance liability. Store a prompt_hash instead — the first 16 characters of a SHA-256 hash is enough for deduplication analysis. Keep the full prompt encrypted in object storage and retrieve it only when you need to investigate a specific incident.

The feature field is how you do cost attribution. Two calls to the same model can belong to completely different parts of your product — a document summarizer versus a support reply bot. Without this field, your monthly bill is just a number. You can't break it down, and you can't prioritize what to optimize.

Always record finish_reason. When length (the truncation signal) appears in more than 5% of your responses, it usually means your prompt design is off or max_tokens is set too low. You can only detect this pattern at the log layer.

For framework choices: Python projects work well with structlog and a JSON handler; Node.js projects work well with pino. Both integrate directly with Datadog and Grafana Loki. For sampling, 10% of normal traffic is fine — but keep 100% of error requests.

Distributed Tracing: Following a Request Across Models and Services

A single user-facing request often triggers a chain of AI calls. A RAG pipeline might query a vector store, assemble a prompt for Claude Sonnet 4.6, then call a smaller model to score the output quality. Without tracing, when latency spikes you have no idea which step is responsible.

OpenTelemetry is the only tracing standard worth investing in right now. Here's a minimal setup:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("ai-service")

with tracer.start_as_current_span("llm_call") as span:
    span.set_attribute("llm.model", "claude-opus-4.8")
    span.set_attribute("llm.prompt_tokens", 842)
    span.set_attribute("llm.provider", "anthropic")
    response = client.messages.create(...)
    span.set_attribute("llm.completion_tokens", response.usage.output_tokens)

Name your span attributes using the OpenTelemetry Semantic Conventions for Generative AI — the gen_ai.* prefix is the emerging standard, and tooling is starting to auto-recognize it.

For backends: self-hosted Jaeger plus Tempo is a solid combination. On the commercial side, Datadog APM and Honeycomb both have mature LLM span support. Honeycomb's BubbleUp feature automatically surfaces shared attributes across high-latency requests, which is particularly useful for identifying problematic prompt patterns.

Core Metrics: What to Measure and How

Here's the full set of metrics to collect, along with recommended aggregation and alert thresholds:

Metric	Type	Aggregation	Alert Threshold
llm_request_duration_ms	Histogram	P50/P90/P99	P99 > 10s → Warning
llm_tokens_total	Counter	by model, feature	Day-over-day > 50% → Warning
llm_request_errors_total	Counter	by error_type	5-min error rate > 2% → Critical
llm_cost_usd	Counter	by model, feature	Hourly spend > 80% of budget → Warning
llm_finish_reason_length	Counter	as percentage	> 5% → Info
llm_context_utilization	Gauge	used / max context	> 90% → Warning

There's a common latency trap to avoid: conflating Time to First Token (TTFT) with total response time. In streaming scenarios, users feel TTFT — not the time the full response completes. At high concurrency, TTFT can differ by 3-5x between models like GPT-5.5 and Claude Opus 4.8. If you only measure total latency, you'll miss this entirely. Capturing TTFT in Prometheus means instrumenting the streaming callback, not relying on the SDK's default timing.

For cost metrics: token pricing varies per model, so do the USD conversion at the collector layer, not in your dashboards. Configure a token-to-USD transform in your OTel Collector processor so Grafana shows dollar costs directly. Otherwise every engineer querying the dashboard has to mentally apply a pricing table.

Alerting: A Tiered Strategy That Doesn't Burn Out Your Team

The fastest way to break an alerting system is routing everything to PagerDuty. Within two weeks, the team stops responding. A three-tier structure keeps signal meaningful.

Critical (respond immediately): Error rate exceeds 5% over any 5-minute window. API is completely unavailable — three consecutive health check failures. Hourly spend exceeds the daily budget ceiling.

Warning (handle during business hours): P99 latency exceeds 10s for 10 consecutive minutes. Daily token consumption is up more than 50% day-over-day. finish_reason=length exceeds 5% of responses.

Info (review in weekly report): Context utilization trends. Performance comparisons when new model versions roll out. Cost distribution shifts across features.

Route Critical alerts to PagerDuty and Slack. Route Warnings to Slack only. Aggregate Info-level signals into a weekly digest. Crucially, every alert message should carry a trace_id or a direct link to a pre-built Grafana query. If an on-call engineer receives a page and has to manually construct a query to start investigating, that's friction you can eliminate.

One alert most teams miss: model fallback detection. If you route requests across a flagship model and a cheaper one for cost optimization, monitor the ratio of traffic hitting the lower-tier model. An abnormal spike in that ratio usually means the flagship model node is degraded — but your error rate won't trigger, because the requests technically succeed.

Rollout Order: How to Build This Without Doing It All at Once

You don't need to stand up the full stack in a single sprint.

Week 1: Add structured logging with structlog or pino. Get model, tokens, latency_ms, and feature fields onto every AI call.

Week 2: Build a Grafana dashboard on top of those logs. Start with latency distribution and token trends. Identify your most expensive features.

Week 3: Instrument with OTel tracing. Focus on multi-step AI workflows first.

Week 4: Set alert thresholds based on the real data you've collected. Ship Critical-level alerts and run a response drill with the team.

For the underlying stack: Loki for log storage if you're small and cost-conscious; Elasticsearch if you need richer querying at scale. Prometheus with Thanos for long-term metrics retention. Tempo or Jaeger for traces. Grafana as the unified frontend. If your team has grown beyond ten engineers and budget isn't a constraint, Datadog's all-in-one platform — including its LLM Observability module with out-of-the-box auto-instrumentation for OpenAI and Anthropic SDKs — can save significant operational overhead.

---

Once this observability layer is in place, the next lever is API pricing. My team routes traffic through XycAi — a single OpenAI-compatible endpoint covering 200+ models, including the full GPT and Claude lineups, starting at 14% of official list prices. It supports Claude Code, Codex, and Gemini CLI without any config changes, so the tooling you already have just works.