Why AI stacks broke traditional monitoring
Classic observability was designed for deterministic request and response systems. A web server either returned a page or it did not. A database query either completed within its threshold or it did not. Dashboards, thresholds, and on-call rotations all assumed that the same input produces roughly the same output, and that failure announces itself as an error code or a latency spike.
AI systems break that assumption in four distinct ways.
Non-deterministic output. The same prompt can produce different completions on two consecutive calls. Temperature, sampling, retrieval results, and upstream model updates all introduce variance that no static threshold can describe. A service can return HTTP 200 with a perfectly valid payload while the answer inside it is wrong, off-brand, or hallucinated.
Quality as a first-class signal. Latency and error rate tell you whether the service is alive, not whether it is useful. Teams increasingly need signals like groundedness, retrieval hit rate, refusal rate, format validity, and user correction frequency. None of those appear in a standard APM dashboard.
Accelerator-bound resource profiles. GPU memory pressure, batch queue depth, and kernel occupancy behave very differently from CPU and heap metrics. A model server can look idle from the outside while a queue inside it quietly grows for minutes before timing out.
Fast-moving configuration. Model versions, prompt templates, retrieval indexes, guardrail rules, and embedding models change weekly. Without version identifiers attached to telemetry, a quality regression is nearly impossible to attribute to a cause.
The practical consequence is that monitoring has to expand from "is the system up" to "is the system behaving." That is exactly the gap where open source diagnostics and AI analysis meet: open standards supply trustworthy, portable evidence, and AI supplies pattern recognition over that evidence at a scale humans cannot read manually.
What open source diagnostics brings to AI systems
The phrase covers a broad set of projects, but in practice it means four capabilities:
- Vendor-neutral instrumentation. OpenTelemetry has become the default way to emit traces, metrics, and logs from application code. Its generative-AI semantic conventions describe model calls, token usage, tool invocations, and retrieval steps in a shared vocabulary, so a trace from one framework looks structurally similar to a trace from another.
- Self-hostable storage and query. Prometheus, VictoriaMetrics, ClickHouse, Loki, and object storage backends let you keep raw evidence on infrastructure you control, with no per-host licensing ceiling.
- Data lineage and experiment tracking. OpenLineage and Marquez record how datasets and model artifacts flow through a pipeline; MLflow and similar tools record which run produced which artifact.
- Evaluation and drift libraries. Open evaluation harnesses and drift detectors turn quality into numbers that can be stored next to latency and error rate.
The strategic value is not cost alone. It is auditability. When a customer asks why a summarization feature returned a specific answer three weeks ago, a proprietary black box may only offer aggregated charts. An open pipeline lets you retrieve the exact span, the model version, the prompt template hash, the retrieved document identifiers, and the sampling decision that produced that record.
That property matters more, not less, as AI-generated analysis enters the loop. An AI system that ranks probable root causes is only as trustworthy as the evidence it points to. Open diagnostics make that evidence available, addressable, and reproducible.
Where AI changes the diagnostics loop
AI does not replace dashboards. It changes what happens between "a signal fired" and "an engineer knows what to do." Three tiers of usefulness are worth distinguishing.
Assistive analysis. Summarize an incident, cluster thousands of near-duplicate log lines into a dozen templates, translate a natural-language question into a query, or draft a timeline from spans. These are low-risk because a human verifies the output before acting.
Predictive analysis. Forecast queue saturation, detect multivariate anomalies that no single threshold captures, or flag a slow drift in answer quality before users complain. These require careful baseline management, because a model trained on last month's traffic distribution will happily call normal growth an anomaly.
Autonomous action. Automatically roll back a model version, drain a node, or shift traffic to a fallback path. This is where the risk concentrates. Autonomy should be reserved for narrow, reversible actions with clear blast radius limits and an audit trail.
A useful rule: every AI-generated hypothesis must link back to raw evidence with a query that a human can rerun. If the system says "retrieval failures caused the quality drop," it should be able to produce the traces showing empty retrieval results, the timestamp range, and the index version involved. Explanation without a reproducible query is just a plausible-sounding guess.
Building a verifiable telemetry pipeline
The foundation is unglamorous: identity, sampling, and retention. Get these wrong and no amount of downstream intelligence will save you.
Schema and identity
Every request needs one trace identifier that survives from the edge gateway through retrieval, model inference, tool calls, and post-processing. Attach semantic attributes to each span: model name and version, prompt template hash, decoding parameters, token counts, retrieval index version, document identifiers, guardrail decisions, and cache hit or miss.
Hash prompts rather than storing them raw in high-cardinality metric labels. Prompts are effectively unbounded, and a metric label with a million distinct values will destroy your time-series database. Keep the full prompt in sampled span attributes where it is retrievable but not indexed as a dimension.
A minimum viable attribute set for a model call span:
gen_ai.systemandgen_ai.request.modelgen_ai.usage.input_tokensandgen_ai.usage.output_tokens- a deployment or artifact revision identifier
- a prompt template hash
- a retrieval index revision identifier
- a request-scoped correlation identifier that also appears in application logs
Sampling without blind spots
Head-based sampling at a fixed rate is cheap and simple, and it is the fastest way to lose the rare traces you care about. Tail-based sampling at the collector keeps complete traces that contain errors, exceed a latency threshold, or hit a specific quality score, while sampling successful traces at a much lower rate.
Add a second, separate path for quality evaluation: run offline scoring on a sampled subset and write the results back as metrics. This keeps expensive evaluation out of the request path while still producing trend data.
Retention and cost
Use tiered storage. Keep raw spans in a hot, indexed store for a short window measured in days. Move compressed raw payloads to cheap object storage for a longer window measured in weeks or months. Keep pre-aggregated metrics essentially forever.
Control cardinality aggressively. Bucket latency histograms, avoid unbounded user identifiers in labels, and never put raw document text into a metric dimension. Most observability cost overruns in AI systems come from cardinality, not from volume of requests.
Integration patterns for AI-generated insights
There are four common places to put the intelligence layer, and the right choice depends on latency requirements and how much you trust the output.
Collector-side enrichment
A pipeline processor inspects each span batch, calls a small local inference service, and tags spans with an anomaly score, a cluster identifier, or a classification label. Application code stays untouched, and the enrichment is uniform across every service that exports telemetry. The tradeoff is that collectors become stateful and need their own monitoring.
Agent and sidecar deployment
A DaemonSet-based agent collects host and accelerator metrics through exporters for GPU utilization, memory, temperature, and queue depth. This pattern is essential for AI workloads because the most informative signals often live outside the application process entirely.
Batch analytics layer
A nightly job reads the previous day's traces from object storage, computes drift metrics, embedding distribution shifts, and evaluation scores, then writes the results back into the metrics store. Batch is the right home for anything expensive or requiring a full-corpus view.
Streaming pipeline
Spans and events flow through a message bus into a feature service that scores them in near real time. This supports live anomaly detection and dynamic routing, but it multiplies operational complexity. Adopt it only when the latency reduction changes a decision you actually make.
A pragmatic default for most teams: start with collector-side enrichment for tagging and a batch layer for quality and drift. Add streaming only when a human process is measurably too slow.
Security, privacy, and auditability
The moment you capture prompts and completions, your observability pipeline becomes a system that stores user content. Treat it accordingly.
Redact at the collector, not just in the application. Applications get refactored, new services get added, and one forgotten code path can leak personal data into a store with a long retention window. Enforce redaction rules centrally so the default is safe.
Encrypt sensitive fields. Field-level encryption for prompt and response payloads limits the blast radius of a storage compromise. Pair it with short retention for anything containing raw user text.
Scope access by role. Trace data is unusually revealing. Give engineers access to the traces they need for their service rather than to everything. Log who queried what, and make those audit logs queryable themselves.
Verify your supply chain. Record container image digests, model artifact hashes, and dependency inventories alongside deployments. When a model behaves differently after an update, you want to prove which artifact changed.
Separate diagnostic AI from production inference. The model that summarizes your incidents should not share credentials, network paths, or rate limits with the model serving customers. Keep the diagnostic path read-only with respect to production systems wherever possible.
Operationalizing: from insight to autonomous action
Design alerts around symptoms
Alert on user-visible outcomes: task failure rate, answer rejection rate, cost per successful request, queue wait time. Cause-based alerts on individual components generate noise and encourage threshold tuning instead of investigation. Define service level objectives for quality and cost, not just availability, and use error budgets to decide when to stop shipping features and fix reliability.
Build an automation ladder
Move up one rung at a time and require evidence before climbing:
- Notify. Send an alert with a link to the relevant trace query.
- Enrich. Attach a summary, the top correlated signals, and the recent change history.
- Suggest. Propose a specific action with a confidence indicator and the evidence behind it.
- Execute with approval. A human clicks once; the system performs a scoped, reversible action.
- Auto-remediate. Reserved for narrow, well-understood, reversible cases such as failing over to a known-good model version.
Keep a human in the loop where it matters
Every automated action needs a rollback path, a rate limit, and a record of what it did and why. If an automated rollback happens three times in a day, that is a signal that something deeper is wrong, not a sign that automation is working well.
Choosing tools: a practical decision framework
Instead of comparing feature lists, evaluate candidates against the constraints of your own system.
- Data gravity. Where does the telemetry already live? Moving a petabyte of spans to evaluate a tool is rarely worth it.
- Query latency versus retention. Real-time debugging and long-horizon trend analysis usually need different stores. Plan for two.
- Cardinality ceiling. Ask what happens at ten million unique label combinations. Get a concrete answer.
- Self-host versus managed. Self-hosting buys control and portability; managed services buy speed. Many teams run a managed tier for real-time debugging and a self-hosted tier for long-term raw storage.
- Escape hatch. Confirm you can export raw data in an open format. Lock-in is measured in export difficulty, not in license terms.
- Team fit. A sophisticated pipeline nobody understands is worse than a simple one everybody can operate at three in the morning.
A phased rollout
Week one: instrument one high-value AI path end to end with a shared trace identifier. Week two: add tail-based sampling and tiered retention. Week three: attach model versions, prompt hashes, and retrieval index revisions to spans. Week four: add one AI-assisted capability, ideally incident summarization or log clustering, and measure whether it reduces time to diagnosis. Resist the urge to add five capabilities at once; you will not be able to attribute any improvement to a specific change.
Common mistakes that quietly break diagnostics
- Instrumenting the application but not the model call. You end up with a beautiful request waterfall and a blank spot exactly where the interesting behavior happens.
- Sampling away failures. Fixed-rate head sampling discards the rare traces that explain incidents.
- Treating AI root-cause suggestions as conclusions. Without a linked, rerunnable query, they are hypotheses.
- Exploding cardinality with prompt or user labels. Prompts belong in span attributes, not metric dimensions.
- Omitting version identifiers. Model and prompt versions are the two most valuable attributes in the entire schema, and they are the two most commonly forgotten.
- Capturing raw user content without redaction. Convenient during debugging, expensive during a security review.
- Building dashboards nobody reads. Tie every dashboard to a decision someone actually makes.
- Automating before you understand. Autonomy amplifies whatever your process already does, including its mistakes.
FAQ
Do I need AI to run observability at all?
No. A well-instrumented pipeline with good sampling, clear SLOs, and solid dashboards solves most problems. AI becomes valuable when volume or dimensionality exceeds what humans can scan, and when you need pattern detection across many signals at once.
What is the single highest-impact change for an AI system already in production?
Attaching model version and prompt template hash to every model call span. It converts unattributable quality regressions into traceable ones, and it costs almost nothing to implement.
How do I evaluate quality without slowing down requests?
Score a sampled subset asynchronously and write the results back as metrics. Keep expensive evaluation out of the request path, but ensure the sample includes failures, not just successes.
Should I self-host everything?
Not necessarily. A common pattern is a managed tier for fast real-time debugging plus self-hosted object storage for raw long-term retention. The important property is exportability, not where each component runs.
How do I keep diagnostic AI from leaking sensitive data?
Redact at the collector, encrypt sensitive fields, scope trace access by role, and keep the diagnostic inference path separate from production serving paths with read-only access to production data.
When is autonomous remediation safe?
When the action is narrow, reversible, rate-limited, and well understood, and when the failure mode it addresses has been seen enough times that its false positive rate is known. Start with failover and rollback; avoid anything that mutates data.
How much telemetry is too much?
When the cost of storing and querying it exceeds the value of the decisions it informs. The practical fix is tiering and cardinality discipline, not blanket sampling reductions that hide the rare events you need.
Does this only apply to large teams?
No. A single well-instrumented AI feature with a shared trace identifier, tail sampling, and version attributes delivers most of the benefit. Complexity should grow only when a specific decision requires it.
The direction of travel is clear: open standards provide the verifiable substrate, and AI provides the pattern recognition. Teams that invest in that substrate first will get compounding returns from every intelligence layer they add later.
