Your Logs, Metrics, and Traces Can't Talk: Fix Correlation Before the Next Incident

It is 2am, checkout is throwing 500s, and you have three browser tabs open. Grafana shows a latency spike at 01:52. Your log tool shows thousands of lines around 01:52. Your tracing tool shows a few slow traces near 01:52. And there you sit, squinting at three clocks, trying to decide which of the thousand log lines belongs to the one slow trace that caused the metric spike.

This is the observability failure nobody buys tooling to fix, because each tool looks fine on its own. The metrics tool is great at metrics. The log tool is great at logs. The problem is that they share no identifier, so a human has to do the join by hand, under pressure, with timestamps as the only common key. That is not observability. That is three dashboards and a stopwatch.

The fix is not a fourth tool. It is one identifier, a trace ID, propagated through every log line, every span, and every metric exemplar, so that "this spike" becomes a click to "these logs" and "this trace" instead of a timestamp-matching exercise. The catch is that you have to build it in before the incident, because retrofitting it mid-outage is impossible.

Why three great tools still cannot answer one question

Teams do not choose to have disconnected observability. It accretes. You adopt a log aggregator first, because logs are the oldest habit. Later you add Prometheus because you want alerting on latency and error rate. Later still you add distributed tracing because a microservice call graph got too deep to reason about. Three purchasing decisions, three vendors, three data models, made years apart. None of them was wrong. But nobody ever propagated a single request identifier through all three, so at the data layer they have nothing in common except wall-clock time.

Wall-clock time is a terrible join key. Under load you have thousands of log lines per second, clock skew between hosts of tens or hundreds of milliseconds, and multiple concurrent requests that all look identical in aggregate. "Show me the logs at 01:52:03" returns a haystack, not the needle. The needle is "show me the logs for the one request whose trace took 4 seconds", and you cannot ask that question if the log lines do not carry the trace ID.

The one thing that makes correlation work: shared trace context

A distributed trace has a trace_id that stays constant for the whole request as it fans out across services, and a per-operation span_id. That trace context is the join key you have been missing. The job is to get that same trace_id onto all three signal types:

1. Logs: every structured log line emitted while handling a request carries the current trace_id (and usually span_id) as fields.
2. Traces: the tracing system already has it by definition.
3. Metrics: latency histograms carry exemplars, sample data points tagged with the trace_id of an actual request that landed in that bucket, so a point on the p99 line links straight to a real slow trace.

OpenTelemetry exists precisely to generate this context once and share it across all three signals from a single instrumentation pass, rather than instrumenting logs, metrics, and traces three separate times. That single-context model is the reason to standardise on it, more than any individual feature. See the OpenTelemetry traces documentation for the context propagation model.

Building it, step by step

Step 1: Generate and propagate trace context at the edge

The first service to touch a request must either accept an incoming traceparent header (W3C Trace Context) or generate a new one, and then pass it on every downstream call. With OpenTelemetry auto-instrumentation this is handled for common frameworks and HTTP clients. The one rule you must not break: never start a fresh, unrelated trace per service. If service B ignores the incoming context and mints its own trace_id, the chain snaps and correlation dies at that hop.

Step 2: Put the trace ID into every log line

Structured logs only. A log line has to be JSON (or another parseable format) with a trace_id field. Pull the ID from the active span context. In Python this looks like:

import logging
from opentelemetry import trace

class TraceIdFilter(logging.Filter):
    def filter(self, record):
        span = trace.get_current_span()
        ctx = span.get_span_context()
        record.trace_id = format(ctx.trace_id, "032x") if ctx.is_valid else "-"
        return True

# attach the filter and include %(trace_id)s in the formatter,
# or add the field in your JSON log formatter

Now a request that errors produces a log line like {"level":"error","trace_id":"7f3a...","msg":"payment declined"}, and that ID is the same one on the trace and the exemplar.

Step 3: Enable exemplars on latency metrics

Exemplars are how a metric spike points back to a specific trace. When you record a request-duration histogram, the metrics SDK can attach the current trace_id as an exemplar on the bucket that request fell into. In Prometheus this arrives via the OpenMetrics exemplar format and renders in Grafana as clickable dots on the graph. Turn it on in your metrics exporter configuration; without exemplars, your latency panel knows that p99 spiked but not which requests caused it.

Step 4: Make the three tools link to each other

Configure your visualisation layer so the IDs become clickable. In Grafana this is a data-source-derived field: a log line's trace_id becomes a link into the tracing backend, and a trace view offers a "logs for this trace" button that queries your log store for trace_id = X. This last step is what converts three tools into one workflow. The plumbing from Steps 1 to 3 makes it possible; this step makes it a single click.

Verification

You have real correlation when you can start from any one signal and reach the other two without typing a timestamp. Test it deliberately:

1. Trigger a slow or failing request in staging.
2. Open the latency panel, click the exemplar dot on the spike. It should open the exact trace for that request.
3. From that trace, click through to logs. You should see only the log lines for that trace_id, not everything that happened in that second.
4. Grep your log store directly for the trace ID to confirm it is really there:

# the trace_id from the trace view should return that request's lines only
grep '"trace_id":"7f3a' /var/log/app/*.log | head

If step 2 lands you on the right trace and step 3 shows only that request's logs, correlation works. If you had to read a clock at any point, it does not.

What people get wrong

Buying a fourth dashboard. A "single pane of glass" that shows all three pillars side by side but shares no ID is still three panes stacked in one window. The value is the join key, not the layout. Ask any vendor exactly how a metric point links to a specific trace and to that trace's logs; if the answer is "by time range", it does not solve your problem.

Logging the trace ID in only one service. Correlation is only as strong as its weakest hop. One service that logs without the trace ID, or one client call that drops the context header, and the chain breaks exactly where you most need it. It has to be a platform default, applied everywhere, not a per-team choice.

Trying to retrofit it during the incident. The instinct at 2am is "let me just add trace IDs to the logs now". You cannot. The requests that are failing right now were emitted by code that was not instrumented, and you cannot go back and tag log lines that are already written. This gap is invisible in day-to-day operation and only becomes obvious mid-incident, which is exactly why it is chronically underfunded relative to its real cost. Treat correlation as a launch requirement, like auth or backups, not a nice-to-have you will get to later. The same discipline applies to reading Linux logs during a 3am outage: the time to make logs searchable is before the outage, not during it.

When it is still broken

• Trace IDs appear in logs but the trace is missing. Your sampling dropped it. Head-based sampling can discard the trace while the log line survives, leaving a dangling ID. Use tail-based sampling so error and slow traces are always kept, or sample at 100% for errors.
• Exemplars never show up in Grafana. Exemplar storage has to be enabled on the Prometheus side and the scrape must accept the OpenMetrics format. A histogram without exemplar support will silently drop them.
• The chain breaks at one specific service. That service is probably starting a new root span instead of continuing the incoming context, or an HTTP client there is not propagating traceparent. Check that its inbound and outbound instrumentation are both active.
• Async work loses the context. Background jobs and message queues do not automatically carry trace context across the boundary. Serialise the traceparent into the job payload or message headers and restore it on the consumer side.