Prometheus Cardinality Explosion: Find the Label Eating Your RAM

Prometheus was fine for months. Then one morning queries that used to return instantly start timing out, the box's memory graph is a staircase heading for the ceiling, and eventually the process gets OOM-killed and restarts, only to climb again. Nobody deployed a change to Prometheus. Nobody changed the scrape config. And yet it is eating gigabytes it never used to.

What changed is not Prometheus. It is one of your services, months ago, quietly starting to attach a label with too many possible values. A user_id. A raw URL path with an ID in it. A request ID. Each unique combination of label values is a brand-new time series, and Prometheus holds every active series in memory. A labelling decision that looked harmless in staging (five endpoints, ten test users) becomes a multi-gigabyte problem in production (fifty thousand real users) with no code change to blame. This is a cardinality explosion, and throwing RAM at it only buys you time before the next OOM.

The symptom: gradual query slowdown, then OOM

There is rarely a clean error to grep for. The first sign is query latency creeping up: dashboards that loaded in 200ms start taking seconds, then timing out. In the Prometheus logs you eventually see the kernel step in:

level=error msg="Error on ingesting samples" ...
# and from the OS:
Out of memory: Killed process 1234 (prometheus) total-vm:...

The tell is that this developed gradually and with no Prometheus-side change. A one-time spike (a bad deploy that got rolled back) looks like a step. A cardinality leak looks like a slope: a steady, unrelenting climb in series count that never plateaus, because something is generating novel label combinations continuously.

Why unbounded labels destroy Prometheus

Prometheus stores one independent time series per unique combination of metric name and label values. http_requests_total{method="GET", path="/api/users"} and http_requests_total{method="GET", path="/api/orders"} are two separate series. That is by design and it is fine when the label values come from a small, fixed set (HTTP methods, status codes, a handful of endpoints).

It becomes catastrophic when a label's values are effectively unbounded. Put user_id in a label and you get one series per user. Put a request ID in and you get one series per request, forever. Put a raw URL with an ID baked into the path (/api/users/8f21c/orders) and every user creates new paths. Each active series lives in the in-memory "head block" and costs roughly 3-4KB of RAM plus index overhead. Do the arithmetic: one million series is already 4-6GB before Prometheus does a single query. Ten million and you are dead. The label looked reasonable when it was written; its cardinality only revealed itself in production traffic.

Step 1: Confirm total series is the problem

Prometheus exposes its own series count. Query it (in the Prometheus expression browser or Grafana):

prometheus_tsdb_head_series

This is the number of active series in the head block. If it is in the millions and the rate of new series is consistently positive with no matching expiry, you have a leak, not a spike. Check the growth rate directly:

rate(prometheus_tsdb_head_series_created_total[1h])

A sustained positive rate here means something is minting new series continuously. That is the fingerprint of an unbounded label.

Step 2: Find the metric responsible

Rank metrics by how many series each one owns. This query counts series per metric name:

topk(10, count by (__name__)({__name__=~".+"}))

The top result is almost always your culprit, one metric holding hundreds of thousands or millions of series while everything else is in the hundreds. On Prometheus's own web UI, Status, TSDB Status shows the same information without a query: the top metric names by series count and the label names with the highest cardinality, side by side.

Step 3: Find the exploding label

Once you know the metric, find which label is doing the damage. Count distinct values per label:

count(count by (user_id)(http_requests_total))

Run it for each suspect label. The one that returns tens of thousands (matching your user or request count) is the offender. The TSDB Status page's "highest cardinality labels" list gives you this immediately too.

Step 4: Drop the label

The correct fix is at the source: stop putting the unbounded value in the metric. That data belongs in logs or traces, keyed by request, not in metric labels. If you cannot ship a code change immediately, drop or rewrite the label at scrape time with metric_relabel_configs, which runs before ingestion:

scrape_configs:
  - job_name: my-app
    metric_relabel_configs:
      # Remove the user_id label from this metric entirely
      - source_labels: [__name__]
        regex: http_requests_total
        target_label: user_id
        replacement: ""
      # Or collapse ID-bearing paths to a template
      - source_labels: [path]
        regex: "/u/[^/]+/orders"
        target_label: path
        replacement: "/u/:id/orders"

Reload Prometheus (SIGHUP or the /-/reload endpoint if enabled). New scrapes stop creating the bad series. Existing series age out of the head block after the standard retention window.

Verification

Watch the series count stop climbing and start falling:

prometheus_tsdb_head_series

After the relabel takes effect, the creation rate should return to roughly zero and total series should plateau, then drop as the old high-cardinality series expire from the head. Memory follows the same curve down. If the number keeps climbing, you dropped the wrong label or there is a second offender, go back to Step 2.

What people get wrong

"Just give Prometheus more RAM." This is the most common and most expensive mistake. If a label's cardinality is genuinely unbounded, series count grows without limit, so any amount of RAM is eventually exhausted. Doubling memory doubles the time to OOM; it does not fix anything. You are paying more to postpone the same crash.

"Shorten retention to save memory." Retention affects on-disk history, not the head block. Active series still all live in memory regardless of how long you keep old data. This does almost nothing for a live cardinality problem.

"Add a label for everything, we might need it later." This is how the problem is born. Every label is a multiplier on series count. Labels are for values you will actually group or filter by and that come from a small bounded set. Identifiers go in logs and traces. If you need per-user analytics, that is a job for a logging or tracing backend, not metric labels.

When it is still broken

• Series count plateaus but memory is still high. The old series have to age out of the head block, which takes up to the head block duration (a couple of hours) plus a compaction cycle. Give it time, or restart Prometheus during a quiet window to reset the head immediately.
• You found the metric but cannot change the app. The metric_relabel_configs drop above is your interim fix; it works entirely on the Prometheus side.
• Multiple metrics are exploding at once. Often they share one bad label injected by a middleware or exporter. Fix it once at the source of that label rather than metric by metric.
• You want to never be surprised again. Alert on prometheus_tsdb_head_series crossing a threshold tuned to your baseline (say, sustained above 1M when you normally sit at 200k), and on a persistently positive rate(prometheus_tsdb_head_series_created_total[1h]). That catches the leak weeks before the OOM. If Prometheus itself is running in Kubernetes and getting killed, the pattern can look like a generic CrashLoopBackOff until you correlate it with memory.

The durable lesson: a metric label is a promise that the set of values is small and bounded. The moment you attach something per-user or per-request, you have quietly signed Prometheus up for unbounded memory growth, and the bill arrives months later with nothing in the recent git history to explain it. Watch prometheus_tsdb_head_series the way you watch disk and CPU, and this stops being a 3am surprise.