How to Limit Prometheus Metric Cardinality at the Source

Identify where the growth begins

A storage bill, a failed scrape and an application running out of memory can share the same cause: an unbounded label. Inspect recently changed metrics before changing retention. Check the target's memory, exposition size, scrape duration and active series together. A backend limit may protect ingestion while the instrumented process continues allocating series.

Prometheus label guidance (opens in a new tab) warns against dimensions such as user IDs and email addresses. The safe replacement depends on the question: use a route template for endpoint behavior, a bounded outcome for success or failure, and logs or traces for individual requests.

Write a label budget

For one counter, estimate the upper bound as the product of its label-set sizes and instances. A teaching example with 20 routes, 4 methods, 5 status classes and 10 instances permits up to 4,000 combinations. Actual observed combinations may be fewer. A classic histogram exposes buckets and additional series, so a counter-sized estimate is insufficient for the same labels on a histogram.

Do not normalize by merely deleting query strings if path segments still contain IDs. Prefer the route pattern provided by the framework after routing. Set a single explicit fallback such as unmatched for unknown routes; keep untrusted input out of that fallback value.

Enforce the budget before series creation

Validate label values at the instrumentation boundary, before calling the client library's label lookup. A finite mapping is easier to audit than a generic cache that admits arbitrary values until memory runs low. Decide how unknown values are counted and expose a bounded counter for rejected or collapsed observations.

If you use a library that limits series, inspect whether it freezes, evicts or aggregates when full. Eviction can create counter resets and series churn. Aggregation into an overflow bucket preserves some totals but loses the dimension you collapsed. Neither behavior should silently turn errors into successes or suppress the only signal used by a paging rule.

For an existing deployment, introduce a new bounded metric alongside the old one briefly, with explicit resource limits and a small canary. Update dashboards and rules deliberately; do not leave two equivalent metrics counted together indefinitely.

Measure the effect without overloading Prometheus

On a healthy, appropriately sized test instance, this instant query lists metric names with the most active series:

topk(10, count by (__name__) ({__name__!=""}))

It scans a broad set of series and can itself be expensive. Narrow the selector to a job or inspect the TSDB status API (opens in a new tab) when investigating a stressed system. Compare the result with exporter memory and scrape size; active-series counts alone do not describe historical churn.

Load-test with many unique request identifiers and unfamiliar routes. The bounded metric should stop growing after the intended combinations while its total observations continue reflecting the test traffic. Test ordinary routes, errors, the fallback and process restart. Compare the counters with an independent request count and check that alert queries still return their expected labels.

Understand the downstream limits

metric_relabel_configs can drop unwanted samples before ingestion, but does not remove exporter allocations. A labeldrop action removes labels; it does not sum series whose remaining labels become identical. That collision can create invalid duplicate samples instead of a useful aggregate. Prefer bounded instrumentation or explicit recording rules with understood costs.

Keep a rollback to the last working instrumentation and alert definitions. If an emergency ingestion drop is necessary, record which coverage is missing and restore it deliberately. Follow with the unused metrics audit rather than assuming every expensive metric is dispensable.