How to Find Unused Prometheus Metrics Without Breaking Alerts

Establish what the inventory can see

Start with a defined Prometheus deployment, job and observation period. Capture metric names and series counts, the loaded recording and alerting rules, dashboard definitions and known external consumers. Include federated readers, remote-write destinations, autoscalers, scheduled reports and runbooks that query metrics only during an incident.

The Prometheus query log (opens in a new tab) records queries when configured. It needs its own file permissions, rotation and disk capacity. Query text and selectors can contain sensitive information, so keep the log within the same access boundary as the monitoring data. An empty log is not proof of inactivity if logging was disabled or queries passed through a different instance.

Trace dependencies in both directions

For every candidate metric, identify its direct consumers and any derived series. Suppose a page uses service:request_errors:rate5m, and a recording rule builds that series from http_requests_total. Searching only the alert expression for the raw metric would wrongly label the input unused.

Use a PromQL-aware parser or a tool that understands selectors rather than relying only on text search. Regular expressions over __name__, joins, variables and generated expressions deserve manual review. Follow recording-rule dependencies until you reach collected metrics; finding only the final output is insufficient.

Classify candidates before changing collection

Place each candidate in one of four groups: required now, retained for rare diagnosis, unknown owner or safe to trial dropping. Attach an owner and reason. A metric covering an uncommon disk failure can be worthwhile even if nobody queried it during an uneventful week.

Estimate savings using samples and bytes as well as metric-name count. One metric with unbounded labels can cost more than hundreds of small metrics. If the problem is a runaway label, use source cardinality controls first instead of removing the diagnostic signal entirely.

Make a narrowly scoped canary change

The following fragment is an example inside one reviewed scrape_config. Replace the deliberately specific metric name with a candidate you have actually approved. It drops samples for that exact metric from that scrape job.

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'demo_obsolete_counter_total'
    action: drop

This is not a complete Prometheus configuration. Validate the assembled configuration using promtool check config, then reload through your deployment's normal mechanism and inspect its result. A failed reload can leave the previous configuration active. Confirm the candidate's absence at the intended job and the continued presence of required neighbors.

Do not apply a broad prefix expression across every scrape job because a UI marked the prefix unused. metric_relabel_configs and remote-write relabeling affect different parts of the pipeline; choose the point corresponding to the actual storage or transfer cost.

Test the consumers and the rollback

Run the affected rule fixtures, open the dependent dashboards across their usual time ranges, and inspect rule evaluation errors and missing data. Include a replay of the rare failure the owner cares about. Compare canary and unchanged collection using equivalent traffic and intervals.

Restore the prior configuration if a required query or rule loses its inputs. That rollback resumes future scraping; it cannot reconstruct the samples that were never ingested. Keep the trial short enough that this gap is acceptable and document its interval.

After a representative observation window, review the measured reduction and unresolved dependencies before expanding. Keep the approved drop list in source control with owners and reasons. A future alert author needs to know a missing metric was intentionally filtered rather than spending an incident debugging the exporter.