How to Test Prometheus Alert Rules Before Removing the Old Rule

Separate the four acceptance gates

A merged configuration file is evidence of source control, not runtime coverage. Before changing a rule, save its expression, labels, duration, receiver and reason for paging. State the intended difference: for example, “brief scrape failures should not page, but two continuous minutes of failure should.”

Create a minimal behavioral fixture

Use promtool matching your deployed Prometheus version. Save this teaching rule as rules.yml. It observes one named demo target, not every service in your estate.

groups:
  - name: demo-health
    rules:
      - alert: DemoTargetDown
        expr: up{job="demo"} == 0
        for: 2m
        labels:
          severity: test
        annotations:
          summary: Demo target unavailable

Save the fixture as rules.test.yml in the same directory:

rule_files:
  - rules.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'up{job="demo",instance="demo:9090"}'
        values: '1 0 0 0 1'
    alert_rule_test:
      - eval_time: 0m
        alertname: DemoTargetDown
        exp_alerts: []
      - eval_time: 2m
        alertname: DemoTargetDown
        exp_alerts: []
      - eval_time: 3m
        alertname: DemoTargetDown
        exp_alerts:
          - exp_labels:
              job: demo
              instance: demo:9090
              severity: test
            exp_annotations:
              summary: Demo target unavailable
      - eval_time: 4m
        alertname: DemoTargetDown
        exp_alerts: []

promtool check rules rules.yml
promtool test rules rules.test.yml

The first failing sample is at minute one; at minute two the rule is still pending. This illustrates why checking only a final firing state misses timing errors. The unit-test specification (opens in a new tab) defines fixture labels and evaluation times.

Add independent cases for continuously healthy input, missing input, stale input and a mismatched job label. This expression produces no alert when its target series disappears entirely. If target absence must page, design an explicit absence check against the expected target inventory; do not claim up == 0 covers it.

Verify the running instance

Deploy through your existing configuration workflow. Inspect the authenticated instance's /api/v1/rules?type=alert endpoint, documented in the HTTP API (opens in a new tab). Match the rule name, expression, duration and labels; inspect health, errors and evaluation timestamps. Repeat for every evaluator replica that matters.

An inactive rule can be healthy when its condition is false. An empty match or old expression means deployment has not been proven. For an operator-managed installation, inspect the generated rule as well as the resource you edited.

Exercise notification and recovery

Route the candidate rule to a dedicated test receiver before turning it on. Make that route explicit and confirm it does not continue into production receivers. In a lab, create the intended signal, observe pending and firing states, receive the notification, then clear the signal and check recovery behavior. Verify the receiver's resolved-notification configuration rather than assuming a recovery message is enabled.

For production comparison, preserve the old paging path while the candidate is observational. Compare event windows and labels. Before cutover, write the exact changes that enable the new production route and disable the old one; rehearse how active incidents and deduplication behave across that boundary.

Retire with evidence

Keep the old configuration version, behavioral tests, API observations, receiver timestamps and rollback steps together. Restore the previous rule and route if the candidate fails coverage. Clean up the shadow rule and test route when comparison ends. A passing lab test is evidence for its cases, not a promise that arbitrary production data behaves identically.