How to Suppress Downstream Alerts During a WAN Outage

Establish the dependency before suppressing anything

A branch losing its WAN link can make every device appear unreachable to a central probe. Those observations share a plausible dependency. A disk filling inside the branch or a sensor reporting an unsafe value is not automatically explained by that same link failure.

Classify child alerts by their detection path. Suppress only the reachability symptoms whose observer actually depends on the affected WAN. Keep independent local signals and customer-impact checks visible through the appropriate route.

Confirm how the upstream condition is detected. If the parent alarm relies on the same unavailable path but cannot distinguish an observer failure, name it accordingly. “Central observer cannot reach site” is more honest than claiming that a particular router is broken.

Label the suppression scope explicitly

Every relevant event needs stable tenant, environment and site labels. Reject or quarantine malformed events before they enter a broad inhibition rule. Missing labels must not accidentally make unrelated alerts look equal.

Here is an Alertmanager configuration fragment for an illustrative naming scheme:

inhibit_rules:
  - source_matchers:
      - 'alertname="SiteWanUnreachable"'
      - 'tenant=~".+"'
      - 'environment=~".+"'
      - 'site=~".+"'
    target_matchers:
      - 'alertname="DeviceUnreachableFromCentral"'
      - 'tenant=~".+"'
      - 'environment=~".+"'
      - 'site=~".+"'
    equal: ['tenant', 'environment', 'site']

Adapt the names to actual events; this fragment does not create either alert. The Alertmanager configuration reference (opens in a new tab) explains source matchers, target matchers and equal labels, including the importance of absent label values.

Separate grouping, delay and inhibition

Grouping combines related notifications. Inhibition prevents a target notification while a matching source is active. A notification delay may give the upstream alert time to arrive before the first child message, but increasing it also delays genuine independent detection.

Measure the typical arrival difference using test events. Choose a small, justified delay only if the resulting detection time remains acceptable. Do not rely on timing alone: a parent event can be late or missing, and an unconditionally delayed child may still need attention.

Keep a dashboard or incident view showing inhibited children. Responders need to know what became unreachable and whether an unusual condition sits outside the expected set.

Use a six-case acceptance exercise

Run the following against a nonpaging receiver first:

Record actual timing as well as counts. A parent that eventually arrives does not explain a ten-minute period without usable coverage.

Keep the topology small enough to maintain

Dependency suppression decays when networks change. Put an owner and review trigger on the rule, particularly for site moves, backup links and split routing. Google's monitoring guidance (opens in a new tab) discusses the maintenance difficulty of complex dependency hierarchies.

If the dependency cannot be stated confidently, group related observations for triage instead of automatically hiding them. Keep the old configuration available and remove the inhibition when a controlled independent failure is missed.