How to Test Composite Alert Parity During a Migration

State the operational question behind the composite

Start with a sentence such as: notify when customers see elevated errors and the affected service is receiving meaningful traffic. Then record the exact conditions, scope and required action. A copied expression can look correct while combining different populations.

List each constituent rule's evaluation interval, persistence requirement, recovery condition, grouping and no-data policy. Include any constituent notifications that remain enabled. A composite and its children may all notify unless the destination configuration prevents duplicate paging.

Build the Boolean baseline

For two healthy Boolean inputs, a simple table establishes the intended logical result:

This table is a starting point. It does not specify what warning, unknown, pending or missing states mean, and it says nothing about when A and B were observed.

Add missing and asynchronous states

Create fixtures where A fires while B has no data, B errors while A remains healthy and one condition arrives a full evaluation interval later. Define whether the combined result should wait, become unknown or notify through a separate data-quality route.

Grafana's No Data and Error documentation (opens in a new tab) illustrates why those states need explicit configuration. Do not replace every missing value with zero solely to make a candidate expression return a number.

Test recovery in both orders. The composite may clear when one child clears, while a different policy expects both conditions to remain healthy for a period. Write the actual requirement before comparing engines.

Verify group identity

Create two services and make A true for the first while B is true only for the second. An intended per-service AND must not merge those into one incident. Then test a shared service group, an absent group and a renamed grouping label.

Datadog's composite-monitor documentation (opens in a new tab) describes common reporting sources and grouping constraints. A destination expression over numeric series may need explicit matching to represent the same population. Treat group mismatch as a functional difference, not a cosmetic naming issue.

Compare a timeline in two engines

Feed or replay the same fixture sequence into the candidate and reference paths using a nonpaging review destination. Record evaluation times, constituent states, composite state, first notification and recovery notification.

Use a fixed clock or timestamped fixtures where the tooling permits it. For Prometheus-compatible expressions, unit tests (opens in a new tab) can verify local rule expectations, but an end-to-end exercise is still needed for routing and delivery.

Switch only the accepted behavior

Document any intentional differences, such as a different delay or an explicit unknown-state alert. Have the service owner approve their effect on response before making the candidate authoritative.

Keep the old configuration and a restoration procedure through the pilot. After switching, repeat an urgent case and a recovery case, then confirm the old and constituent routes no longer cause duplicate pages. A successful migration preserves the intended decision and response, even when the configuration syntax changes completely.