How to Add Alert and Recording Rules to a PromQL Backend

Separate the capabilities in your requirement

A backend may accept a PromQL query and return a graph while providing no scheduler for alerting or recording rules. Write the missing capability precisely: periodic expression evaluation, pending and firing state, notification output or storing derived metrics.

Also distinguish API compatibility from expression compatibility. An endpoint can resemble the Prometheus API while differing in supported functions, timestamps, label handling or treatment of absent series. List the expressions your production rules actually use rather than testing only a constant query.

Inventory the external ruler's connections

vmalert documentation (opens in a new tab) describes one external evaluator and its query, notification and recording-rule integrations. Treat it as a candidate whose documented protocols must match your backend. Do not assume a database exposing queries also implements the required write protocol.

Start with two deliberately simple rules

Use one alert expression with a short, known persistence requirement and one recording rule that produces a predictable aggregate from a fixture series. Evaluate the original expression directly and compare it with the recorded output over the same timestamps.

For the alert, observe inactive, pending, firing and resolved behavior. Prometheus alerting-rule documentation (opens in a new tab) provides a reference model, but the candidate engine's supported behavior and persistence settings need their own verification.

Keep the example's evaluation interval and tolerance explicit. If source data arrives late, a ruler querying exactly the present instant can miss measurements that a dashboard later displays. Select and test an evaluation offset where the engine supports it and the latency requirement permits it.

Test absence, errors and recovery

Feed a missing series, a query timeout, a malformed response and a delayed successful response. Record whether the rule becomes unknown, retains state, resets its timer or resolves. Each choice affects how much responders can infer from silence.

Restart the ruler while the test alert is pending and again while it is firing. Check which state is restored and what happens to repeat notifications. Then interrupt the recording destination while queries still succeed; an apparently healthy evaluator must expose failed writes.

Use rule unit testing for compatible expressions, while recognizing that a unit test of Prometheus semantics cannot certify another engine's transport or persistence behavior.

Switch authority only after comparison

Run the candidate against a review destination while the current production rule remains authoritative. Compare detection, grouping identity and recovery timestamps for representative events. Agree acceptable differences before seeing the results so a mismatch is not waved away after the fact.

Keep only the selected path paging when the transition happens. Retain the previous rules and a rollback instruction until the candidate passes normal and failure exercises. If your rules live in Kubernetes resources, handle CRD discovery and conversion as a separate deployment contract.