How Grafana, Prometheus and Alertmanager Share Alerting Work
Sources and verification
- Source dates
- Oldest source check: 12 September 2026.
- Technical verification
- Separate technical verification has not been recorded.
Publication approval and technical verification are recorded separately. Automated link checks establish reachability, not accuracy. A checked example verifies only its stated test cases, not the whole article or your production setup.
In short
Identify which process collects data, which process evaluates each rule and which component routes its resulting alert. Grafana can evaluate Grafana-managed rules, while Prometheus evaluates its own configured rules and sends alerts to Alertmanager. A browser dashboard is not the evaluator. Test each dependency separately, because closing a panel and stopping a server are very different failures.
Key takeaways
- Write the evaluator beside every rule instead of inferring it from the dashboard.
- A rule can fire successfully while its notification route fails.
- Closing a browser tests browser independence, not server resilience.
Draw the path for one real alert
Choose a known service condition and write down every hop between the source and the responder. Include the data source, query endpoint, evaluator, notification router and paging destination. A line labeled only “Grafana monitoring” hides too many independent responsibilities.
| Responsibility | Questions to answer |
|---|---|
| Collection | Who reads the target or receives telemetry? |
| Storage and query | Where does the evaluator obtain measurements? |
| Rule evaluation | Which process executes the condition on a schedule? |
| Alert routing | Who groups, suppresses and selects a destination? |
| Delivery | Who attempts email, push, SMS or voice delivery? |
| Response | Which person owns the incident and escalation? |
Two alerts displayed next to each other may follow different paths. Record the answer per rule family, especially after a migration.
Who evaluates the rule and who sends the notification
Prometheus alerting rules (opens in a new tab) run in Prometheus and produce alert state. An Alertmanager configuration (opens in a new tab) then determines grouping and notification routing for alerts it receives.
Grafana Alerting (opens in a new tab) distinguishes Grafana-managed rules from data source-managed rules. A dashboard that visualizes Prometheus data does not automatically tell you which type a particular alert uses. Open the rule's configuration and identify its actual evaluation location.
Write one sentence on each runbook: “This rule is evaluated by this service, reading this data source, and routed through this notification service.” Link to the configuration owner as well as the graph.
Where persistent processing belongs
If an integration polls vendor snapshots and turns them into incident transitions, it needs a service that runs without an operator's browser. Give that service durable transition state, retry handling and a health signal. Putting the output in a dashboard does not provide those properties by itself.
Likewise, a dashboard transformation may change the displayed number without changing the evaluator's query. Compare the exact rule expression, time range and reduction with the panel that responders use. A visual threshold line is not proof that an equivalent notification rule exists.
Test the browser and server separately
First close every dashboard tab and trigger a controlled condition in a test environment. The expected rule should still evaluate and reach the review receiver. This proves only that the browser is not required.
Next test a blocked query source, evaluator restart and notification destination failure independently. Observe the last successful evaluation, current alert state and delivery errors at each stage. Do not stop shared production components just to perform this demonstration.
For critical paths, extend the exercise to a central-service outage using the Kubernetes monitoring continuity guide. Whether local collection or alerting survives depends on the topology you actually deploy.
Define ownership for failed hops
An alert rule can be healthy while notification delivery is broken. A delivery test can also succeed while the rule never evaluates. Assign an owner to evaluation errors, stale collection and failed destinations, then test the full path periodically.
Keep configuration changes in the appropriate source of truth. If rules are provisioned, editing a similar-looking UI object may not change the deployed resource. Confirm the loaded rule and destination after each change, and keep enough timestamps to identify which hop last worked.
Finish the audit with one controlled incident that progresses from detection through recovery. The useful result is an explainable timeline, including the parts that remained unavailable or untested.
Did this help?
Your answer helps us improve this guide. We save only the page and your choice for 30 days.
No name, email, or incident details are requested.
Sources
Vendor facts change. Each source below shows the date this page last checked it.
- Grafana Alerting fundamentals — Grafana Labs. Checked 12 September 2026.
- Alerting rules — Prometheus. Checked 12 September 2026.
- Alertmanager configuration — Prometheus. Checked 12 September 2026.
Related
One practical idea, occasionally
The On-Call Brief: short field notes, templates, and operational lessons.
Follow the field guide
Email subscriptions are not open yet. Read new guides in your feed reader — no email address needed.
Subscribe with RSS