How to Get Alerts When Your Internet or Monitoring Server Goes Down
Sources and verification
- Source dates
- Oldest source check: 10 September 2026.
- Technical verification
- Separate technical verification has not been recorded.
Publication approval and technical verification are recorded separately. Automated link checks establish reachability, not accuracy. A checked example verifies only its stated test cases, not the whole article or your production setup.
In short
Put the missing-heartbeat detector outside the network you are monitoring. Have the local monitoring process report regularly, and let the outside detector notify you when reports stop. Add a separate external HTTP check for public services. Test host failure, internet failure, and application failure separately: these checks observe different things, and none alone proves the whole system is healthy.
Key takeaways
- An inside monitor cannot report through an internet connection that has failed.
- A heartbeat and an external HTTP check cover different failure paths.
- The detector and your notification channel need independence from the monitored site.
Why the local dashboard stayed green
A monitor running beside your services shares their electricity, router and upstream connection. If those disappear together, it cannot send a final failure message. Moving a dashboard to another container on the same machine does not change that dependency.
The useful question is: who notices when this monitor stops speaking? An outside watchdog can notice silence without receiving a final message. A public HTTP probe answers a different question: can a client outside your network use the service?
Set up two independent observations
- Choose an outside heartbeat receiver in a separate failure domain. A small remote VM or hosted receiver can serve this role; a VM behind the same router cannot.
- Create a distinct check for the monitoring host. Configure its expected interval and grace period explicitly. Healthchecks documents these settings in its check configuration guide (opens in a new tab).
- Send a heartbeat only after a local health check of the monitoring process succeeds. A separate timer that always sends a ping proves the timer is alive, not that the monitor is doing useful work.
- Configure the receiver's missing-heartbeat alert to reach a phone over a path that survives the local network outage. A chat server hosted at home is not an independent destination.
- From outside, probe each important public service over HTTPS and check expected content. Blackbox exporter (opens in a new tab) is one implementation of external probes.
Record both observations; do not combine them into an unexplained green badge.
| Heartbeat | External service | First investigation |
|---|---|---|
| Recent | Available | Both observed paths worked at the sample times |
| Missing | Available | Monitor, sender, or outbound heartbeat path |
| Recent | Unavailable | Public DNS, inbound access, TLS, or application |
| Missing | Unavailable | Shared site failure is possible; confirm before naming a cause |
Choose a detection budget
For an illustrative lab, suppose reports arrive every minute and the detector allows two additional minutes of lateness. If your receiver measures interval plus grace from the last report, it declares a miss after about three minutes without a successful report, before notification delays. Check your receiver's exact semantics rather than assuming this arithmetic applies everywhere.
Choose the budget from how quickly someone can act and how much routine jitter exists. A five-second alarm for a router nobody can reach for an hour produces urgency without a useful response. Record the last accepted report, expected deadline and actual notification time during the test.
Prove the separation
Run these on a lab host or during an agreed maintenance window, preserving management access.
| Controlled change | Expected evidence | What a failure reveals |
|---|---|---|
| Stop the monitor process | Heartbeat becomes late | Sender may be checking only its own timer |
| Block the lab site's outbound path | Outside detector reports silence | Receiver or notification depends on that same path |
| Stop only the demo web service | External HTTP check fails | Probe targets the wrong endpoint or checks too little |
| Restore each component | Recovery recorded once per check | Recovery rules or duplicate notifications need work |
Keep a usable response note
Save the check name, site, owner, last-seen time, deadline, outside probe result and a backup contact in the runbook template. Start investigation from those observations. Silence alone cannot distinguish a failed power supply from a blocked outbound request.
Re-test after moving the monitoring host, router or notification service. For cameras and other services that should remain private, follow the outbound-only heartbeat guide.
Did this help?
Your answer helps us improve this guide. We save only the page and your choice for 30 days.
No name, email, or incident details are requested.
Frequently asked
- Does a heartbeat prove my website is reachable?
- No. It proves the sender reached the receiver at a particular time. Use an external HTTP check to observe inbound service access.
- Is a second VM enough?
- Only if its relevant dependencies are independent. Document shared power, network, provider and notification dependencies before relying on it.
Sources
Vendor facts change. Each source below shows the date this page last checked it.
- Configuring checks — Healthchecks. Checked 10 September 2026.
- Blackbox exporter — Prometheus. Checked 10 September 2026.
Related
One practical idea, occasionally
The On-Call Brief: short field notes, templates, and operational lessons.
Follow the field guide
Email subscriptions are not open yet. Read new guides in your feed reader — no email address needed.
Subscribe with RSS