How to Monitor Intermittently Connected Sites Without False Alarms

Define the site contract

A continuously connected server and a customer site that connects once per shift should not share the same missing-heartbeat rule. Record each site's expected connection window, time zone, permitted lateness, owner and local escalation capability.

Keep customer isolation in credentials and authorization as well as labels. A customer identifier in a message is useful routing context but is not, by itself, an access boundary.

Store distinct observations

If local health is unknown, keep it unknown. A received heartbeat from a site agent does not automatically establish that its applications are working.

Choose the deadline mechanism

For simple periodic checks, a receiver with explicit interval and grace semantics can be enough; Healthchecks' configuration documentation (opens in a new tab) describes one such model. More complex customer calendars may require a scheduler or state service that understands those windows. Verify that capability rather than assuming every heartbeat product supports it.

For an illustrative site expected to connect during a morning window, the deadline belongs to that window plus agreed lateness. Show the next expected contact in the operator view so “offline” is interpretable without opening a separate calendar.

Handle reconnects carefully

Keep both observed-at and received-at times, a site identity and an event or sequence identifier. On reconnection, retain delayed records for history, but require a fresh local check before marking current health good. Detect implausible clocks and document how you handle clock skew.

Choose how old fault events affect current paging. A fault that started and recovered while disconnected may belong in a review record rather than a brand-new urgent page. A still-active fault should produce the current incident with its true observed start where trustworthy.

Do not assume telemetry buffering is unlimited. Remote-write guidance (opens in a new tab) documents limits for one forwarding implementation; test the deployed agent's actual retention and recovery behavior.

Exercise the state transitions

In a lab site, disconnect during a planned window, miss a required window, fail a local demo service while connected, and replay older events after restoring access. Confirm the first case does not page, the second becomes overdue, the third reaches the local service owner, and replay does not incorrectly resolve or duplicate a current incident.

Also test an unexpected disconnect during an active local fault. If response must continue while disconnected, it needs a local operator or independent communication path. Central dashboards cannot create reachability that does not exist.

Save the contract and test evidence with the site runbook. This covers monitoring semantics, not the full provisioning, upgrade and lifecycle management of an edge fleet.