Skip to main content

Correlate Uptime Checks Across Two Locations

GuideWritten by oncall.fyi editorialPublication approved by Burak YApproval recorded 12 September 2026
Sources and verification
Source dates
Oldest source check: 12 September 2026.
Technical verification
Separate technical verification has not been recorded.

Publication approval and technical verification are recorded separately. Automated link checks establish reachability, not accuracy. A checked example verifies only its stated test cases, not the whole article or your production setup.

In short

Collect the same application check from independent locations and correlate only fresh observations from the same time window. Keep location and address-family labels. Two failures strengthen evidence of broad impact, but one regional failure still matters. A missing probe is unknown, not a healthy vote. Define and test the incident rule rather than assuming two monitoring instances coordinate automatically.

Key takeaways

  • Independent locations provide context, not automatic proof of a global cause.
  • A majority rule can hide a real regional outage.
  • Stale or missing observations must not count as healthy votes.

Make the observations comparable

Use the same hostname, method, authentication contract, expected content, timeout, and address family from each location. Record what differs intentionally, such as regional CDN resolution. A check of the login page in one region and a private API in another is not a useful quorum.

Choose locations with different relevant failure domains. Two virtual machines in one data center may share transit, power, or a provider control plane. Independence is a property of the network path and operation, not just having two process names.

The multi-target exporter pattern (opens in a new tab) is one way to collect protocol observations separately from their presentation. Whatever implementation you choose, preserve location identity through storage and alert evaluation.

Correlate a bounded time window

Store target, location, observation time, receipt time, protocol, address family, and result. Define a maximum observation age and tolerated clock skew. If locations run once per minute, a short comparison window can be useful, but choose its width using the actual schedule and delay. Never compare today's failed observation with yesterday's successful one.

Check the probes themselves against a small set of reference endpoints. If a location cannot reach any reference, classify that location's target observations as suspect and report a monitoring-path problem. This should not erase independently confirmed target failures elsewhere.

Use more than a binary global vote

Location ALocation BUseful interpretation
Fresh successFresh successTested paths currently healthy
Fresh failureFresh failureBroad failure evidence; investigate shared dependencies
Fresh failureFresh successRegional or path-specific failure
Fresh successMissing or staleOne healthy path; coverage incomplete
Fresh failureMissing or staleConfirmed path failure plus uncertainty

These are proposed decision states, not built-in Uptime Kuma coordination features. Two independent instances do not necessarily share observations or incident keys. Use a supported aggregator or implement a small explicit rule layer with a clear owner.

A two-of-three rule may reduce noisy global paging, but it must not hide the customers served by the failing region. Route regional degradation to an appropriate incident or investigation policy even when the global condition does not fire.

Preserve diagnostic detail

Attach resolved address and failure stage to the incident. Different CDN edges, WAF policy, regional DNS, and IPv6 availability can produce different outcomes without the application origin being completely down. Keep expected protocol settings (opens in a new tab) aligned when upgrading probes.

Test the voting rule with controlled failures

Use fixtures that fail everywhere, fail only from one permitted test network, and stop reporting from one probe. Also replay a delayed healthy observation after a fresh failure. Verify that the aggregate view represents each scenario accurately and that the delayed sample cannot clear a current incident.

Record which observations led to the decision and when they were accepted. After a probe recovers, wait for a new target observation before restoring its vote. This makes regional monitoring useful evidence instead of a mechanism that turns incomplete coverage into an unjustified green status.

Did this help?

Your answer helps us improve this guide. We save only the page and your choice for 30 days.

No name, email, or incident details are requested.

Sources

Vendor facts change. Each source below shows the date this page last checked it.

  1. Prometheus multi-target exporter pattern Prometheus. Checked 12 September 2026.
  2. Blackbox exporter probe configuration Prometheus. Checked 12 September 2026.

Related

One practical idea, occasionally

The On-Call Brief: short field notes, templates, and operational lessons.

Follow the field guide

Email subscriptions are not open yet. Read new guides in your feed reader — no email address needed.

Subscribe with RSS