Measure Remote-Site Latency with Push Checks

Specify the path you want to measure

For forty remote hosts, first decide whether the question is internet access, reachability to headquarters, or user experience against a particular service. A ping to a nearby resolver measures a different path from a TLS request to your SaaS endpoint. Select a small number of stable targets you own or are permitted to probe.

Use a local target to diagnose the site gateway separately from an external target. Keep each result identified by site, target, protocol, and address family. A site-specific problem should not disappear because the other thirty-nine sites are healthy.

Make the sample interpretable

Run a fixed, bounded sample with a total execution deadline. Record how many attempts were sent and received, rather than reporting an unexplained percentage. For latency, state whether the number is minimum, mean, median, or a percentile. A ten-packet sample cannot provide the same confidence as a long measurement window.

{
  "site": "branch-07",
  "target": "regional-gateway",
  "protocol": "icmp",
  "observed_at": "2026-09-12T10:05:00Z",
  "sample_seconds": 30,
  "sent": 10,
  "received": 9,
  "loss_percent": 10,
  "latency_mean_ms": 42,
  "sequence": 813
}

This is a suggested payload for a receiver you control. Do not send it to a provider endpoint without checking that provider's schema. Some push checks carry a single timing value and status, while others support richer metrics. Store detailed measurements in a suitable metrics system if the receiver only understands healthy or unhealthy.

Separate the measurement from its delivery

The producer should authenticate to the receiver using a per-site credential, send only allowed fields, and keep a bounded local buffer during disconnection. When reconnecting, preserve the original observation times; delayed samples are history, not fresh proof of connectivity.

The central service records its own receipt time and watches a freshness deadline. Deadline checks (opens in a new tab) can detect an absent report. A separate latency rule handles recent but degraded samples. If the local checker stops entirely, the missing-report alert must still work.

Choose thresholds using the actual workload

Compare a quiet baseline with a period when users experience the problem. An ICMP timeout can reflect filtering or rate limiting rather than application failure. Add a small HTTP or TCP observation where it represents the service; the blackbox exporter configuration (opens in a new tab) shows protocol-specific options for one implementation.

Avoid thresholding a single slow packet. Use an agreed observation window, enough samples, and explicit recovery criteria. Keep raw sample count visible so the responder can distinguish persistent degradation from a thin sample.

Exercise reporting failures

Disconnect the test site's uplink, block only the push receiver, and stop only the local measurement process in separate exercises. These may all produce missing reports centrally, but local evidence should distinguish them after access returns. Finally introduce controlled delay in a lab path: recent reports should continue arriving while the quality rule changes state. That proves freshness and performance are being measured independently.