Skip to main content

Alert on DNS Failure Without Depending on That DNS

GuideWritten by oncall.fyi editorialPublication approved by Burak YApproval recorded 12 September 2026
Sources and verification
Source dates
Oldest source check: 12 September 2026.
Technical verification
Separate technical verification has not been recorded.

Publication approval and technical verification are recorded separately. Automated link checks establish reachability, not accuracy. A checked example verifies only its stated test cases, not the whole article or your production setup.

In short

Send the diagnostic DNS query explicitly to the local resolver, but give the notification process an independent working resolution path. Verify the actual resolver inside its runtime and test a local-DNS outage using a dedicated fixture. Independence from DNS does not provide independence from shared power or internet access, so add an outside missing-heartbeat check where those failures matter.

Key takeaways

  • The resolver being tested and the notifier resolver have different jobs.
  • Two nameserver entries are not proof of deterministic failover behavior.
  • Test that an alarm actually leaves while the target resolver is unavailable.

Break the circular dependency

Consider a monitor that checks Pi-hole and sends failures to a chat provider. If resolving the chat provider's hostname depends on that same Pi-hole, the monitor may identify the failure correctly and still be unable to deliver the message. Draw the path from target query through message delivery to the phone.

Use an explicit resolver address for the DNS test. Separately configure the monitoring process or dedicated notifier to resolve external notification endpoints through an approved independent resolver. Do not “fix” the diagnostic check by letting it use the healthy external resolver; that would stop testing the failed local service.

Verify the runtime's actual DNS path

A Docker host and its containers may not use the same resolution path. Check the running container's DNS configuration and the network mode rather than only inspecting the host. Docker's DNS documentation (opens in a new tab) describes its embedded DNS behavior and custom configuration options.

Do not assume that listing a second nameserver guarantees the behavior you want. Resolver libraries, timeouts, caching, and network policy affect which server is queried. Test with the target unavailable and with a cold lookup of the notification hostname so a cached answer does not create false confidence.

Keep internal names private. A notifier using a public resolver will not necessarily resolve private service names, and sending private search-domain queries outside may be inappropriate. A dedicated notifier with only the external resolution needs can make this boundary easier to understand.

Prove that the diagnostic query reaches the target

This BIND query shape (opens in a new tab) selects the target explicitly. The address and domain are documentation placeholders.

bash
dig @192.0.2.53 allowed.monitor.example A +time=2 +tries=1

Require the expected DNS response, not merely command completion. Include a controlled allowed record and, when filtering matters, a blocked fixture. The DNS answer guide describes that distinction.

Run the independence test

Use a test resolver or a rule scoped to the monitor's access to the test DNS destination. Avoid stopping the entire household or production resolver without a planned maintenance exercise.

ConditionExpected result
Target DNS healthyFunctional query succeeds
Target DNS unavailableQuery fails; notification still arrives
Target returns wrong answerSemantic check fails
Independent notifier resolver unavailableSeparate delivery-path failure
Entire site uplink unavailableOutside deadline detects silence

Capture query time, attempted send time, and received alert time. A local error log is not evidence that the message escaped the broken DNS path. Re-run after router, container-network, or notification-provider changes.

State the remaining shared dependencies

An external DNS resolver cannot help if the whole site's internet connection or power is down. Use a separately hosted heartbeat receiver to cover that absence, with a notification channel accessible from outside the site. Keep the site alarm and the specific DNS alarm distinct so the responder knows whether the evidence identifies the resolver or only a wider loss of visibility.

Did this help?

Your answer helps us improve this guide. We save only the page and your choice for 30 days.

No name, email, or incident details are requested.

Sources

Vendor facts change. Each source below shows the date this page last checked it.

  1. Docker DNS services Docker. Checked 12 September 2026.
  2. BIND dig reference ISC. Checked 12 September 2026.

Related

One practical idea, occasionally

The On-Call Brief: short field notes, templates, and operational lessons.

Follow the field guide

Email subscriptions are not open yet. Read new guides in your feed reader — no email address needed.

Subscribe with RSS