Building DNS that survives bad days

DNS is often treated as a small configuration file that points a name at an address. In production it is a distributed control system: a delegation at a parent, several authoritative servers, recursive caches, network paths, signing keys, registries, and people changing data under pressure. A web service can be healthy and still appear down when that chain cannot answer or cannot be trusted.

Resilience therefore means more than having two NS records. It means designing so that likely failures are independent, keeping a recoverable copy of correct zone data, preserving the DNSSEC chain of trust, and knowing what caches and health checks can and cannot do. This guide turns those ideas into an architecture, operating controls, tests, and a short incident runbook.

Open the full-size diagram

Start with the failure you are trying to survive

Write the availability objective before choosing a provider. Ask whether the requirement is that a name continue to resolve, that it resolve to a healthy endpoint, that changes can be published during an outage, or all three. These are different properties. A zone can continue serving yesterday’s records while a new address is unavailable; a health check can detect a failed application while the delegation itself is broken; and a control panel can be down while the authoritative data plane keeps answering.

Build a failure inventory at four layers. At the data layer, consider a bad record, a missing record, a stale secondary, a corrupted backup, and a serial that did not advance. At the service layer, consider an authoritative process failure, an expired zone, a DNSSEC signing or validation error, and exhausted capacity. At the network layer, consider a site, transit provider, route, IPv4 path, IPv6 path, firewall, or DDoS event. At the operation layer, consider a compromised account, an unavailable registrar, an unreviewed emergency edit, and the absence of someone who knows how to restore the zone.

Turn each item into a measurable control. For example, define the maximum acceptable age of a secondary, the time to detect a missing NS response, the time to recover a signing pipeline, and the longest period for which an intentionally stale answer may be tolerated. This prevents a vague claim such as highly available from hiding an untested assumption.

The DNS standards describe redundancy as a basic property of a zone, but redundancy only helps when the copies and their paths fail independently. RFC 1035 describes a primary acquiring zone data locally and secondaries obtaining copies through zone transfer. RFC 2182 explains why multiple servers exist: to keep data reachable when one server is unavailable, and to place service closer to resolvers. Treat these as design constraints, not as proof that any particular vendor topology is resilient.

Keep authoritative and recursive DNS separate

An authoritative server answers from a zone for which it has local authority. It returns the zone’s records and, for an authoritative answer, sets the AA bit. A recursive resolver is a client-facing cache that follows referrals on behalf of an application and returns an answer, an error, or a cached result. A resolver may query authoritative servers, but it is not the source of truth for a public zone. This distinction follows the query model in RFC 1034, which defines non-recursive service as local answers or referrals and recursive service as a complete answer for a requester.

Run public authoritative service without open recursion. An authoritative-only cluster should not accept arbitrary recursive work, and a corporate recursive resolver should not be listed as authoritative for the public zone merely because it can answer from cache. Mixing roles increases attack surface, makes capacity planning ambiguous, and can turn an internal outage into a public delegation failure. For internal names, explicitly document whether the source is an authoritative private zone, a forwarding path, or a split-horizon view.

The roles also fail differently. An authoritative outage prevents cache misses and refreshes; a recursive outage affects clients that use that resolver even if every authoritative server is healthy. Test both paths. From outside, query the listed NS addresses directly with recursion disabled and verify an AA response. From client networks, query the configured recursive resolvers and check cache, DNSSEC validation, timeout, and transport behavior. Never infer authoritative availability from a successful lookup at one recursive resolver.

Make every NS a different failure domain

Count failure domains, not NS records. A resilient listed set should spread across independent facilities, power and cooling systems, network paths, autonomous systems or upstreams, and operational teams. A second IP in the same rack is protection against one process crash, not a site or provider outage. Two hostnames served by the same anycast cluster may also be one provider failure domain even when their addresses look unrelated.

RFC 2182 explicitly warns against placing all servers on one LAN, in one building, or behind one link. It recommends topological and geographic dispersion and says three servers are suitable for most organization-level zones, with four or five potentially useful for higher reliability. These are guidance, not a universal magic number. More servers add packet size, cost, and the chance of an undetected misconfiguration. Choose a number after mapping the failures you need to tolerate.

For each NS, record the provider, account boundary, region, facility, transit and peering dependencies, IPv4 and IPv6 addresses, software, data source, and on-call owner. Ask whether a single credential, API, route policy, or deployment pipeline can change all of them. A useful target is at least one independently operated secondary that can continue serving the last accepted zone if the primary provider and its control plane are unavailable.

Diversity must include the parent delegation. The registry publishes the NS and glue information that resolvers use to find your servers. A perfectly distributed zone is still unreachable if a registrar account is locked, DS or glue data is wrong, or a nameserver hostname is inside a broken child zone. Keep registrar access, recovery codes, contacts, and the approved delegation set in a separate emergency procedure, and periodically verify that the parent returns the intended NS, glue, and DS records.

Use providers as independent copies, not as a logo checklist

A multi-provider design is valuable only when the providers are operationally independent. If provider B receives its zone through provider A’s API, uses the same signer, shares the same cloud account, or is deployed by the same failing pipeline, it may not survive the incident you care about. Independence is a property to demonstrate: separate credentials, billing and support paths, change pipelines, networks, data stores, and escalation channels where practical.

The cleanest pattern is one controlled source of zone data with authenticated distribution to several authoritative systems. One system can be the primary for transfers while other providers act as secondaries; alternatively, a release pipeline can publish a validated artifact to each provider. Avoid hand-editing each copy. If a provider cannot accept transfers, export a signed or unsigned zone artifact and publish it through an audited API, then compare authoritative answers after every release.

Do not confuse provider diversity with data consistency. Multiple primaries that accept independent edits can return different RRsets. RFC 2182 notes that unusual multiple-primary configurations can create inconsistencies and are not advisable. Prefer a single write authority, immutable versioned artifacts, and explicit promotion when the primary must change. If an emergency edit is made directly on a secondary, capture it, reconcile it into source control, and prevent it from being overwritten silently.

For a private or hybrid estate, document the forwarding dependencies as carefully as public NS. Google Cloud’s DNS best practices recommends a hybrid approach for some environments but warns that on-premises authoritative servers must remain available and reachable for forwarded names; it also notes that Cloud DNS does not automatically synchronize with them. That is a useful general lesson: a forwarding link is a dependency, not redundancy.

Treat zone data as a release artifact

A zone is a database with a contract. Store its source in version control, review changes, validate syntax and delegation, and retain the exact artifact that was published. A minimum validation set checks that the zone loads, every listed NS has address reachability, glue is coherent, CNAME rules are valid, MX targets exist as intended, TTLs are within policy, and DNSSEC records are generated from the same content that will be served.

The SOA serial is the change indicator used by primary-secondary workflows. Adopt one monotonic policy and make it hard to violate. A date-based serial can be readable; an incrementing integer can be simple. Whichever you choose, a rollback must still produce a larger serial than the one already served, or a secondary may correctly decide that an apparently older version should not be installed. Watch for serial wrap or clock-generated duplicates, especially when multiple automation systems can publish.

Use authenticated transfer and update controls. RFC 2136 defines DNS UPDATE with prerequisites and atomic behavior: either all prerequisites are satisfied and the update is applied, or no update operations take place. That does not make an update safe by itself. Restrict who can update, log the actor and reason, require review for high-impact records, and make changes idempotent. For transfers, allow only intended peers and protect the transfer channel with the mechanism supported by the implementation, commonly TSIG.

Separate low-risk and high-risk records. A short-lived verification TXT record is not the same blast radius as changing the apex, MX, NS, DS, or an authentication endpoint. Require two-person review or a break-glass approval for delegation and DNSSEC changes. Back up both unsigned source and signed output, plus the configuration that tells each provider how to serve it. Test restore into an isolated authoritative instance; a backup that has never loaded successfully is only a hope.

Make DNSSEC a continuity concern

DNSSEC adds authenticity and integrity, not confidentiality. A validator checks signatures and the chain from the parent’s DS record to the child’s DNSKEY. Google’s DNSSEC overview correctly emphasizes that signing the zone is only one step: the parent must publish the DS, and clients need a validating resolver for full protection. A broken signature can make a name fail even when every authoritative server is reachable, so DNSSEC belongs in the availability plan.

Maintain time and signing capacity. RRSIGs have absolute inception and expiration times, unlike ordinary TTLs. RFC 6781 is an Informational IETF operational-practices document from December 2012, not a new protocol standard, but it remains useful guidance on key generation, storage, signing, and rollover. Keep signing keys protected, monitor remaining signature validity, synchronize clocks, and ensure every serving node receives the same signed version before old signatures expire.

Plan ZSK and KSK rollovers separately. A zone-signing key signs ordinary RRsets; a key-signing key signs the DNSKEY set. KSK changes often require a parent DS change, so the parent’s propagation and support process are part of the schedule. RFC 7583, an Informational IETF document from October 2015, describes double-KSK, double-DS, and double-RRset approaches and the waiting states that let cached old data remain verifiable. Do not remove an old key because the new key is visible at your own authoritative server; wait for the relevant cache and parent timing.

Keep a standby emergency key only under a documented, controlled process. Test a rollover in a non-production zone, then rehearse the exact parent interaction and rollback decision. During a provider migration, preserve the DS and DNSKEY relationship until the new operator serves valid signatures from all required addresses. Never publish a DS for a key that is not actually served, and never leave a stale DS after retiring the only key that can validate the zone.

Use TTLs to balance freshness and shelter

A TTL is a cache freshness limit, not a failover switch. Lowering it does not evict records already cached under a longer TTL, and it increases query load precisely when an incident may reduce capacity. Raising it can shelter a stable answer during a short authoritative outage, but it delays legitimate changes. Set TTLs per record class and change them ahead of a planned migration rather than as an emergency reaction.

Remember negative caching. When an authoritative server returns NXDOMAIN or NODATA, RFC 2308 requires the SOA in the authority section so a resolver can cache the negative result, using the lower of the SOA TTL and MINIMUM for that purpose. A mistaken nonexistence response can therefore outlive the record change. Keep negative TTLs intentional, test creation of previously absent names, and do not assume that deleting a bad record instantly makes the corrected record visible everywhere.

Resolver behavior is part of the service contract. A client may use a local stub, a corporate resolver, a public resolver, and intermediate forwarders, each with its own cache and retry policy. Test from representative networks and ask for both positive and negative names. Record the response flags, selected server, TTL remaining, and DNSSEC status. A single successful query from a laptop proves little.

Serve-stale can add a deliberate buffer, but it belongs to recursive-resolver policy, not authoritative configuration. RFC 8767 is an Internet Standards Track document published in 2020 that permits a recursive resolver to use expired data when it cannot refresh an authoritative answer, and suggests a seven-day cap. Stale data trades freshness for availability; it cannot help a name that was never cached, and it may preserve a wrong record. Define which records may be stale, alert when stale mode is active, and keep attempting refresh.

Be honest about health checks and DNS failover

DNS health checking changes answers; it does not move a TCP connection or repair an application. A probe can test an HTTP status, a port, or an endpoint from selected regions, but it may miss a client-specific path failure, an authorization problem, a dependency outage, or a bad application response that still returns status 200. Use more than one vantage point and make the check represent the user journey without coupling it to a single fragile dependency.

Failover is also delayed by caching. A resolver can continue returning the old address until its cached TTL expires, while another resolver has already switched. Clients may cache answers beyond the resolver, and connection pools may retain the old destination. Never promise instantaneous DNS failover from a health check interval alone. Define detection, decision, cache, client, and reconnection times separately.

Document the all-unhealthy behavior. Managed services differ: some return a backup, some return all configured addresses, and some retain the last state. Google Cloud’s routing policies and health checks documentation states that particular policies can return all endpoints when all are considered unhealthy, and describes restrictions when DNSSEC and health checks are combined. Read the exact provider semantics, then test the no-healthy-target case; do not infer safety from the word failover.

Prefer application-layer redundancy behind stable names when possible. DNS can select among regions or providers, but it is a coarse control plane. For stateful systems, ensure a candidate is genuinely ready before returning it. For critical APIs, pair DNS failover with connection draining, replicated state, client retry discipline, and an operator decision to prevent a health-check flap from sending traffic back and forth.

Understand anycast and transport paths

Anycast lets multiple sites advertise the same service address so routing normally sends a resolver to a nearby node. It can absorb a local failure and reduce latency, but it does not turn one operator, signer, or data pipeline into several independent operators. A routing error can affect many nodes at once, and a provider-wide control-plane or security incident can still be common-mode.

RFC 4786 is a December 2006 Best Current Practice for deploying distributed anycast services. It warns that monitoring is harder because availability varies by client location, and recommends representative probes and recording which node answered. Couple route advertisement to service health, damp rapid withdrawals, and provide a node-identification mechanism such as NSID where appropriate.

Test both UDP and TCP on port 53, and test IPv4 and IPv6 independently. Large DNSSEC responses, truncation, firewalls, and path MTU problems can make a service appear healthy to a small query while failing real clients. Verify that each listed address answers the zone, that TCP fallback works, and that an IPv6 address is not delegated merely because it exists in a template. If you use encrypted transport for recursive clients, test that separately from public authoritative reachability; it does not replace ordinary authoritative DNS paths.

Observe answers, not just servers

A process-up metric is insufficient. Monitor from several networks: query each NS directly for SOA, NS, A, AAAA, MX, TXT, and DNSKEY; check AA, RCODE, response time, and consistency; validate DNSSEC from a validating resolver; and measure TCP fallback. Track serial convergence across providers and the age of the oldest secondary. Alert on delegation changes, DS changes, missing glue, unexpected RRsets, signature expiry windows, transfer failures, and sudden increases in SERVFAIL or NXDOMAIN.

Use synthetic probes for both expected and intentionally absent names. An expected name tests the positive path; an absent name tests SOA-based negative caching and DNSSEC authenticated denial. Keep samples from multiple regions and autonomous systems so an anycast or route-local failure is visible. Correlate authoritative logs with resolver telemetry, but remember that authoritative operators cannot see every cached response.

Make dashboards distinguish control plane from data plane. A failed API or console is urgent if a change is needed, but it is not the same as failed authoritative queries. Conversely, a green control plane says nothing about a bad signed artifact. This distinction is explicit in many managed systems; the practical rule is universal: page on user-visible answer failure and separately track the ability to make a safe change.

Drill the recovery, migration, and rollback

Run a quarterly drill with a bounded scenario. Examples include taking one NS offline, blocking a transit provider, serving an old serial from one secondary, expiring a signature in a test zone, losing the primary control plane, and returning all health-check targets as unhealthy. The observer should verify what clients actually see while the operator follows the runbook without privileged tribal knowledge.

A migration drill should start with inventory and end with a reversible delegation. Export and validate the zone, create the new provider’s authoritative copies, compare SOA serial and RRsets from every address, test DNSSEC and TCP, and lower selected TTLs well before changing NS or DS. Add the new NS at the parent while the old set still works. Watch answer consistency from outside. Remove old servers only after caches and operational records have converged, and retain the old provider long enough to roll back safely.

Do not use a short TTL as permission to rush. Parent NS, glue, DS, child RRsets, negative answers, and application clients can have different effective lifetimes. Record the observed maximum and the conservative rollback deadline. If DNSSEC is enabled, treat a DS change as a separate migration with an explicit key state and validation check at each stage.

After the drill, fix the runbook, not just the dashboard. Capture the exact command or API call that restored service, the evidence used to declare convergence, the person authorized to change delegation, and the decision point for serving a stale record or disabling a health policy. Resilience is demonstrated by a repeatable recovery, not by a diagram.

A practical DNS resilience runbook

Before an incident: keep the zone source, signed artifact, provider configuration, registrar contacts, DS details, transfer keys, and restore procedure in an access-controlled emergency location. Record every NS failure domain. Verify direct authoritative answers, serial convergence, DNSSEC validation, TCP fallback, and representative recursive lookups on a schedule.

When users report resolution failure: first classify scope. Query the delegated NS set directly from multiple networks, then query representative recursive resolvers. Check whether the failure is timeout, SERVFAIL, NXDOMAIN, wrong data, DNSSEC validation, or an application endpoint problem. Preserve outputs, timestamps, source locations, and queried names before changing records.

If an authoritative node is bad: remove only that node from service or route advertisement if the procedure is tested, and confirm that the remaining NS answer identically. Do not delete the delegation in panic. If all listed nodes share a provider, activate the independently operated secondary or publish the last validated artifact there, following the approved serial and DNSSEC procedure.

If data is wrong: stop automation, identify the last known-good version, validate it, and publish a new monotonic serial. Avoid deleting records to force refresh; deletion can create negative-cache damage. If the error is limited to an endpoint, correct the record and verify the application before changing TTLs or health policies.

If DNSSEC is failing: check clock skew, RRSIG validity, DNSKEY and DS alignment, missing signatures, and whether every provider serves the same signed version. Do not remove validation protection as a reflex. Use the tested key-recovery or rollover path, involve the registrar for parent changes, and communicate that caches may preserve either the old valid chain or the new failure for different periods.

After recovery: confirm direct authoritative answers and validating recursive answers from multiple vantage points, measure serial and signature convergence, and watch for stale mode, SERVFAIL, NXDOMAIN, and traffic imbalance. Write a timeline, identify the common-mode dependency, and turn the lesson into a test or control.

Conclusion

Two NS records and a short TTL are useful ingredients, not resilience. A DNS design survives bad days when authority and recursion are separated, listed servers cross real failure domains, zone data has one controlled and recoverable source, DNSSEC keys and parent DS records are operated as a continuity chain, and caching and health-check limits are explicit. Add independent observation, migration rehearsal, and a runbook that names the evidence required for recovery. The result is not a promise that DNS never fails; it is a system that fails in bounded ways and gives operators a practiced path back to correct answers.

References and further reading

1. RFC 1034: Domain Names—Concepts and Facilities
2. RFC 1035: Domain Names—Implementation and Specification
3. RFC 2182: Selection and Operation of Secondary DNS Servers
4. RFC 2308: Negative Caching of DNS Queries (DNS NCACHE)
5. RFC 2136: Dynamic Updates in the Domain Name System (DNS UPDATE)
6. RFC 6781: DNSSEC Operational Practices, Version 2
7. RFC 7583: DNSSEC Key Rollover Timing Considerations
8. RFC 8767: Serving Stale Data to Improve DNS Resiliency
9. RFC 4786: Operation of Anycast Services
10. Google Cloud DNS best practices
11. Google Cloud DNSSEC overview
12. Google Cloud DNS routing policies and health checks