Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

DNS, Anycast, and the CDN Edge: The Addressing Pipeline

Overview

Before any request can be served, something has to answer two questions: where is a copy of this data near the user? and is that copy still alive? In the web’s production architecture, three subsystems collaborate to answer them: the DNS hierarchy (a global, cached, eventually-consistent database of names), anycast routing (BGP’s answer to “nearest replica”), and the CDN edge (caches that hold the actual content, plus a control plane that can rewrite what every edge believes). Each subsystem caches at a different layer with a different coherence story — a resolver caches a name-to-address answer for a TTL, BGP caches a route until a withdrawal arrives, an edge caches an object until TTL or purge — and the end-to-end guarantee the user experiences (“I reached the right nearby server and got fresh content”) is assembled out of three weaker, independently failing guarantees.

That assembly is the connection this page maps. The coupling points are sharp: the TTL bounds how long a resolver may lie after failover; a purge API bounds how long an edge may lie after origin content changes; BGP convergence bounds how long an anycast site may blackhole traffic after a withdrawal. Each boundary turns a correctness question (“where is the truth?”) into a staleness budget question (“how long may a copy disagree?”), and every outage in this pipeline — stale DNS after a datacenter failover, a purge that propagates too slowly, an anycast site draining — is one of those budgets being exceeded. Treat them as leases with measured propagation, not as instant truth, and the design space opens up.

The Subsystems, Independently

The DNS hierarchy (see ../../networks/dns/resolution.md). DNS is a tree of zones — root, TLD, authoritative — queried through recursive resolvers that cache answers. A full uncached walk costs a chain of sequential UDP round trips (root, then TLD, then authoritative, often two queries each for A/AAAA), which is why resolvers exist: they cache, and every layer between browser and authoritative server may cache (see ../../networks/dns/caching.md). Every answer carries a TTL — a per-record, author-chosen lease on its truth, with formally defined semantics in RFC 2181 (caches may serve a stale answer during lookups in flight, and negative answers are cached under their own TTL derived from the SOA record). DNS is thus an eventually-consistent read-mostly database: writes (record changes, failovers) propagate at the speed of TTL expiry across millions of independent resolvers, and no update can revoke a lease already granted.

Anycast and GeoDNS (see ../../networks/cdn/anycast-and-geo-routing.md). Anycast announces the same IP prefix from many datacenters; BGP’s route-selection rules (shortest AS path, then tie-breaks) steer each client’s packets to whichever announcement is nearest in topology, not in kilometers — see ../../networks/routing/bgp.md. The selection is done per-flow by the routing system, requires no per-client state, and re-homes a client silently when routes change (dangerous for TCP: connections do not survive the hop change). GeoDNS is the complementary, stateful mechanism: an authoritative server consults the resolver’s IP or EDNS Client Subnet, latency and health telemetry, and per-region policy to return different answers to different askers. Anycast gives coarse, automatic, connection-hostile proximity; GeoDNS gives fine, configurable, TTL-bound proximity. Most large CDNs run both: anycast inside regions for resilience, GeoDNS-style steering at the zone apex for cost and capacity shaping.

The CDN edge and origin shielding (see ../../networks/cdn/how-it-works.md). An edge cache stores HTTP objects keyed by origin + URL with per-object cache-control metadata; on a miss it fetches from the next tier and stores. Origin shielding inserts one extra tier: edges that miss fetch from a small set of shield datacenters, and only shields fetch from the origin. This converts origin fetch fan-out from O(number of edges) to O(number of shields) — the same tiered-cache move as a buffer pool sitting between the executor and the file system in ./virtual-memory-caching.md. The CDN also has a write path: the purge/invalidation API, which pushes “this URL is no longer valid” (or “revalidate before serving”) to all edges. Purge is the CDN’s coherence protocol, and it is the only mechanism that can shorten a lease the origin has already handed out.

The Shared Principle

Every subsystem is a cache of authoritative truth with a bounded staleness budget, and the budget is set by a different mechanism at each joint: TTL leases at DNS, BGP convergence at the routing layer, purge propagation at the CDN control plane. More precisely: a DNS answer is a time-bounded grant — the resolver keeps serving it until expiry no matter what the authoritative server now believes, which is exactly the lease structure of ./consensus-leases.md minus the fencing token (DNS has no way to reject a holder of an expired-in-spirit answer). An anycast route is a location-based grant with liveness but no lease: BGP withdrawal can kill it in seconds or leave it stale for minutes, and it is invisible to application logic. An edge object is a validated grant: TTL bounds its age, purge can revoke it early, conditional revalidation (ETag/Last-Modified) checks truth cheaply.

The second shared idea: all three subsystems prefer “serve the stale copy” over “serve nothing.” Resolvers may serve expired records while a refresh is pending; CDNs offer stale-while-revalidate and stale-if-error as explicit product features; BGP keeps forwarding on the best route it has until told otherwise. Staleness is the designed-in failure mode of each layer — graceful degradation for reads, catastrophic ambiguity for writes — and every one of the pipeline’s failure stories is a stale-copy story.

Where the Coupling HOLDS and BREAKS

JointCoupling that HOLDSWhere it BREAKS
Resolver ↔ authoritativeTTL bounds resolver staleness; negative caching bounds misses; resolver load scales as ~1/TTL for a fixed query rateTTL is advisory: misbehaving resolvers ignore it; DNS has no revocation, so a too-long TTL cannot be undone — the authoritative server can only wait out the leases it sold
BGP ↔ anycast reachabilityRoute withdrawal removes a PoP from the anycast set without any DNS change; clients fail over by route convergenceConvergence time is unbounded in principle (route flap dampening, slow withdrawal propagation); a client re-homed mid-TCP-flow breaks the connection silently; the routing layer cannot see application health
GeoDNS ↔ edge healthHealth-checked DNS withdraws A records for dead PoPs at the TTL cadenceThe dead PoP’s routes still exist in BGP and its old answers still live in resolver caches — health-checked failover’s floor is check interval + TTL, and keepalive connections pinned to the dead PoP do not even respect that
Origin ↔ edge (read path)Shielding tiers the fetch fan-out; cache-control sets per-object policyLow TTL on hot objects converts every expiry into an origin stampede — the same thundering herd as any expiry-driven cache (see ./virtual-memory-caching.md)
Origin ↔ edge (write path)Purge APIs push invalidation to every edge in secondsPurge is out-of-band: correctness now depends on a control plane, and a bug in the control plane propagates to every edge at purge speed — global in under a minute (the Fastly failure mode, below)

The deepest break: the pipeline’s layers do not share a failure domain or a truth notion. DNS knows names, BGP knows prefixes, the CDN knows objects. A single user-visible failure (“the site is down”) can originate in any layer, and each layer’s own monitoring looks healthy: DNS answers correctly with the old address, BGP routes correctly to the dead site, the edge serves correctly stale content. Only end-to-end checks see the composition.

Failure Propagation Across the Pipeline

Stale DNS after failover. You fail a datacenter over: GeoDNS health checks withdraw its records within the check interval, but every resolver holding the old A record keeps serving it until TTL expiry. Worst-case window ≈ check interval + TTL + propagation jitter; with 60 s checks and a 300 s TTL, up to ~6 minutes of new connections still land on the dead site. The classic mitigation ladder: shorten the TTL in advance of a planned cutover (the pre-warming move — drop TTL to 30–60 s well before the change, since TTL decays only at lease-renewal speed), then fail over, then raise it again. Note the asymmetry that makes this a connection-map question rather than a DNS question: clients that already hold keepalive connections to the old edge never re-resolve at all — HTTP connection reuse silently extends your effective TTL to “as long as the connection lives,” so aggressive keep-alive timeouts on the old site (or a fast connection drain) matter as much as the TTL itself.

Purge lag and the write path. Content corrections travel at purge speed; if the control plane is slow or partial, edges disagree about truth for the purge window. The Fastly June 2021 incident (../failure-casebook/fastly-june-2021.md) is the inverted version: a customer config change that enabled a latent bug propagated to ~80% of the network in about a minute, globally erroring. The lesson generalizes: the CDN control plane is a global, fast, privileged write path, and every mechanism that makes the read path good (edge fan-out, long TTLs, aggressive caching) makes the control plane’s blast radius bigger. Facebook’s October 2021 outage (../failure-casebook/facebook-bgp-2021.md) is the third flavor: BGP withdrawals of the authoritative DNS servers’ own prefix made the name database unreachable, so every layer downstream of DNS was healthy and useless. Three incidents, three different layers, one pipeline.

A Concrete Worked Example

Sizing the failover budget, with numbers. A commerce site serves 120,000 sessions/minute through one GeoDNS zone and a 40-PoP anycast CDN. Steady state: each recursive resolver refetches the zone’s A records roughly once per TTL, so global authoritative QPS ≈ (number of distinct resolvers × query rate per resolver) ≈ 4,000 resolvers × 0.5 qps each = 2,000 qps at TTL 300 — cheap. Planned failover: 15 minutes ahead, drop the TTL to 60 s (authoritative load rises 5× to ~10,000 qps — still trivial), flip traffic, observe the dead site’s request rate: it decays as resolver leases expire, reaching ~0 within check-interval + TTL ≈ 75 s, except for ~2% of traffic on long-lived keepalive connections, which needs a connection drain (server closes idle connections, clients re-resolve) to fall below 0.1%. Now the unplanned version with default TTL 3600: the same drain takes up to ~an hour, and a 60-minute SLO breach is pre-ordained before a single packet is dropped. The arithmetic says: failover speed is a TTL policy you set in peacetime, not a knob you find during the incident.

Shielding arithmetic. Origin normally sees misses from 40 edges; a cold object requested once per edge sees 40 origin fetches. With 3 shields, cold objects are fetched 3× (one per shield) plus 40 edge→shield fetches that never touch the origin — origin load drops by ~13× for cold content and far more for popular content whose shields serve all 40 edges from one copy. The cost: one extra hop of latency on every miss (edge→shield RTT) and a new failure domain (a dead shield redirects its edges — often via the same health-checked DNS machinery that steers users).

When This Connection Is Useful

  • Problem-solving heuristics. When a “users in region X see old content” or “traffic did not fail over” ticket arrives, walk the pipeline joint by joint and ask which staleness budget is being exceeded: resolver TTL (DNS), BGP route state (reachability), CDN TTL or purge propagation (content), or connection reuse (transport). Each joint has a distinct, checkable signature — resolver logs, bgp route dumps, edge cache Age headers, connection lifetimes — and the fix lives at the joint, not in the layer that “looks slow.”
  • Interview answers. “How does a user get to the nearest CDN node?” is a junior question; the senior form is “what bounds how long the system believes something false after a failure?” — answered with TTL as a lease, purge as an out-of-band revocation channel, anycast as routing-level dispatch with BGP convergence as its error term, and connection reuse as the hidden fourth cache. Candidates who say “DNS TTL 60 seconds so failover is instant” have not connected the layers.
  • Engineering decisions. Set TTLs per record class: seconds for endpoints that must fail over fast, minutes-to-hours for stable targets; pre-warm TTLs before planned changes; put health-checked failover in DNS for capacity and in the load balancer for correctness (DNS health checks cannot see per-app health); use stale-while-revalidate to convert expiry stampedes into background refreshes; keep purge latency and coverage measured, because it is the write path of your content system (see ../../sre/cache-patterns.md for the caching-pattern side of these choices).

Interview Questions

  1. Walk a browser request from URL to bytes at a CDN edge, naming at every joint what is cached, what bounds its staleness, and who can revoke it early. Where is the only place the origin can shorten a lease it has already granted?
  2. Your failover runbook claims “DNS TTL is 60 s, so we are back within two minutes,” but 4% of users still hit the dead datacenter 20 minutes after failover. Give three distinct mechanisms that extend the effective TTL and a mitigation for each.
  3. Why does origin shielding reduce origin load by more than the number of edges divided by the number of shields for popular content, but by exactly that factor for cold content? What new failure mode does shielding introduce?
  4. Compare anycast and GeoDNS as “pick a nearby replica” mechanisms: who does the selection, what state each requires, how each fails, and why large CDNs run both instead of either.
  5. The Fastly 2021 and Facebook 2021 outages both broke “the read path” globally in under an hour. Identify at which pipeline joint each failure lived, and explain why each layer’s local monitoring reported healthy.

Key Takeaways

  • The addressing pipeline is DNS (name → any address) → anycast/BGP (address → nearest site) → CDN (site → cached content), and every joint is a cached copy of truth with a bounded staleness budget: TTL lease, BGP convergence, purge propagation.
  • A DNS answer is a lease that cannot be revoked — only expired — so failover speed is set by TTL policy in peacetime; pre-warming TTLs before planned changes is the standard move.
  • Health-checked GeoDNS failover’s floor is check interval + TTL, and long-lived keepalive connections ignore even that floor: connection reuse is a fourth, rarely-budgeted cache.
  • Shielding trades one miss-hop for an order-of-magnitude cut in origin fetch fan-out; purge and config push are the CDN’s write path, fast, global, and able to take the whole network down (Fastly 2021) — read-path mechanisms and write-path blast radius are two sides of one design.
  • Each layer can report healthy while the composition is broken (DNS answers the old name, BGP routes to the dead site, the edge serves stale bytes); only end-to-end synthetic checks cover the pipeline.

Cross-References