Multi-Region Failover with Anycast and DNS

Multi-region failover fails in practice for a reason that has nothing to do with the gateway: the traffic-steering layer is slower, or dumber, than the incident. DNS caches ignore the record you just changed; anycast withdraws a route in seconds but cannot tell that a region is serving 500s. This page covers what each mechanism can actually detect and how fast, and how to combine them so a regional failure is a minute of degradation rather than an hour.

Prerequisite concepts

This assumes the single-region topologies in high availability topologies, and the state placement decisions in active-active vs active-passive gateway failover, because a regional failover moves the same state question up a level.

What each layer can see, and how fast

Three steering mechanisms, three different things they can detect Anycast withdrawal reacts in seconds but only to a node that has stopped announcing, not to one that is answering incorrectly. Health-checked DNS can detect an application-level failure but is bounded below by record time-to-live and by resolvers that ignore it. Client-side retry across endpoints is the fastest and only works for clients you control. mechanism can detect time to shift anycast withdrawal BGP route pulled a node that stopped announcing seconds health-checked DNS record swapped on failure an application returning errors TTL, plus resolvers that ignore it client-side retry next endpoint in the list any failure the client sees one request The gap in row one is the important one: anycast cannot tell a healthy region from one serving 500s at full speed.

The combination that works is anycast for the fast, coarse case — an edge that has gone away entirely — plus health-checked DNS for the slow, precise case, plus a withdrawal hook so the application can pull itself out of anycast when it knows it is unhealthy.

# The health check that matters is the one the application controls.
# When it fails, stop announcing rather than waiting for DNS.
#   /etc/bird/bird.conf  (BIRD 2.x)
protocol static {
    ipv4;
    route 198.51.100.0/24 via "lo" {
        # withdrawn automatically when the health file disappears
    };
}
# Health endpoint that reflects real dependencies, not process liveness
# Envoy 1.32+ — fail the check when the upstream cluster has no healthy hosts
health_check:
  - name: envoy.health_checkers.upstream
    path: /healthz
    cluster_name: orders_cluster
    # a 200 here must mean "this region can serve", not "this process is running"

A health endpoint that returns 200 while every upstream is failing is the single most common reason a regional failover does not happen.

Where the state goes

The failover is only as fast as the state it needs Region-local state with no sharing fails over instantly and loses quota accuracy. A single primary in one region means a failover of that region is a write outage everywhere. Multi-primary replication survives a regional loss and introduces conflict resolution, which has to be designed rather than discovered. region-local only failover: instant quota: per region a consumer gets N times their limit across N regions single primary failover: fine, unless it is the primary region quota: exact cross-region latency on writes multi-primary failover: survives a region quota: eventually consistent conflict resolution is now your design problem Most platforms end at the left box for rate-limit counters and the middle box for anything billable, which is a deliberate split rather than an inconsistency: shaping traffic tolerates drift, invoicing does not. Whichever you choose, state the intended behaviour during a partition before the partition happens. Failover is a capacity plan, not a routing plan Three regions each running at seventy percent of capacity look comfortable. When one is lost its share is split between the survivors, taking each to a hundred and five percent — past their ceiling. Planning for a regional failure means running each region low enough to absorb the largest peer it might have to carry. steady state — three regions at 70% eu-west us-east ap-south after losing ap-south — survivors at 105% eu-west us-east both survivors are now past their ceiling Three regions need to sit near 50% each to absorb one loss cleanly; two regions need 50% each to absorb the other. That is the real cost of the topology, and it is a bill paid every day rather than only during an incident.

Decision matrix

Requirement Mechanism
Survive a whole region going dark anycast withdrawal
Survive a region serving errors health-checked DNS, or application-driven withdrawal
Sub-second failover for your own clients client-side endpoint list with retry
Regulatory data residency per-region hostnames, no automatic cross-region failover
Exact global quotas single primary, accepting the write-latency cost
Fast local quotas per-region counters, accepting drift

Gotchas and failure signals

A TTL you cannot enforce. Many resolvers clamp low TTLs upward, so a 30-second record may behave like a 300-second one. Measure real propagation with distributed probes before assuming your failover time.

Failover into a region without capacity turns one regional outage into two. Every region must be able to carry its share plus the largest failover it might absorb, and that headroom needs testing rather than arithmetic.

Health checks that only test the gateway pass while every upstream is failing. Make the check depend on a real downstream call.

Failing back is riskier than failing over because the recovered region starts cold: empty caches, cold connection pools, unwarmed JITs. Ramp back gradually rather than switching in one step.

The arithmetic is worth doing before the design is committed, because it changes what multi-region actually costs. Two regions that must each survive the loss of the other cannot exceed fifty percent utilisation in normal operation. Three regions can each run at around sixty-five percent and still absorb one failure. The pattern generalises: N regions can run at roughly (N-1)/N of capacity, so the marginal cost of resilience falls as regions are added, which is one of the few genuine arguments for a third region rather than a second.

Validation

  • Health endpoint returns non-200 when upstreams cannot serve, verified by fault injection
  • Anycast withdrawal exercised in a game day, with the time to shift measured
  • Real DNS propagation measured from several networks, not assumed from the TTL
  • Each region tested at the load it would carry after losing the largest peer
  • Fail-back procedure ramps traffic rather than switching it
  • Quota behaviour during a regional partition documented and agreed
  • Someone named owns the steering configuration, and the runbook says explicitly who may withdraw a region and on what evidence, because an unowned failover is one that nobody triggers during the incident it was built for

FAQ

Why is anycast not enough on its own?

Because it reacts to a node that has stopped announcing its route, not to one that is answering every request with a 500. Anycast withdrawal is fast and coarse. Pair it with a health check that reflects real downstream capability and can withdraw the announcement itself, so an application-level failure produces the same fast shift as a dead machine.

How fast is DNS failover really?

Slower than the record time-to-live suggests. Many resolvers clamp low values upward and some ignore them entirely, so a thirty-second record can behave like a five-minute one for a meaningful share of clients. Measure propagation from several networks with distributed probes rather than assuming the configured value, and plan the failover budget around what you measured.

What breaks first when a region fails over?

Capacity, usually. If each of three regions runs at seventy percent, losing one asks the survivors to carry a hundred and five percent of their own capacity. Every region has to be able to carry its share plus the largest failover it might absorb, and that headroom needs to be exercised in a game day rather than calculated on a slide.

Should rate-limit counters be shared across regions?

Usually not for shaping and usually yes for anything billable. Per-region counters let a consumer exceed the intended global limit by the number of regions, which is acceptable when the limit exists to protect a backend and unacceptable when it defines what a customer is charged for. Whichever you pick, decide what happens during a partition before the partition happens.

Parent: High Availability Topologies