Multi-region deployments: decide what replicates, fails over, and can be lost

TL;DR — Multi-region is not one feature; it is three separate decisions: what state you replicate, what traffic you fail over automatically, and what data you are willing to lose or replay. The most common correct design is: stateless app in two regions, primary database in one region with async replica in another, object storage replicated, DNS or load balancer failover for reads, and a documented manual promotion step for writes. Reading time: ~7 min

What it is and where it sits

A multi-region deployment is an architecture where parts of your system run in more than one geographic region, but not all parts behave the same way during failure. The useful question is not "are we multi-region?" but:

• Which components are live in both regions right now?
• Which components have current copies of state?
• Which components can take writes after a regional outage?
• What inconsistency or data loss window do we accept?

In a typical web app, the pieces split cleanly:

• Edge/routing: DNS, CDN, global load balancer. Decides where requests go.
• Stateless compute: app containers, workers, API servers. Easy to run in multiple regions.
• Stateful data: relational DB, Redis, queues, object storage. This is where the real design work is.
• Background integrations: email, webhooks, payment callbacks, cron jobs. Often forgotten in failover plans.

What it replaces: a single-region deployment where all requests, writes, jobs, and storage live in one place. What talks to it: browsers, mobile apps, internal services, batch jobs, third-party callbacks.

A common request/data flow looks like this:

Client
  |
  v
DNS / Global LB
  |-------------------- if region healthy -------------------> Region A nginx/app
  |                                                            |
  |                                                            +--> Postgres primary
  |                                                            +--> Redis/queue
  |
  |-------------------- if region unhealthy -----------------> Region B nginx/app
                                                               |
                                                               +--> Postgres replica or promoted primary
                                                               +--> replicated object storage

The key architecture context: routing failover is fast; state failover is where correctness gets expensive. You can redirect HTTP traffic in seconds. You cannot get zero-loss database failover across continents without paying in latency, throughput, or both.

How it actually works

Walk one realistic example: SaaS app with users logging in, updating profiles, and uploading invoices.

Design target:

• Region A: primary for writes.
• Region B: warm standby for app and read traffic.
• App servers deployed in both regions continuously.
• Postgres primary in A, async streaming replica in B.
• Object storage replicated cross-region.
• Redis used only for cache, not as source of truth.
• DNS failover moves traffic to B if A health checks fail.
• During failover, profile edits may lose up to 30 seconds of writes. Uploaded invoices are durable because object storage replication is confirmed before DB commit or the upload is replayable.

Step-by-step failure path

1. Normal operation

   • Client resolves app.example.com to the global entry point.
   • Health checks prefer Region A.
   • nginx in A proxies to app pods.
   • App writes user profile changes to Postgres primary in A.
   • Postgres streams WAL to B asynchronously.
   • App stores uploaded file in object storage with cross-region replication enabled, then writes metadata row in Postgres.

2. Region A degrades

   • Health endpoint /healthz starts timing out or returning 500.
   • DNS or global LB marks A unhealthy after N failed checks.
   • New traffic starts landing in B after TTL expiry or LB convergence.
   • Existing TCP connections to A die; clients retry.

3. What still works immediately in B

   • Static assets and app servers: yes.
   • Reads from Postgres replica: yes, if app supports read-only mode.
   • Login sessions: only if session state is shared or stateless. If you stored sessions only in Redis A, users get logged out.
   • File downloads: yes, if object replication completed.

4. What does not work until promotion

   • Any write path needing the DB primary.
   • Queue consumers that require exactly-once semantics unless they are fenced.

5. Promotion

   • Ops confirms A is actually unavailable, not just partitioned.
   • Replica in B is promoted to primary.
   • App config in B switches DB endpoint from read-only replica to local primary.
   • Write traffic resumes.

6. Accepted loss

   • Any transactions committed in A but not yet replicated to B are gone unless recovered from A later.
   • If your measured replication lag was 8 seconds before failure, your practical RPO is roughly 8 seconds, not the "near real-time" wording from a slide deck.

The concrete mechanics to watch

For Postgres streaming replication, the signals that matter are not abstract HA terms but actual values:

psql "$DATABASE_URL" -c "select now() - pg_last_xact_replay_timestamp() as replication_delay;"

Typical output shape on replica:

 replication_delay
-------------------
 00:00:03.412451
(1 row)

If replication is broken, you may see:

 replication_delay
-------------------

(1 row)

That NULL usually means the replica has not replayed any transaction yet, or replay is stopped. On primary, pg_last_xact_replay_timestamp() is also NULL, so run this only where you know the role.

For HTTP failover, clients often show the symptom before your monitoring does:

curl -I https://app.example.com/healthz

Misconfigured redirect during failover often looks like:

HTTP/2 301
location: http://app.example.com/healthz
server: nginx

That is bad because it downgrades to http and may loop behind a TLS terminator. The correct shape is usually either 200 from the health endpoint or a redirect preserving https://.

When to use it (and when not to)

Use multi-region when the outage cost is larger than the complexity cost, and when you can state your RTO/RPO in numbers.

You probably do not need this if:

• Your actual requirement is backup/restore, not live failover.
• You have not measured the cost of a 1-hour outage.
• Your team cannot rehearse failover quarterly.
• Your app assumes local mutable state everywhere: sticky sessions, local filesystem, in-memory job state.

Trade-offs

Every benefit here has a bill attached.

• Benefit: lower regional outage risk

  • Cost: double infrastructure for app tier, extra data transfer, replication monitoring, runbooks, on-call complexity.

• Benefit: faster user access from multiple geographies

  • Cost: cross-region cache invalidation, more TLS/DNS/LB moving parts, harder debugging because symptoms vary by resolver and region.

• Benefit: warm standby for disaster recovery

  • Cost: async replication means non-zero RPO. If you want near-zero loss, you pay with synchronous commit latency or a different database model.

• Benefit: active-active app servers

  • Cost: hidden state breaks it: sessions, rate limits, idempotency keys, queue leases, file uploads, webhook dedupe.

• Benefit: object storage replication improves durability

  • Cost: replication lag still exists; delete propagation can turn an accidental delete into a multi-region delete.

• Benefit: automated traffic failover

  • Cost: split-brain risk if you also automate DB promotion without fencing. Two writable primaries is usually worse than downtime.

The experienced-dev question is usually: "Why not active-active everything?" Because most mainstream relational systems still make you choose among write latency, conflict complexity, and correctness. For many products, active-active compute + active-passive database is the sane middle ground.

In practice

Example 1: nginx health endpoint and proxy behavior

server {
    listen 443 ssl http2;
    server_name app.example.com;

    ssl_certificate     /etc/ssl/certs/fullchain.pem;
    ssl_certificate_key /etc/ssl/private/privkey.pem;

    location = /healthz {
        access_log off;
        add_header Content-Type text/plain;
        return 200 'ok';
    }

    location / {
        proxy_pass http://app_upstream;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout 2s;
        proxy_read_timeout 30s;
    }
}

This gives your LB or DNS health checker a deterministic 200 without touching app dependencies. Gotcha: if /healthz checks the database, a DB incident can eject a healthy region from service and amplify an outage; use a separate deep health check for alerting, not necessarily for routing.

Test it directly:

curl -I https://app.example.com/healthz

Expected shape:

HTTP/2 200
content-type: text/plain

Example 2: Postgres replica inspection and promotion

⚠️ Promoting a replica is a one-way role change in normal operations. If the old primary comes back and accepts writes, you can create split brain and permanent divergence. Isolate or shut down the old primary before promotion.

# On the replica in Region B
psql -d postgres -c "select pg_is_in_recovery();"
psql -d postgres -c "select now() - pg_last_xact_replay_timestamp() as replication_delay;"
psql -d postgres -c "select pg_promote(wait_seconds => 60);"
psql -d postgres -c "select pg_is_in_recovery();"

Typical output before promotion:

 pg_is_in_recovery
-------------------
 t
(1 row)

After successful promotion:

 pg_promote
------------
 t
(1 row)

 pg_is_in_recovery
-------------------
 f
(1 row)

This checks role, estimates lag, promotes, then confirms the node is writable. Gotcha: your app may still point at the old primary hostname; promotion alone does nothing unless you also switch connection strings, service discovery, or VIP/LB targets.

Example 3: low-TTL DNS failover record planning

dig +short app.example.com
dig app.example.com | grep -E 'ANSWER SECTION|\sA\s'

Typical output shape:

203.0.113.10

;; ANSWER SECTION:
app.example.com.  30  IN  A  203.0.113.10

A 30-second TTL helps failover converge faster for new lookups. Gotcha: some clients, proxies, and language runtimes cache longer than TTL; Java and some containerized apps are repeat offenders unless explicitly configured.

Further reading

• PostgreSQL Documentation: "Warm Standby and Streaming Replication"
• PostgreSQL Documentation: "High Availability, Load Balancing, and Replication"
• MDN Web Docs: the "HTTP Redirections" and "Caching" chapters
• Google SRE Book: the chapters on "Addressing Cascading Failures" and "Handling Overload"
• RFC 9110: HTTP Semantics

This article was written by an AI system and published pending human review. Verify anything you intend to act on.