REPLICATION: THE ART OF MAKING DATA SURVIVE EVERYTHING

Jul 13
16 min read

From 1970s mainframe tape backups to modern globally distributed databases, a complete guide to why copying data is the hardest easy thing in computing

WHY BOTHER COPYING THE SAME DATA TWICE ANYWAY?

Here's a question worth sitting with: if your database is running fine right now, on your server, producing correct results, why would you want to maintain a second copy of all that data somewhere else?

The answer is almost embarrassingly simple: machines break. Networks get cut. You never trust the networks. Data centers flood. Someone trips over a power cable. A software bug corrupts half your tables at 3 a.m. The world is relentlessly hostile to data at rest.

But it goes beyond survival. There are three genuinely different reasons you might want multiple copies of data and they have almost nothing to do with each other:

High availability. Keep working when some part of the system fails. If one server dies, another can take over without users noticing.
Low latency. Put data physically close to your users. A request from Singapore shouldn't travel to Virginia and back just to read a user profile. Serving from a Singapore replica cuts that trip to microseconds.
Read throughput. Spread the reading work across many machines. If your database gets 100,000 read requests per second, you can split that load across 10 replicas at 10,000 each.

These three motivations each pull in different directions. Keeping data physically close to users means copies are geographically spread out, making them hard to keep perfectly in sync. Maximizing throughput means having many replicas, more nodes to update when a write comes in. The tensions are real and that’s why replication is such a rich topic.

CORE INSIGHT If the data never changed, replication would be trivial. Copy it once, done forever. The entire difficulty of replication comes down to one word: WRITES. Every time data changes on one machine, that change needs to find its way to all the others. How you handle that propagation determines everything about your system's behavior.

A BRIEF, HUMBLING HISTORY OF COPYING

Long before distributed databases existed, organizations ran entire departments dedicated to data redundancy. Banks in the 1960s would run parallel punch-card systems, two separate decks, two separate operators, processing the same transactions independently, then comparing results. Getting them to match was considered an achievement. Getting them to stay matched over time was a daily battle.

1970s: Tape-based replication. IBM mainframes exported transaction logs to magnetic tape. Secondary sites imported them hours later. "Disaster recovery" meant "we can rebuild from last night's backup." Your data might be a day old after a failure.

1980s–90s: Oracle Data Guard & shared-nothing architectures. Commercial databases introduced real-time log shipping. A primary database would stream its redo logs to a standby, which could be promoted in minutes rather than hours. Tandem Computers built entire fault-tolerant systems where every component was doubled.

Early 2000s: The rise of internet scale. Google, Amazon, and Yahoo hit traffic levels no single database could handle. Engineers started asking: what if we had hundreds of copies? The CAP theorem, formalized by Brewer in 2000, framed the fundamental trade-off: you can't have consistency, availability and partition tolerance all at once.

2007 onward: Dynamo, Cassandra and the leaderless era. Amazon's Dynamo paper described a radically different approach: no primary node at all. Any replica could accept writes. Conflicts were resolved after the fact. Cassandra and Riak brought this to the open-source world. Eventually, consistent became not just acceptable but desirable.

2012–now: Globally distributed consistency. Google Spanner proved you could have strong consistency across globally distributed data — if you were willing to pay for atomic clocks and dedicated fiber. CockroachDB and YugabyteDB brought similar ideas to commodity hardware. The frontier is still being pushed.

What's striking about this history is that the hard problems haven't gone away. We've just gotten more sophisticated about naming them and making deliberate trade-offs. Every modern database is still grappling with questions that IBM engineers were arguing about in 1974.

SINGLE-LEADER REPLICATION: THE CLASSIC MODEL

Start simple. One node is the leader (also called primary or master). All writes go there. The leader writes them to its local storage and simultaneously sends the change to its followers (also called replicas, standbys, or secondaries). Reads can go to either the leader or any follower.

This is how PostgreSQL streaming replication works. It's how MySQL binlog replication works. It's how MongoDB replica sets elect a primary. It's how Redis Sentinel manages failover. It's the default mental model for most engineers and for good reason, it's simple and it works.

HOW THE REPLICATION LOG ACTUALLY TRAVELS

Data doesn't teleport. The leader has to package up what changed and send it over the wire. There are four approaches:

① Statement-Based Replication: The leader logs every SQL statement and sends those statements to followers, which execute them. MySQL used this by default before version 5.1.

Problem: Non-deterministic functions, NOW(), RAND(), triggers with side effects, might produce different results on different machines. Call NOW() one millisecond apart and you have a data divergence.

② Write-Ahead Log (WAL) Shipping: PostgreSQL does this. The database writes every change to a WAL file first, a binary append-only log of every byte changed on disk. That log gets streamed to followers, which replay the same byte-level changes. No ambiguity, no non-determinism.

Catch: The log is tightly coupled to the storage engine version. You can't do zero-downtime upgrades by running a new-version follower while the leader is on the old version, the WAL format might be incompatible.

③ Logical (Row-Based) Replication: Instead of raw byte changes, the leader sends a logical description: "Row with id=42 in the orders table had these columns changed to these values." This is version-independent, parseable by external tools and can even replicate to a different database system entirely. MySQL's row-based binlog works this way.

Trade-off: More verbose (a bulk update sends one entry per affected row) but vastly more practical for real-world operations.

④ Trigger-Based Replication Register a database trigger that fires on every change and writes to a separate replication table. An external process reads that table and applies changes elsewhere. Tools like Bucardo use this approach.

Trade-off: Flexible, but slower (triggers add overhead to every write) and it's application-level code — which means it can have bugs.

ADDING A NEW FOLLOWER, WITHOUT DOWNTIME

At first glance, adding a new replica sounds trivial: copy the data over. But think about what "copy the data over" actually means when your database is handling thousands of writes per second. By the time you finish copying, the source has already changed. You're chasing a moving target. Lock the database to stop writes while you copy? Now you have downtime. That's a non-starter.

The elegant solution threads this needle by leaning on something the database already maintains: the replication log. Every change ever made to the database is recorded there, in order, with a position marker. That position marker is your anchor point, the thing that lets you say "I have a complete picture of the data as it existed at exactly this moment."

Here is how it works in practice:

Step 1: Take a consistent snapshot at a known log position: The database takes a point-in-time snapshot of its data without locking writes. In PostgreSQL this is done with pg_basebackup; in MySQL, with mysqldump --single-transaction or xtrabackup. What makes this snapshot special is that it is tagged with a precise position in the replication log, in PostgreSQL this is called the Log Sequence Number (LSN), in MySQL it is the binlog coordinates (file name + offset). This position is the critical link. It says: "this snapshot reflects the state of the database at exactly position X in the log, and not a byte more."

Step 2 : Transfer the snapshot to the new follower: The snapshot is copied to the new follower node. This can take minutes or hours depending on database size, terabyte-scale databases might take many hours to transfer even on fast internal networks. During this entire transfer window, the leader keeps running normally, accepting writes, advancing its replication log. The follower doesn't care. It isn't connected yet.

Step 3: The follower connects and requests the missing writes: Once the snapshot is loaded, the follower connects to the leader's replication stream and says, "give me everything from position X onward." X is that log sequence number from Step 1. The leader streams every write that happened since the snapshot was taken. The follower replays them in order, advancing its own state until it matches the leader's current position. This catch-up phase is called initial sync or recovery. It can itself take minutes or hours if the snapshot was stale by the time transfer finished.

Step 4: The follower enters streaming replication: Once the follower's position matches the leader's (or gets close enough — within normal replication lag), it transitions from catch-up mode into live streaming replication. From this point it receives writes in real time, just like any existing follower. The process is complete.

Zero downtime. Zero locks on the leader. Users never notice.

WHAT CAN GO WRONG

This process sounds clean, but production adds friction.

Snapshot too old, log already rotated. Replication logs don't grow forever, databases periodically rotate and delete old log segments to reclaim disk space. If your snapshot transfer takes so long that the leader has rotated past position X before the follower connects, the follower can't catch up. It needs log entries that no longer exist. The fix is to either configure the leader to retain log segments long enough for slow followers (wal_keep_size in PostgreSQL, expire_logs_days in MySQL), or use a replication slot (more on that below).

Replication slots, guaranteed log retention. PostgreSQL introduced replication slots precisely to solve the log rotation problem. When a follower registers a replication slot on the leader, the leader promises not to rotate away any log segment that the slot hasn't consumed yet. The follower can take as long as it needs, even go offline for hours, and the log will be there when it comes back. The danger: if a follower with a slot disappears permanently and nobody cleans up the slot, the leader will retain logs indefinitely, growing until it fills the disk. Monitor your replication slots. A forgotten slot is a ticking disk-fill bomb.

Large databases and the snapshot transfer window. For a 5 TB database, even a 1 Gbps internal network takes over 11 hours to transfer the snapshot. During those 11 hours, the leader is generating new writes. If the database is write-heavy, the catch-up phase after transfer might itself take hours. Tools like pgBackRest and Percona XtraBackup exist specifically to make this process faster and more reliable, streaming compression, parallelism, incremental snapshots.

Schema changes mid-sync. If a DDL migration (ALTER TABLE, DROP COLUMN) runs on the leader while a follower is mid-catch-up, the follower might try to replay row changes against a table structure that no longer matches what the snapshot established. This is an edge case in most setups but a real hazard during schema migrations. The safest approach is to avoid large schema changes while actively adding new followers.

HOW POSTGRESQL AND MYSQL ACTUALLY DO IT

PostgreSQL uses pg_basebackup to take the initial snapshot. It connects to the primary, opens a replication connection, and streams a byte-for-byte copy of the data directory, tagged with the LSN at the moment the backup started. The resulting snapshot can be restored on a new node, which then connects to the primary and replays WAL from that LSN forward using recovery.conf (or postgresql.auto.conf in newer versions). The whole process is well-documented and largely automated by tools like Patroni and repmgr in production environments.

MySQL uses either mysqldump --single-transaction (for logical snapshots) or xtrabackup (for physical snapshots). After restoring the snapshot on the new replica, you run CHANGE REPLICATION SOURCE TO (formerly CHANGE MASTER TO) pointing at the primary's hostname and binlog coordinates from the snapshot. The replica then connects and replays binlog events from that position. MySQL 8.0+ supports GTID-based replication (Global Transaction Identifiers), which makes this even cleaner, instead of tracking a file name and byte offset, you track a globally unique transaction ID, which makes failover and re-pointing replicas significantly less error-prone.

THE BIGGER PICTURE

What makes this process powerful is that it is completely non-disruptive by design. The leader doesn't know or care that a new follower is being provisioned. It just keeps appending to its log as it always does. The follower bootstraps itself by using that log as a shared source of truth. This same mechanism, snapshot at a known log position, replay forward from there, is also how you recover a crashed follower, how you rebuild a corrupted replica, and how you migrate data to a new server. Understanding it deeply means you understand a large chunk of how database operations actually work in production.

REPLICATION LAG — THE BUGS YOU DON'T SEE COMING

Here's where things get genuinely weird, and where most distributed systems bugs actually hide.

Asynchronous replication means the leader confirms a write to the client before the followers have actually applied it. The write is in the leader's log, it's being sent, but at the exact moment of confirmation, followers are maybe 50 milliseconds behind. Under load, or with a slow network, they might be seconds or minutes behind. This gap is called replication lag.

PROBLEM 1: READING YOUR OWN WRITES

Imagine you post a comment on a forum. The write goes to the leader. You immediately refresh the page and your comment is gone. The read went to a follower that hadn't received your write yet.

This is called read-your-own-writes consistency. Fixes:

Always read from the leader when accessing data the user might have just modified (e.g. always serve a user's own profile from the leader).
Or track the latest write's log timestamp in the user's session, and route reads to a sufficiently caught-up replica until that timestamp is reflected.

PROBLEM 2: MONOTONIC READS

You refresh a live chat stream twice. First refresh: 20 messages. Second refresh: 18 messages. You just went back in time. This happens when two sequential reads hit followers with different amounts of lag.

Monotonic Reads Invariant If read(T) = v, then for any later T' > T, read(T') ≥ v. Once you have seen a value, you will never see an older one.

Fix: Route each user's reads to the same replica consistently, using a hash on the user ID.

PROBLEM 3: CONSISTENT PREFIX READS

Suppose you're reading a conversation and the answer arrives before the question, because they were written to different shards with different lags. The conversation is nonsense.

Consistent prefix reads guarantee that causally related writes are seen in the order they were written.

Anomaly	What you see	Fix
Read-your-own-writes	Your writes disappear and reappear	Read from leader for your own data
Monotonic reads	Time goes backward	Sticky routing per user
Consistent prefix reads	Answers before questions	Same shard for causally related writes

SYNCHRONOUS VS. ASYNCHRONOUS: THE FUNDAMENTAL BET

Every replication system forces you to take a position on one of the deepest trade-offs in distributed computing: how much durability are you willing to pay for?

Synchronous replication The leader waits for at least one follower to confirm before telling the client "done."

Guaranteed no data loss if the leader dies.
You're only as fast as your slowest synchronous follower. If that follower is unreachable, every write blocks.

Asynchronous replication Fast, non-blocking.

No waiting, high throughput.
If the leader dies before the followers receive the write, that write is gone. Not buffered, not recoverable.

Semi-synchronous (the practical middle ground) One synchronous follower, the rest asynchronous. You always have at least one up-to-date copy for instant promotion, and you're not blocked by all followers being healthy. MySQL calls this "semi-synchronous replication." It's pragmatic and widely deployed.

Fully synchronous replication across all followers is dangerous because a single node failure blocks all writes. This is why you seldom want it.

MULTI-LEADER REPLICATION: WHAT IF EVERY DATACENTER COULD WRITE?

Single-leader has a critical weakness: every write must go through one specific node. If your users are in Tokyo, São Paulo, and Berlin, and your leader is in Virginia, every write takes a round trip to Virginia and back, 150+ milliseconds of avoidable latency.

The solution: let each datacenter have its own leader. Writes in Tokyo go to the Tokyo leader. Each leader replicates to the others asynchronously. The catch comes when two datacenters modify the same data at the same time.

WRITE CONFLICTS: THE DARK HEART OF MULTI-LEADER

Two users in different data centers both edit the title of the same Wikipedia article at the same time. Both writes succeed locally. They replicate to each other. Now what? The database can't know which version is "correct" - that's a business logic question.

Strategy	How it works	Trade-off
Last Write Wins (LWW)	Higher timestamp wins; the other write is discarded	Lossy, valid data silently deleted. Cassandra's default.
Replica ID priority	Higher-numbered replica always wins	Lossy, arbitrary, no relationship to intent
Merge values	Concatenate both results	Works for sets/counters; produces nonsense for others
Preserve all, resolve on read	Keep all versions; resolve in application code	No data loss. Amazon's shopping cart used this.
CRDTs	Data structures that merge mathematically	Automatic, loss-free. Only works for CRDT-compatible types.

Operational tip: The best way to handle multi-leader conflicts is to avoid them. Route all writes for a particular record to the same datacenter, use the user's home region as the routing key. Most applications can live with this and never deal with conflicts at all.

MULTI-LEADER TOPOLOGIES

Circular: each node replicates to the next. One broken node breaks the entire ring.
Star (central hub): all nodes replicate through one central node. Hub failure = total outage.
All-to-all: every node replicates to every other node. Most resilient, but writes can arrive out of order, requiring version vectors to track causal dependencies.

LEADERLESS REPLICATION: THROWING OUT THE RULEBOOK

In the late 2000s, Amazon's engineers needed a shopping cart service that could accept writes even when parts of the infrastructure were failing. A single-leader system couldn't give them that. Their answer: get rid of the leader entirely.

In leaderless replication, every node is a peer. Clients send writes to multiple replicas in parallel. Clients send reads to multiple replicas in parallel and pick the most recent value. There's no primary or master. Dynamo pioneered this. Cassandra, Riak, and Voldemort followed.

CATCHING UP: READ REPAIR AND ANTI-ENTROPY

Read repair: When a client reads from multiple nodes and gets back different versions, it writes the newer value back to the stale node. Lazy, on-demand healing. Works great for frequently read data; rarely read data can stay stale for a long time.

Anti-entropy: A background process continuously compares data across replicas and copies missing writes. It uses a Merkle tree, the same data structure used in certificate transparency and cryptocurrency, to efficiently find diverged sections without comparing every single record.

QUORUMS: THE MATH THAT MAKES THIS SAFE

How can you trust a system where anyone can write and there's no master? The answer is a beautiful piece of mathematics.

Say you have N total replica nodes. You require:

W nodes to acknowledge a write before it's considered successful
R nodes to respond to a read, taking the most recent value

THE QUORUM RULEW + R > NIf the write quorum (W) plus the read quorum (R) exceed the total replicas (N), the two sets must overlap by at least one node. That overlapping node holds the latest write, so every read is guaranteed to see it.

Common configurations with N=3:

Config	W	R	Good for	Trade-off
Strong consistency	2	2	Correct reads after writes	Can't tolerate more than 1 failure
Write-optimized	1	3	High-velocity writes	Any single failure risks data loss
Read-optimized	3	1	Read-heavy, caching	Writes blocked if any node is down

SLOPPY QUORUMS: AVAILABILITY OVER CORRECTNESS

What if a network partition means you can't reach enough of your designated N nodes? Some systems offer a sloppy quorum: write to any W reachable nodes, even outside the "home" set for this data. When the partition heals, do a hinted handoff, transfer those writes to the correct nodes. Dynamo does this. Riak does this.

The limits of quorums: Quorums don't protect you from everything. Two clients writing to the same key concurrently can both get quorum and produce a conflict. A write that partially succeeds leaves the cluster in an ambiguous state. Quorums give you probabilistic consistency in the common case, not ironclad guarantees.

THE CONSISTENCY GUARANTEE LADDER

"Consistency" means different things in different contexts. Here's the hierarchy, from strongest to weakest:

LINEARIZABILITY: THE GOLD STANDARD

Linearizability means the system behaves as if there is only one copy of the data. Every read gets the most recent write, globally. No stale reads, ever.

The cost is brutal:

You need a consensus protocol like Raft or Paxos.
Every operation requires multiple round trips for global ordering agreement.
The system will reject operations during network partitions rather than serve stale data.

Google Spanner achieves this globally using TrueTime: GPS + atomic clocks. Nobody else has atomic clocks in their data centers.

CAUSAL CONSISTENCY: THE SWEET SPOT

Causal consistency is weaker than linearizability but dramatically cheaper. The rule: if operation A caused operation B (A happened before B), then everyone sees A before B. Operations with no causal relationship can be seen in any order.

To implement this, each operation carries a version vector, a compact record of which operations it causally depends on. You'll never see an answer before the question that caused it, because that dependency is explicitly tracked. This provides most of what applications actually need, at a fraction of the cost of linearizability.

THE CAP THEOREM AND WHY IT'S SLIGHTLY MISUNDERSTOOD

CAP THEOREM Consistency + Availability + Partition tolerance → pick any 2. Network partitions aren't optional, they happen. So in practice, you're really choosing between CP and AP whenever the network splits.

CP - When the network splits, reject requests that can't reach a quorum. Your system goes partially unavailable, but no stale data is served. (HBase, etcd, Zookeeper)
AP - When the network splits, serve potentially stale data. Your system stays up, but some reads might return old values. (Cassandra, CouchDB, Riak)

Neither choice is wrong. It depends entirely on what your application tolerates: a user seeing a product at the wrong price for 2 seconds, or an error page?

PUTTING IT ALL TOGETHER: HOW TO CHOOSE

Replication model	Writes go to	Conflict possible?	Consistency	Best for
Single-leader	One node only	No	Strong or eventual	Most OLTP, RDBMS workloads
Multi-leader	Any datacenter leader	Yes	Eventual (conflict resolution required)	Multi-datacenter writes, offline-first apps
Leaderless	Any N nodes (quorum)	Sometimes	Tunable via W+R>N	High availability, write-heavy, IoT, analytics

FIVE THINGS THAT WILL BITE YOU IN PRODUCTION

Forgetting about replication lag. Your tests run against a single-node database where everything is consistent. In production, follower lag is real. Build your application to tolerate stale reads from day one, or pin critical reads to the leader explicitly.
LWW in Cassandra losing data silently. Last Write Wins sounds reasonable until you realize it discards valid concurrent writes. If you have concurrent updates, some are being silently dropped. Know your conflict resolution policy.
Not monitoring replication lag. You should have an alert if follower lag exceeds your SLA. Rising lag is a canary, it often precedes a leader failure. Most databases expose this as a metric; watch it.
Assuming quorum = consistency. Quorums give you statistical guarantees, not hard ones. They don't protect against clock skew, partial writes, or concurrent writes to the same key. If you need true linearizability, you need Paxos or Raft.
Underestimating the complexity of multi-leader. Many engineering teams add multi-leader for low write latency across regions, then spend months fighting conflict resolution bugs. Start with single-leader and geo-routing; graduate to multi-leader only when you've exhausted simpler options.

WHERE THIS IS ALL GOING

The frontier in replication is making the hard trade-offs disappear.

Google Spanner showed that with enough hardware, atomic clocks, dedicated fiber, global infrastructure, you can get linearizable consistency at global scale. Calvin, CockroachDB, and YugabyteDB are trying to do the same on commodity hardware by getting very clever about transaction ordering.

CRDTs (Conflict-free Replicated Data Types) are making multi-leader replication safer by designing away conflicts at the data structure level. If your data type can only be merged, not conflicted, the whole problem goes away. Redis, Riak, and several academic systems are pushing this frontier.

And the operational side is getting better too. Automatic failover that once required expensive external tooling is now built into most databases. Replication topology changes that required downtime can happen live. What required a specialist in 2005 is table stakes in 2025.

But the fundamental physics hasn't changed. Light travels at the speed of light. Networks partition. Clocks drift. Data on two machines will sometimes disagree. Replication is the art of making those facts of the universe as invisible as possible to the people depending on your system.

The more you understand the mechanisms, why a follower can lag, why W + R > N matters, why a multi-leader system needs conflict resolution, the better your instincts for where your system's invisible weaknesses are hiding. And the less surprised you'll be at 3 a.m. when something unexpected surfaces.