Distributed Systems Overview

Overview

A distributed system is a collection of independent computers that appears to its users as a single coherent system. These computers communicate and coordinate their actions by passing messages over a network. Distributed systems enable scalability, fault tolerance, and geographic distribution, but introduce fundamental challenges around consistency, coordination, and failure handling.

Why Distributed Systems?

graph TB
    REASONS[Why Distribute?] --> SCALE[Scalability<br/>Handle more load]
    REASONS --> FAULT[Fault Tolerance<br/>No single point of failure]
    REASONS --> LATENCY[Low Latency<br/>Servers closer to users]
    REASONS --> AVAIL[Availability<br/>24/7 service]

Core Definitions

Four terms carry specific technical meanings in interviews; using them loosely (“we scale by adding servers”) reads as junior. Anchor each to a measurement and a design decision.

The subtle point interviewers probe: partition tolerance is not optional. Scalability and availability are goals you trade off; partitions are a physical fact, which is why CAP really says “when P happens, choose C or A” (see the diagram below).

Fundamental Challenges

graph TB
    CHALLENGES[Distributed System Challenges] --> TIME[Time & Ordering<br/>No global clock]
    CHALLENGES --> CONSENSUS[Agreement<br/>Getting nodes to agree]
    CHALLENGES --> FAILURE[Failure Detection<br/>Is it slow or dead?]
    CHALLENGES --> CONSISTENCY[Consistency<br/>Keeping data in sync]
    CHALLENGES --> PARTITION[Network Partitions<br/>Messages can be lost/delayed]

The Eight Fallacies of Distributed Computing

Peter Deutsch’s fallacies (1994):

1. The network is reliable — It isn’t. Messages get lost, connections drop.
2. Latency is zero — It isn’t. Cross-datacenter communication takes milliseconds.
3. Bandwidth is infinite — It isn’t. Network congestion is real.
4. The network is secure — It isn’t. Every communication can be intercepted.
5. Topology doesn’t change — It does. Nodes join and leave constantly.
6. There is one administrator — There isn’t. Multiple teams manage different parts.
7. Transport cost is zero — It isn’t. Serialization, encryption, and routing cost CPU and time.
8. The network is homogeneous — It isn’t. Different hardware, protocols, and configurations.

Each fallacy has a canonical production countermeasure; naming the mitigation (not just “it’s wrong”) is what makes the list an interview weapon rather than trivia.

The Consistency Models Ladder

CAP’s “C” is the top rung of a ladder; most interviews ask you to place a workload on it. Know what each rung forbids, which system defaults there, and the mechanism that implements it.

The ladder pays rent in design interviews: a shopping cart wants read-your-writes for the buyer and eventual consistency for everyone else; a lock service wants linearizability; a social feed usually accepts causal. Answering “which rung, which mechanism, why is the weaker rung cheaper?” is the senior signal.

Failure-Mode Dictionary

Vocabulary precision matters as much as the algorithms — these are the failure classes the theory assumes, in increasing severity:

Note that real clouds are crash-plus-omission systems almost all the time and byzantine almost never — which is why Raft suffices for most infrastructure and BFT remains a niche (blockchains, safety-critical federations). That scoping argument is itself a favorite senior question.

CAP Theorem in One Diagram

CAP (Gilbert & Lynch, 2002): a distributed data store can guarantee at most two of Consistency, Availability, and Partition tolerance — and since partitions are unavoidable in any real network, the actual choice is C vs A during a partition.

graph TD
    CAP["CAP theorem - pick 2"] --> C["Consistency"]
    CAP --> A["Availability"]
    CAP --> P["Partition tolerance"]
    C -->|"CP: linearizable reads, refuse requests during P"| P
    A -->|"AP: always answer, possibly stale data"| P
    C -.->|"CA: only valid when no partition can happen"| A

The one-line system placements to memorize: CP — ZooKeeper, etcd, Spanner, HBase; AP — Cassandra, DynamoDB, Riak, Eureka. PACELC extends the story: even without partitions you choose between latency and consistency, which is why “eventual consistency with low latency” (AP/EL) and “sync replication with higher latency” (CP/EC) are both legitimate designs. Interviews frequently ask where the nuance is: real systems tune per-operation (Dynamo-style tunable quorums; Spanner offers read-only transactions without TrueTime waits), and CAP’s C means linearizability, not the looser “consistency models” most apps need.

Topics in This Section

Reading Map

This section is large; navigate by layer. Start with fundamentals/ for the theory every interviewer assumes, advanced/ for the senior-level mechanisms, systems/ for concrete architectures to cite as evidence, and testing/ for the “how do you know it works” questions that close senior loops.

A high-yield 3-day pass: day 1 — cap, consistency, time/vector-clocks, raft; day 2 — quorums + replication + partitioning; day 3 — one systems deep-dive (Spanner or etcd) plus jepsen for the verification story. That arc covers ~80% of distributed-systems interview questions end to end.

Real-World Distributed Systems

Interview Focus

• Explain the CAP theorem and its real-world implications
• Describe the difference between consistency models
• Explain why distributed consensus is hard
• Describe how vector clocks capture causality
• Give examples of distributed systems you use daily

Interview Questions

1. Why is CAP “choose C or A” rather than “choose two of three”? Partition tolerance is not a design choice — links will fail — so the theorem effectively governs behavior when a partition occurs. A CP system (etcd, ZooKeeper) refuses writes/reads that would break linearizability until the partition heals, sacrificing availability; an AP system (Cassandra, DynamoDB) keeps answering but may serve divergent data that needs reconciliation (CRDTs, last-write-wins, read-repair). Systems also relax per-operation: tunable quorums let you pick per request, and PACELC adds that even partition-free you trade latency vs consistency.

2. A product manager asks for “100% availability.” What do you actually promise, and how? You promise a measured SLO (e.g., 99.95% of requests succeed within 300 ms over a rolling month) and design degradation paths, because 100% is unreachable under partitions and crashes. Mechanisms: redundant replicas behind health-checked load balancers, retry budgets with idempotency keys, circuit breakers that fail fast to fallbacks, and read-your-writes sessions so degraded consistency is explicit. The interview point is converting an availability number into the replication/consistency budget that pays for it.

3. Why can’t you just detect that a node is dead and remove it? In an asynchronous network, “slow” and “dead” are indistinguishable beyond an arbitrary timeout — this is the FLP result and the reason failure suspicion is probabilistic (phi-accrual detectors, SWIM) rather than certain. Acting on suspicion unilaterally causes split-brain: two primaries both serving, which is why removal decisions go through quorum-based leases or fencing tokens. Strong answers cite ZooKeeper sessions, Kubernetes leader election, and fencing tokens as the standard defenses.

4. What does a vector clock give you that a Lamport clock doesn’t? A Lamport clock establishes only a one-way “happens-before or concurrent” ordering — if clock(a) < clock(b), a might still be concurrent with b. Vector clocks carry one counter per process, so comparing vectors tells you exactly which events are causally ordered and which are genuinely concurrent — the property Dynamo-style stores use to detect conflicting writes as concurrent versions. The costs are size O(n) per node and the merge question: you detect the conflict, but you still need an application policy (CRDT merge, sibling resolution) to resolve it.

5. Name a real system and which CAP position it took — with evidence. Spanner is CP-flavored: it offers external consistency via TrueTime commit waits and Paxos groups, and during a partition a replica group simply cannot commit without its quorum. Cassandra is AP: any node can accept writes with consistent hashing + tunable quorums, and conflicts surface as read-repair or timestamped resolution. The strongest answers connect the choice to the workload — Spanner for financial ledgers, Cassandra for always-write-available telemetry — rather than treating the labels as dogma.

6. Why does two-phase commit block when the coordinator dies, and what do real systems do instead? In 2PC, after a participant votes “yes” it is pinned: it can neither abort nor commit until the coordinator announces the decision, so a coordinator crash freezes every participant holding locks — the classic blocking window. Consensus-based commit (Raft/Paxos over the decision) replaces the single coordinator with a replicated one, so a crash costs an election, not a deadlock; Spanner, CockroachDB, and TiDB all commit through consensus per shard. If the follow-up is “why keep 2PC at all”: it’s the lowest-latency option when the coordinator is assumed reliable, and it’s the model XA and many queue-transaction integrations standardized on.

Key Takeaways

• A distributed system = independent machines + message passing + the appearance of one system; every hard problem downstream (consensus, consistency, failure detection) follows from that definition.
• Partitions are a fact, not a choice — CAP’s real decision is C-vs-A during a partition, and PACELC adds the latency-vs-consistency choice even without one.
• The eight fallacies are an engineering checklist: for each, know the failure mode and the standard mitigation (retries+idempotency, caching/batching, mTLS, discovery, schema-registry).
• Availability is a number you design for — \( A = \frac{MTBF}{MTBF+MTTR} \) — and you buy it with redundancy, graceful degradation, and tested failure paths, not slogans.
• Failure detection is always probabilistic; leases, fencing tokens, and quorum decisions are what keep suspicion from causing split-brain.
• Clocks: Lamport orders, vectors detect concurrency, HLCs bridge both for real databases — pick the tool by the anomaly you must exclude.
• Read the section in layers: fundamentals → consensus/replication/partitioning → one systems deep-dive → Jepsen-style verification; that arc answers most interview loops.

Cross References

• CAP Theorem
• Consistency Models
• Consensus
• Replication
• Cloud Overview
• Fundamentals hub — clocks, failure detectors, FLP, gossip: the theory layer under this page
• Advanced hub — CRDTs, HLCs, leases, quorum systems: the senior-level mechanism layer
• Systems hub — Spanner/etcd/ZooKeeper/CockroachDB internals to cite as evidence
• Testing hub — Jepsen and deterministic simulation: how these guarantees are actually verified
• Raft — the consensus protocol every backend interview expects
• Consistent Hashing — the scalability mechanism behind “just add nodes”