Messaging Internals: Which Message System When

Overview

The messaging overview pages in this book explain what Kafka, Pulsar, RabbitMQ, and NATS are; this folder explains how they work inside — log segments and indexes, group coordination wire protocols, BookKeeper quorums, thread-per-core schedulers — and which system to pick given a concrete constraint set. Internals matter in interviews because every senior-level question (“why is Kafka fast?”, “why did you pick Pulsar over Kafka?”, “what breaks during rebalancing?”) is answered at the mechanism level, not the feature level. Start here for the decision map, then descend into the page that matches the question.

Which Message System When

Pick by dominant constraint, not by feature checklist. Throughput numbers below are order-of-magnitude figures for commodity hardware from vendor-neutral benchmarks (OpenMessagingBenchmark-style 3-node clusters); real deployments vary by payload size, durability settings, and client choice.

Two systems are deliberately absent from this table: BookKeeper is not a message system at all (it is the replicated write-ahead-log storage layer under Pulsar — see BookKeeper internals), and Redis Streams is covered in the backend Redis page.

Component Map

Every system in this folder splits into the same four layers; implementations differ in where they place the durable storage and who computes assignments.

flowchart TD
    CLIENT["Clients<br/>producers / consumers / streams"]
    API["Protocol + API layer<br/>wire protocol, batching, compression"]
    COORD["Coordination layer<br/>metadata, membership, assignment"]
    STORE["Durable storage layer<br/>replicated log / ledger"]
    CLIENT --> API
    API --> STORE
    API --> COORD
    COORD --> STORE

The coupling row is the load-bearing one. In Kafka a partition-replica lives in a broker’s local filesystem, so a broker failure forces leader election plus (for permanent failures) data re-replication across the network. In Pulsar a broker holds only an ownership lease on a namespace bundle; a failed broker’s topics are re-owned by survivors in seconds because the bytes were already committed to BookKeeper’s quorums. That trade buys elasticity at the price of one more always-on component — the operational question to ask in any “Kafka vs Pulsar” interview answer.

Internals Index

Read the page that matches the interview question being asked:

Cross-Cutting Dimensions

When comparing two systems in an interview, score them on five dimensions in this order. Most candidate answers fail by comparing features (“Pulsar has key_shared”) instead of mechanisms.

1. Where the log lives. Broker-local file (Kafka, Redpanda) vs third-quorum storage (Pulsar/BookKeeper) vs Raft-replicated per stream (JetStream). This single decision determines failure behavior, elasticity, and tail latency.
2. Who computes assignment. Client-side protocols leak complexity into every client library and make rebalances slow (Kafka ≤3.x); server-side assignment (KIP-848, Pulsar, NATS) makes the broker the single source of truth.
3. Delivery-semantics machinery. At-least-once is the default everywhere; exactly-once is always a mechanism — Kafka transactions, NATS dedup windows, Pulsar transactions — never a checkbox. See exactly-once semantics.
4. Flow-control model. Pull-based (Kafka fetch, JetStream pull consumers) naturally backpressures; push-based (RabbitMQ, JetStream push) needs explicit credit/prefetch controls. See backpressure.
5. Operational topology. Number of always-running components (Kafka: 1, Pulsar: 2-3, JetStream: 1) and blast radius of a node failure (data movement vs ownership transfer).

Rapid Comparison Cheat Sheet

Order-of-magnitude defaults for the four systems this folder dissects. Numbers assume default configs on 3-node clusters with local NVMe — enough precision for interview answers, not for capacity planning.

Three rows deserve attention in any comparison answer. Durability defaults are not comparable directly: Kafka’s acks=all counts replica acks without fsync, while BookKeeper’s Qa counts journal fsyncs on remote bookies — Pulsar’s ack is slower per byte but gives a stronger per-bookie durability floor. Ordering scope differs: NATS’s wildcard subject subscriptions have no Kafka analog, and Pulsar’s key_shared gives per-key ordering without partition pinning. The license row is a real procurement constraint for Redpanda, unlike the Apache 2.0 systems.

Common Interview Threads

The folder is organized so each recurring interview thread has one anchor page. Follow the thread rather than reading linearly:

1. “Why is Kafka so fast?” → log internals: sequential appends, page cache, sparse indexes, zero-copy sendfile, batching. The follow-up “when does it get slow?” lands on TLS (no sendfile), down-conversion, and small-message churn.
2. “What happens when a consumer dies?” → rebalancing: coordinator detects via session timeout, generation ID bumps, eager revoke-all, and how cooperative/KIP-848 change the answer.
3. “Kafka vs Pulsar” → pulsar internals + BookKeeper internals: storage coupling vs decoupling, E/Qw/Qa vs ISR, bundle lease vs replica move.
4. “Could you make Kafka faster / what would you redesign?” → Redpanda and thread-per-core: Seastar shards, Raft-native replication, GC absence — with the benchmark caveats attached.
5. “How do you get exactly-once?” → exactly-once semantics: idempotent producer mechanics, transactional two-phase commit, LSO, and the boundary at the broker’s edge.
6. “Our consumers keep falling behind / the queue keeps growing” → backpressure and flow control: quotas, fetch windows, pause/resume, and the shedding hierarchy.

How This Folder Fits the Rest of the Book

The overview pages stay authoritative for basics: Kafka overview, Pulsar overview, queues, and the backend-level Kafka and NATS pages. Ecosystem tooling lives in Kafka Connect and Kafka Streams; the decision landscape across systems is in messaging vs streaming. When an internals answer touches consensus, link back to Raft and ZooKeeper rather than re-explaining the protocol.

Interview Questions

1. When would you choose Pulsar over Kafka, and what do you pay for it? Choose Pulsar when the dominant requirements are multi-tenancy (thousands of small topics with per-tenant quotas), independent compute/storage scaling, or built-in geo-replication. Brokers are stateless: they own namespace bundles via leases, and a broker failure is a pointer re-assignment, not a data move. You pay with an always-on storage tier (BookKeeper bookies) plus a metadata store, so the minimum production footprint is larger and the number of failure modes grows. Kafka couples compute and storage but runs as one system, which is why small and mid-size deployments usually standardize on it.

2. Why does Redpanda claim lower tail latency than Kafka, and what is the honest caveat? Redpanda is C++ on Seastar’s thread-per-core model: each core owns a subset of partitions with no cross-core locks, so no request queues behind another core’s work, and there is no JVM garbage collector to produce stop-the-world pauses. The honest caveats: Kafka’s tail latency has narrowed substantially on modern JVMs and KRaft, vendor benchmarks choose favorable settings (for example, fsync=false durability vs acks=all), and ecosystem maturity — Kafka Connect, Streams, monitoring — still favors Kafka. Treat the comparison as workload-specific and demand the benchmark config.

3. Two systems both say “exactly-once.” Is that the same guarantee? No. Kafka’s transactions give exactly-once within the Kafka ecosystem only: idempotent producer (PID + per-partition sequence), atomic offsets-and-outputs commit, and read_committed consumers. NATS JetStream’s exactly-once is a producer-side dedup window (Nats-Msg-Id with a duplicate_window) plus double-ack, scoped per stream. Neither covers side effects outside the system — writing to a database, calling an API — which still needs sink idempotency or a transactional outbox. Any “exactly-once” claim must be qualified by its scope boundaries.

4. What is the single most consequential architectural difference between Kafka and Pulsar? Storage coupling. Kafka stores partition replicas in the broker’s own filesystem, so scaling storage means scaling brokers and moving data; a broker replacement re-copies gigabytes. Pulsar brokers own nothing durable — ledgers live in BookKeeper bookies with configurable ensemble/write/ack quorums — so brokers can start, crash, or scale in seconds while data stays put. The cost is one more distributed system to run, monitor, and upgrade. Everything else (bundles, tiered storage ease, failover speed) follows from this decision.

5. Which system for a fleet of 50,000 small IoT gateways that intermittently connect? NATS, typically with leaf nodes on the gateways. A leaf node connects into a NATS supercluster and synthesizes the global subject graph locally, so gateways see only subjects they care about over slow links; the binary is a few tens of MB and runs in <100 MB RAM. Kafka assumes well-connected clients with substantial client-side state (metadata, index files, buffers), and its per-partition model fits poorly with 50k intermittent publishers. JetStream adds durable retention only where it is needed, keeping the edge cheap.

Key Takeaways

• Score systems on five mechanisms — log placement, assignment owner, delivery machinery, flow control, operational topology — never on feature lists.
• Kafka’s coupled storage makes it a single-system win; Pulsar’s BookKeeper decoupling makes it the elasticity and multi-tenancy win; the trade is always extra always-on components.
• Redpanda reuses the Kafka wire protocol but changes the engine: thread-per-core scheduling, Raft-native metadata, no GC — verify benchmarks against your durability settings.
• Exactly-once is a scoped mechanism, not a property: Kafka transactions end at the broker boundary; NATS dedup ends at the stream window; side effects always need their own idempotency.
• JetStream streams, Kafka topics, and Pulsar ledgers are all “partitioned durable logs” — the interesting interview content is how each replicates and fences writers.
• BookKeeper is storage infrastructure, not a message system; understanding it explains both Pulsar’s behavior and Kafka’s KIP-405 tiered-storage design.
• Backpressure is a system property spanning producer batching, broker quotas, consumer lag, and the TCP window — no single knob implements it.

References

• Apache Kafka documentation (design + implementation sections): kafka.apache.org/documentation
• Apache Pulsar documentation: pulsar.apache.org/docs
• Apache BookKeeper overview: bookkeeper.apache.org/docs/overview
• NATS documentation: docs.nats.io
• Redpanda documentation: docs.redpanda.com
• Seastar framework: seastar.io, source: github.com/scylladb/seastar
• Kreps, Narkhede, Rao — “Kafka: a Distributed Messaging System for Log Processing,” NetDB Workshop, 2011 (cite title + venue)
• Hunt, Konar, Junqueira, Reed — “ZooKeeper: Wait-free Coordination for Internet-scale Systems,” USENIX ATC 2010 (cite title + venue)

Cross-References

• Kafka Overview — basics of topics, partitions, consumer groups
• Apache Pulsar — subscription types, functions, feature comparison
• Kafka Connect — source/sink framework whose idempotency EOS does not cover
• Kafka Streams — the canonical consume-transform-produce loop
• Kafka (Backend) — producer/consumer API-level treatment
• NATS (Backend) — core NATS subjects, queue groups, request/reply
• Raft — consensus behind KRaft, Redpanda metadata, JetStream meta-groups
• Backpressure — system-design-level load management