Apache Pulsar (Distributed Messaging and Streaming)

Overview

Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally developed at Yahoo and now an Apache top-level project. It combines the flexibility of a pub/sub messaging system with the durability and scalability of a distributed log, offering a unified solution for both queuing and streaming use cases.

Why Pulsar?

graph TB
    subgraph "Pulsar Unifies"
        MQ[Message Queue] --> Pulsar[Pulsar]
        LOG[Event Log] --> Pulsar
        STREAM[Stream Processing] --> Pulsar
    end

    subgraph "Traditional Stack"
        MQ2[Kafka] --> Flink[Flink/Spark]
        MQ3[RabbitMQ] --> Queue[Queue Processing]
    end

Architecture

Separation of Compute and Storage

graph TB
    subgraph "Compute Layer"
        P1[Broker 1] 
        P2[Broker 2]
        P3[Broker 3]
        Client --> P1
        Client --> P2
        Client --> P3
    end

    subgraph "Metadata Layer"
        ZK[ZooKeeper / Metadata Store]
    end

    subgraph "Storage Layer (BookKeeper)"
        BK1[Bookie 1]
        BK2[Bookie 2]
        BK3[Bookie 3]
    end

    P1 --- ZK
    P2 --- ZK
    P3 --- ZK
    P1 --- BK1
    P1 --- BK2
    P2 --- BK2
    P2 --- BK3
    P3 --- BK3
    P3 --- BK1

This separation means:

• Brokers are stateless: Easy to scale, replace, or upgrade
• Storage is distributed: BookKeeper provides durable, replicated ledgers
• No data loss on broker failure: Data is already persisted in BookKeeper

Core Components

Topics, Partitions, and Ledgers

graph TB
    Topic[persistent://tenant/ns/topic] 
    Topic --- P0[Partition 0]
    Topic --- P1[Partition 1]
    Topic --- P2[Partition 2]

    P0 --- L1["Ledger 1 (Active)"]
    P0 --- L2["Ledger 2 (Sealed)"]
    P0 --- L3["Ledger 3 (Sealed)"]

Topic Types

Subscription Types

This is a key differentiator from Kafka, which only supports consumer groups:

graph TB
    Topic[Topic] 
    Topic -->|All messages| Exclusive[Exclusive: 1 consumer]
    Topic -->|Round-robin| Shared[Shared: N consumers]
    Topic -->|One at a time| Failover[Failover: 1 active, N standby]
    Topic -->|By key| KeyShared[Key_Shared: ordered by key]

Message Delivery Guarantees

Acknowledgment and Cursor

• Each subscription maintains a cursor tracking which messages have been consumed
• Consumers ack individual messages or cumulatively (all messages up to a point)
• Unacked messages are redelivered when a consumer disconnects or a timeout expires

Geo-Replication

graph LR
    subgraph "Region A (us-east)"
        ClusterA[Cluster A] 
        ClusterA --> ReplicatorA[Replicator]
    end

    subgraph "Region B (eu-west)"
        ClusterB[Cluster B]
        ClusterB --> ReplicatorB[Replicator]
    end

    ReplicatorA <-->|Async| ReplicatorB

• Built-in cross-cluster asynchronous replication
• Policies: active-active, active-passive, aggregation
• Configure at the namespace or topic level
• Automatic conflict resolution (last-write-wins per message ID)

Tiered Storage

graph LR
    subgraph "Hot Storage"
        BK[BookKeeper<br/>Fast, SSD]
    end

    subgraph "Cold Storage"
        S3[S3 / GCS / Azure Blob<br/>Cheap, high capacity]
    end

    BK -->|"Offload policy<br/>(size + age)"| S3
    Consumer[Consumer] -->|Seamless read| BK
    Consumer -->|Seamless read| S3

• Offload ledger data to S3/GCS based on size or time
• Consumers read transparently from hot or cold storage
• Reduces cost while keeping data accessible

Schema Registry

Pulsar has a built-in schema registry (no external dependency like Confluent Schema Registry):

// Producer with schema
Producer<Order> producer = client.newProducer(Schema.AVRO(Order.class))
    .topic("orders")
    .create();

// Consumer with schema
Consumer<Order> consumer = client.newConsumer(Schema.AVRO(Order.class))
    .topic("orders")
    .subscriptionName("orders-sub")
    .subscribe();

Pulsar Functions

Lightweight, serverless compute built into Pulsar:

// A Pulsar Function (deployed to the cluster)
public class WordCountFunction implements Function<String, String> {
    @Override
    public String apply(String input) {
        return input + " [processed at " + Instant.now() + "]";
    }
}

Comparison: Pulsar vs Kafka

Comparison: Pulsar vs RabbitMQ

Performance Characteristics

Interview Questions

1. Why would you choose Pulsar over Kafka? Choose Pulsar when you need: built-in geo-replication, multi-tenancy, multiple subscription types, or independent scaling of compute and storage. Kafka when you need maximum ecosystem maturity and simplicity.

2. How does Pulsar handle broker failure? Because brokers are stateless (data is in BookKeeper), a failed broker simply means clients reconnect to another broker. No data recovery, no partition reassignment — the new broker serves the same ledgers from BookKeeper.

3. Explain the four subscription types. Exclusive: one consumer gets everything. Shared: round-robin across consumers (no ordering). Failover: one active consumer with standbys. Key_Shared: messages with the same routing key always go to the same consumer.

4. What is tiered storage and why does it matter? Tiered storage offloads older data from BookKeeper to cheap object storage (S3/GCS). This enables retaining data for months/years at low cost while keeping recent data fast to access. Kafka has similar capabilities but Pulsar’s is more tightly integrated.

5. How does Pulsar ensure exactly-once semantics? Through transactions: producers can publish to multiple topics atomically, and consumers can ack from multiple topics atomically. Combined with idempotent producers (message deduplication by sequence ID) and broker-side transaction log.

Key Takeaways

• Pulsar separates compute (brokers) from storage (BookKeeper) for independent scaling and zero data loss on broker failure
• Four subscription types provide flexibility beyond Kafka’s consumer groups
• Built-in geo-replication, multi-tenancy, schema registry, and tiered storage reduce operational complexity
• Pulsar Functions provide lightweight, serverless stream processing
• Growing adoption but Kafka still leads in ecosystem maturity and community size
• Consider Pulsar for new projects needing multi-region, multi-tenant streaming with diverse consumption patterns

Cross-References

• Kafka — Detailed Kafka comparison
• RabbitMQ — Queue-based alternative
• Pub/Sub Patterns — Messaging patterns overview
• Amazon Kinesis — AWS managed streaming
• EventBridge — AWS event routing
• Consensus (ZooKeeper) — Coordination primitives
• Stream Processing — Processing paradigms