Messaging and Async Processing

Why Async Processing?

Synchronous processing blocks the caller until completion. Async processing decouples producers and consumers, improving responsiveness, reliability, and scalability.

Synchronous:
Client → App → Process → DB → Response (client waits 5s)

Asynchronous:
Client → App → Queue → Response (instant)
                ↓
           Worker → Process → DB (takes 5s, but client doesn't wait)

Message Queues

What is a Message Queue?

A buffer that stores messages between producers and consumers.

[Producer] → ┌─────────────────┐ → [Consumer 1]
              │   Message Queue │ → [Consumer 2]
              │  ┌───┬───┬───┐ │ → [Consumer 3]
              │  │ M1│ M2│ M3│ │
              │  └───┴───┴───┘ │
              └─────────────────┘

Key Concepts

Popular Message Queue Technologies

RabbitMQ vs Kafka

RabbitMQ:
Producer → Exchange → Queue → Consumer (message deleted after ACK)

Kafka:
Producer → Topic → Partition → Consumer Group (message retained)
              └── Partition 2 → Consumer 2

Pub/Sub (Publish-Subscribe)

How Pub/Sub Works

                    ┌──────────────┐
[Publisher A] ────→│              │──→ [Subscriber 1]
                   │  Pub/Sub    │──→ [Subscriber 2]
[Publisher B] ────→│  Topic      │──→ [Subscriber 3]
                   └──────────────┘

• Publishers send messages to a topic
• All subscribers to that topic receive the message
• Publishers don’t know about subscribers (decoupling)

Queue vs Pub/Sub

When to Use Pub/Sub

• Event notifications: “User signed up” → notify email, analytics, CRM
• Real-time updates: Chat messages, live feeds
• Fan-out: Distribute work to multiple consumers
• Microservices communication: Decoupled service-to-service

Event-Driven Architecture

What is Event-Driven?

Components communicate through events rather than direct calls.

Traditional (Request-Response):
App → Order Service → Payment Service → Inventory Service
         (waits)          (waits)           (returns)

Event-Driven:
App → Order Service → emits "OrderCreated" event
                           ↓
              ┌────────────┼────────────┐
              ▼            ▼            ▼
        Payment Service  Inventory   Notification
        (processes)      Service     Service
                         (processes) (notifies)

Event Sourcing

Store all state changes as a sequence of events.

Traditional (State Storage):
Account: { balance: 100 }

Event Sourcing:
Events: [
  { type: "AccountCreated", amount: 0 },
  { type: "Deposit", amount: 150 },
  { type: "Withdrawal", amount: 50 }
]
Current state: replay events → balance: 100

Benefits:

• Complete audit trail
• Can reconstruct any point-in-time state
• Events are immutable
• Easy to add new projections/read models

Drawbacks:

• Complex to implement
• Event schema evolution is hard
• Queries require read models (CQRS)

CQRS (Command Query Responsibility Segregation)

Separate write model (commands) from read model (queries).

Commands (Write) → Event Store → Event Handlers → Read DB (optimized for queries)
                                                         ↓
Queries (Read) → Read DB ← ──────────────────────────────┘

Backpressure

When consumers can’t keep up with producers.

Producer: 10,000 msg/sec
Consumer: 1,000 msg/sec (max)
Queue: GROWING → memory issues → crash

Backpressure Strategies

Consumer Lag Monitoring

Producer offset: 1,000,000
Consumer offset: 995,000
Lag: 5,000 messages → Consumer is behind

Message Delivery Guarantees

At-Most-Once

Producer → Queue → Consumer (ACK after receive, may not process)

• Messages may be lost
• No duplicates
• Fast, simple
• Use case: Logging, metrics

At-Least-Once

Producer → Queue → Consumer → Process → ACK
                                  ↓ (fails before ACK)
                             Re-delivered → Process again

• Messages never lost
• May be processed multiple times (idempotent processing needed)
• Most common choice
• Use case: Order processing, notifications

Exactly-Once

Producer → Queue → Consumer → Process → Dedup → ACK

• Never lost, never duplicated
• Hardest to achieve
• Requires idempotency + deduplication
• Use case: Financial transactions

Message Queue Patterns

Dead Letter Queue (DLQ)

Main Queue → Consumer (fails 3 times) → DLQ
                                         ↓
                                    Manual review / Alert

Priority Queue

┌─────────────────────────┐
│     Priority Queue      │
│ ┌───┐ ┌───┐ ┌───┐     │
│ │ P1│ │ P2│ │ P2│     │
│ │   │ │   │ │   │     │
│ └───┘ └───┘ └───┘     │
└─────────────────────────┘
P1 (high priority) processed first

Delayed Queue

Producer → Delay Queue (wait 5 min) → Main Queue → Consumer

Request-Reply (RPC over Queue)

Client → Request Queue → Server
Client ← Reply Queue ← Server

Real-World Examples

Uber’s Messaging Architecture

• Kafka for event streaming (ride events, location updates)
• Real-time processing for matching drivers and riders
• Event sourcing for ride state machine

Netflix’s Event Processing

• Kafka for data pipeline (billions of events/day)
• Apache Flink for real-time processing
• Event-driven microservices

LinkedIn’s Kafka

• Originally built at LinkedIn
• Handles 7+ trillion messages/day
• Powers activity feeds, metrics, logging

Interview Tips

1. Identify async opportunities — “Sending email doesn’t need to block the response”
2. Choose the right technology — “Kafka for event streaming, SQS for simple tasks”
3. Discuss delivery guarantees — “At-least-once with idempotent processing”
4. Mention DLQ — “Failed messages go to DLQ for investigation”
5. Consider backpressure — “What happens if consumers can’t keep up?”
6. Think about ordering — “Do we need ordered processing? Per-partition ordering in Kafka”
7. Discuss idempotency — “Workers must be idempotent since messages may be delivered twice”
8. Monitor consumer lag — “Alert if lag exceeds 10K messages”

Common Mistakes

• ❌ Using sync processing for everything
• ❌ Not handling message failures (no DLQ)
• ❌ Ignoring ordering requirements
• ❌ Not making consumers idempotent
• ❌ Over-engineering with Kafka when SQS would suffice
• ❌ Not monitoring consumer lag

Cross-References

• Scalability — Async processing enables horizontal scaling
• Availability — Queues improve fault tolerance
• Consistency Tradeoffs — Eventual consistency in async systems
• Monitoring — Monitor queue depth, consumer lag
• Notification Service LLD — Implementation example