Amazon Kinesis (Real-Time Data Streaming)

Overview

Amazon Kinesis is a family of managed services for real-time data streaming at scale. It enables you to ingest, process, and analyze streaming data in real time — clickstreams, IoT telemetry, application logs, financial transactions, and more. Unlike batch processing, Kinesis processes data as it arrives, enabling sub-second insights.

Kinesis Family

graph TB
    subgraph "Kinesis Family"
        KS[Kinesis Data Streams<br/>Real-time data streaming]
        KF[Kinesis Data Firehose<br/>ETL + delivery to destinations]
        KA[Kinesis Data Analytics<br/>SQL-based stream processing]
        KV[Kinesis Video Streams<br/>Real-time video ingestion]
    end

    subgraph "Sources"
        App[Applications]
        IoT[IoT Devices]
        Log[Log Forwarders]
        DB[Database CDC]
    end

    subgraph "Destinations"
        S3[S3]
        Redshift[Redshift]
        OS[OpenSearch]
        Splunk[Splunk]
    end

    App --> KS
    IoT --> KS
    Log --> KF
    DB --> KF
    KS --> KA
    KA --> Redshift
    KS --> KF
    KF --> S3
    KF --> OS
    KF --> Splunk

Kinesis Data Streams

Core Architecture

graph TB
    Producers[Producers] -->|PutRecord/PutRecords| KS[Kinesis Data Stream]
    KS -->|GetRecords| C1[Consumer 1]
    KS -->|GetRecords| C2[Consumer 2]
    KS -->|GetRecords| C3[Consumer 3]

    subgraph "Stream Internals"
        KS --- Shard1[Shard 0]
        KS --- Shard2[Shard 1]
        KS --- ShardN[Shard N]
    end

Shards

A shard is the base throughput unit of a Kinesis stream:

Partition key determines which shard a record lands on. Records with the same partition key are always read in order within that shard.

Data Record Structure

{
  "PartitionKey": "user-12345",
  "Data": "base64-encoded-payload",
  "SequenceNumber": "21269319989653737946712965403778482177",
  "ApproximateArrivalTimestamp": 1705312200.0
}

Consumer Types

Resharding

You can split and merge shards to adjust capacity:

graph LR
    subgraph "Split (Increase Capacity)"
        S1[Shard 1] -->|split| S1a[Shard 1a: hash 0-49]
        S1 -->|split| S1b[Shard 1b: hash 50-99]
    end

    subgraph "Merge (Decrease Capacity)"
        S2a[Shard 2a: hash 0-49] -->|merge| S2[Shard 2: hash 0-99]
        S2b[Shard 2b: hash 50-99] -->|merge| S2
    end

Sequence Numbers and Ordering

Every record gets a monotonically increasing sequence number per shard. Consumers track their position using a checkpoint (sequence number of the last processed record). If a consumer fails, it resumes from the last checkpoint.

Kinesis Data Firehose

Firehose is a fully managed ETL service that delivers streaming data to destinations automatically:

graph LR
    Sources[Sources] --> Firehose[Delivery Stream]
    Firehose -->|Transform| Lambda[Inline Lambda]
    Lambda --> Buffer[Buffering<br/>1-900s, 1-128MB]
    Buffer --> S3[S3]
    Buffer --> RS[Redshift]
    Buffer --> OS[OpenSearch]
    Buffer --> Splunk[Splunk]
    Buffer --> HTTP[HTTP Endpoint]

Firehose vs Direct Streams

Kinesis Data Analytics

Process streaming data using standard SQL:

-- Create a pump from source stream
CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" (
    "user_id" VARCHAR(16),
    "event_type" VARCHAR(32),
    "event_count" INTEGER
);

-- Sliding window aggregation
CREATE OR REPLACE PUMP "STREAM_PUMP" AS
INSERT INTO "DESTINATION_SQL_STREAM"
SELECT STREAM
    "user_id",
    "event_type",
    COUNT(*) AS "event_count"
FROM "SOURCE_SQL_STREAM_001"
GROUP BY
    "user_id",
    "event_type",
    WINDOW SESSION BETWEEN 10 SECOND PRECEDING AND CURRENT ROW;

Window Types

Kinesis Client Library (KCL)

KCL simplifies consumer development by handling:

• Shard distribution across workers
• Checkpointing and fault tolerance
• Load balancing when shards are added/removed
• Coordination via a DynamoDB lease table

// KCL 2.x Worker
KinesisClient kinesisClient = KinesisClient.create();
DynamoDbClient dynamoClient = DynamoDbClient.create();
CloudWatchClient cwClient = CloudWatchClient.create();

Scheduler scheduler = new Scheduler.Builder()
    .recordProcessorFactory(new MyRecordProcessorFactory())
    .streamName("my-stream")
    .kinesisClient(kinesisClient)
    .dynamoDbClient(dynamoClient)
    .cloudWatchClient(cwClient)
    .build();

scheduler.start();

Comparison: Kinesis vs Kafka

Pricing

Limits

Common Use Cases

Interview Questions

1. How do you choose the number of shards? Estimate peak throughput: shards = ceil(max(peak_write_MB/s, peak_records/s / 1000)). Plan for headroom. Remember read throughput is 2x write.

2. What happens when a shard is split? Existing records remain on the parent shard. New records route to child shards based on the hash key range. Consumers see both parent and child shards during the transition.

3. How does KCL handle failover? KCL uses a DynamoDB table to track lease ownership. If a worker dies, its lease expires (default 60s) and another worker claims it, resuming from the last checkpoint.

4. When would you use Firehose over Streams? Use Firehose when you need simple, managed delivery to AWS destinations without custom processing. Use Streams when you need multiple consumers, custom processing, or sub-second latency.

5. How do you achieve exactly-once processing? Kinesis provides at-least-once delivery. For exactly-once, use idempotent consumers (deduplication by sequence number) or Kinesis Data Analytics with checkpointing.

Key Takeaways

• Kinesis Streams provides managed real-time data streaming with shard-based scaling
• Firehose is the zero-code ETL pipeline for delivering streaming data to destinations
• Kinesis Data Analytics enables SQL-based stream processing with windowing
• KCL handles consumer coordination, checkpointing, and fault tolerance
• Shard count directly determines cost and throughput — plan capacity carefully
• Enhanced Fan-Out provides dedicated throughput per consumer at lower latency
• Compare with Kafka: Kinesis for AWS-native simplicity, Kafka for ecosystem richness

Cross-References

• AWS Lambda — Common Kinesis consumer
• Amazon S3 — Common Firehose destination
• Amazon EventBridge — Event routing complement
• ElastiCache — Cache for streaming aggregation results
• Apache Pulsar — Alternative streaming platform
• Kafka — Detailed Kafka comparison
• Stream Processing — Stream processing patterns