Amazon S3 (Simple Storage Service)

Introduction

Amazon S3 is an object storage service offering industry-leading scalability, data availability, security, and performance. It stores data as objects within buckets, with virtually unlimited storage capacity and a pay-as-you-go model.

S3 Architecture

graph TB
    CLIENT[Client Application]
    BUCKET[S3 Bucket - globally unique name]
    OBJECT1[Object 1 - key, data, metadata]
    OBJECT2[Object 2 - key, data, metadata]
    OBJECT3[Object 3 - key, data, metadata]

    CLIENT --> |PUT/GET/DELETE| BUCKET
    BUCKET --> OBJECT1
    BUCKET --> OBJECT2
    BUCKET --> OBJECT3

    subgraph "Object Components"
        KEY[Key - Unique identifier]
        DATA[Data - 0 bytes to 5 TB]
        META[Metadata - System & User]
        VER[Version ID - if versioning enabled]
        ACL[Access Control]
    end

Key Concepts

Storage Classes

graph TB
    S3[S3 Storage Classes] --> S3STANDARD[S3 Standard]
    S3 --> S3IA[S3 Standard-IA]
    S3 --> S3OIA[S3 One Zone-IA]
    S3 --> S3GI[S3 Glacier Instant Retrieval]
    S3 --> S3GF[S3 Glacier Flexible Retrieval]
    S3 --> S3GDA[S3 Glacier Deep Archive]
    S3 --> S3INTELL[S3 Intelligent-Tiering]

    S3STANDARD --> |Frequent access| ST_DUR[99.999999999% durability]
    S3IA --> |Infrequent access| IA_DUR[Same durability, lower cost, retrieval fee]
    S3OIA --> |Recreatable data| OZ_DUR[Single AZ, lower cost]
    S3GI --> |Archive, instant access| GI_DUR[Archive pricing, milliseconds retrieval]
    S3GF --> |Archive| GF_DUR[Minutes to hours retrieval]
    S3GDA --> |Long-term archive| GDA_DUR[12+ hours retrieval]
    S3INTELL --> |Unknown/changing patterns| INT_DUR[Auto-tiering, no retrieval fees]

S3 Intelligent-Tiering

graph LR
    PUT[Object PUT] --> FREQUENT[Frequent Access Tier]
    FREQUENT --> |No access for 30 days| INFREQUENT[Infrequent Access Tier]
    INFREQUENT --> |No access for 90 days| ARCHIVE[Archive Instant Access]
    ARCHIVE --> |No access for 180+ days| DEEP[Deep Archive Access]
    INFREQUENT --> |Access detected| FREQUENT
    ARCHIVE --> |Access detected| FREQUENT
    DEEP --> |Access detected| FREQUENT

Automatically moves objects between access tiers based on usage patterns. No retrieval fees.

S3 Lifecycle Rules

Automate transitioning objects between storage classes and expiring (deleting) them:

graph LR
    UPLOAD[Object Uploaded] --> STD[S3 Standard]
    STD --> |After 30 days| IA[S3 Standard-IA]
    IA --> |After 90 days| GL[S3 Glacier Flexible]
    GL --> |After 365 days| DEL[Delete]

{
    "Rules": [
        {
            "ID": "TransitionToIA",
            "Filter": { "Prefix": "logs/" },
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 30,
                    "StorageClass": "STANDARD_IA"
                },
                {
                    "Days": 90,
                    "StorageClass": "GLACIER"
                }
            ],
            "Expiration": {
                "Days": 365
            }
        }
    ]
}

Lifecycle Rule Components:

• Filter: Which objects to target (prefix, tags, or all)
• Transitions: Move to cheaper storage class after N days
• Expiration: Delete objects after N days
• NoncurrentVersionTransitions: Apply to old versions
• NoncurrentVersionExpiration: Delete old versions

S3 Versioning

sequenceDiagram
    participant Client
    participant S3

    Client->>S3: PUT object (report.pdf)
    Note over S3: Version 1 (ID: aaa)

    Client->>S3: PUT object (report.pdf)
    Note over S3: Version 2 (ID: bbb), Version 1 still exists

    Client->>S3: DELETE object (report.pdf)
    Note over S3: Delete marker added, Version 2 still exists

    Client->>S3: GET object (report.pdf)
    Note over S3: Returns 404 (delete marker is current)

    Client->>S3: GET object?versionId=bbb
    Note over S3: Returns Version 2 (data intact)

Versioning Details:

• Once enabled, cannot be disabled (only suspended)
• All versions are stored (including deletes—they become “delete markers”)
• Protects against accidental deletion and overwrites
• Can be used with lifecycle rules to delete old versions
• MFA Delete: Require MFA to permanently delete versions

S3 Consistency Model

Since December 2020, S3 provides strong read-after-write consistency:

sequenceDiagram
    participant Client_A as Client A
    participant Client_B as Client B
    participant S3

    Client_A->>S3: PUT new object
    Client_B->>S3: GET object
    Note over S3: Strong consistency: always returns new object

    Client_A->>S3: PUT overwrite existing object
    Client_B->>S3: GET object
    Note over S3: Strong consistency: always returns updated object

    Client_A->>S3: DELETE object
    Client_B->>S3: GET object
    Note over S3: Strong consistency: returns 404 after delete

Historical note: Before December 2020, S3 had eventual consistency for overwrite PUTs and DELETEs. This is no longer the case.

S3 Security

Access Control

graph TB
    ACCESS[S3 Access Control] --> IAM[IAM Policies]
    ACCESS --> BP[Bucket Policies]
    ACCESS --> ACL[ACLs - Legacy]
    ACCESS --> PP[Pre-signed URLs]
    ACCESS --> VPC[VPC Endpoints]
    ACCESS --> ENC[Encryption]

    IAM --> |Identity-based| IAM_D[Grant users/roles access]
    BP --> |Resource-based| BP_D[Grant cross-account, public access]
    ACL --> |Object-level| ACL_D[Legacy, not recommended]
    PP --> |Temporary URL| PP_D[Time-limited access to objects]
    VPC --> |Private connectivity| VPC_D[Access S3 without internet]
    ENC --> |At rest & in transit| ENC_D[SSE-S3, SSE-KMS, SSE-C]

Encryption

graph LR
    subgraph "Server-Side Encryption"
        CLIENT_SSE[Client] --> |HTTPS + SSE header| S3_SSE[S3]
        S3_SSE --> |Encrypts with key| STORE[Encrypted Storage]
    end

    subgraph "SSE-KMS Flow"
        CLIENT_KMS[Client] --> |PUT + x-amz-server-side-encryption: aws:kms| S3_KMS[S3]
        S3_KMS --> |Request key| KMS[AWS KMS]
        KMS --> |Data key| S3_KMS
        S3_KMS --> |Encrypt object| STORE_KMS[Encrypted Storage]
    end

Block Public Access

S3 Block Public Access settings override bucket policies and ACLs:

graph TB
    BPA[Block Public Access] --> BPA1[Block all public access]
    BPA --> BPA2[Block public bucket policies]
    BPA --> BPA3[Block public ACLs]
    BPA --> BPA4[Block public and cross-account ACLs]

Best Practice: Enable “Block all public access” at the account level unless you specifically need public buckets (e.g., static website hosting).

S3 Features

Cross-Region Replication (CRR)

graph LR
    SOURCE[Source Bucket - us-east-1] --> |Async replication| DEST[Destination Bucket - eu-west-1]
    IAM_ROLE[IAM Role] --> SOURCE
    IAM_ROLE --> DEST

• Requires versioning enabled on both buckets
• Asynchronous replication (typically minutes)
• Replicates new objects and updates
• Use cases: compliance, latency reduction, disaster recovery

S3 Transfer Acceleration

Uses CloudFront’s edge locations to accelerate uploads to S3:

graph LR
    CLIENT_TA[Client] --> |Slow internet| EDGE[Nearest CloudFront Edge]
    EDGE --> |AWS backbone| S3_TA[S3 Bucket]

• Upload to nearest edge location → fast transfer over AWS backbone
• Useful for long-distance uploads (e.g., uploading from Asia to US bucket)

S3 Event Notifications

sequenceDiagram
    participant App
    participant S3
    participant SQS as SQS / SNS / Lambda

    App->>S3: PUT object (image.jpg)
    S3->>SQS: Event notification
    SQS->>App: Process uploaded image

Triggers on: PUT, POST, COPY, DELETE, restore events. Routes to: SNS, SQS, Lambda.

S3 Static Website Hosting

graph TB
    USER[User] --> |HTTP| CF[CloudFront - Optional]
    CF --> |Origin| S3WEB[S3 Bucket - Static Website]
    S3WEB --> HTML[index.html]
    S3WEB --> ERR[error.html]
    S3WEB --> CSS[styles.css]
    S3WEB --> JS[app.js]

# Enable static website hosting
aws s3 website s3://my-bucket \
    --index-document index.html \
    --error-document error.html

S3 Performance

# Multi-part upload for large files
aws s3 cp large-file.zip s3://my-bucket/ --storage-class STANDARD
# AWS CLI automatically uses multi-part for large files

Interview Questions

Q1: What are the different S3 storage classes and when would you use each?

Answer: S3 Standard for frequently accessed data. Standard-IA for infrequent access with rapid retrieval needs (30-day minimum). One Zone-IA for recreatable infrequent data (single AZ). Glacier Instant Retrieval for archive data needing millisecond access. Glacier Flexible for archive with minutes-to-hours retrieval. Glacier Deep Archive for long-term archive (12-48 hour retrieval). Intelligent-Tiering when access patterns are unknown—it auto-tiering based on usage.

Q2: Explain S3 versioning and its implications.

Answer: Versioning preserves all versions of an object, including overwrites and deletes. When enabled, PUTs create new versions; DELETEs add delete markers. This protects against accidental deletion—you can retrieve any previous version. Cannot be disabled, only suspended. Costs increase since all versions are stored (use lifecycle rules to manage old versions). MFA Delete adds an extra security layer requiring MFA for permanent deletion.

Q3: What is S3’s consistency model?

Answer: S3 provides strong read-after-write consistency for all operations since December 2020. A read immediately after a PUT (new or overwrite) always returns the latest data. DELETEs are also strongly consistent—GETs after DELETE return 404. LIST operations are also strongly consistent. This replaced the previous eventual consistency model for overwrites and deletes.

Q4: How do you secure an S3 bucket?

Answer: (1) Enable Block Public Access at account level, (2) Use bucket policies with least privilege, (3) Enable default encryption (SSE-S3 or SSE-KMS), (4) Use VPC endpoints for private access, (5) Enable access logging and CloudTrail data events, (6) Use pre-signed URLs for temporary access, (7) Enable versioning and MFA Delete for critical data, (8) Use S3 Object Lock for compliance (WORM).

Q5: How would you optimize S3 performance for a high-throughput workload?

Answer: (1) Use multi-part upload for objects > 100 MB, (2) Randomize key prefixes to distribute across partitions (avoid sequential prefixes), (3) Use S3 Transfer Acceleration for long-distance uploads, (4) Use byte-range fetches for parallel downloads, (5) Use CloudFront for read-heavy workloads (caching at edge), (6) Consider S3 Express One Zone for single-digit millisecond latency. S3 scales automatically—5,500 GET and 3,500 PUT per prefix per second.

Common Mistakes

1. Public bucket exposure: Not enabling Block Public Access—leads to data breaches
2. No versioning on critical buckets: Accidental overwrites/deletes are unrecoverable
3. Ignoring lifecycle rules: Keeping all data in Standard forever when it could be archived
4. Using Glacier for frequently accessed data: Retrieval fees make it expensive for frequent access
5. Not using multi-part upload: Large single-part uploads fail and waste bandwidth on retry
6. Sequential key prefixes: Causing S3 partition hot spots and throttling
7. Storing secrets in S3 without encryption: Always encrypt sensitive data

Summary

Cross-References

• EC2: Instances — Compute that accesses S3
• Lambda: Event Sources — S3 event triggers
• VPC: Endpoints — Private S3 access
• CI/CD: Pipelines — Store build artifacts in S3
• Cloud Overview: Storage Models — Cloud storage concepts