Kubernetes Storage Deep Dive

Volume Types

Volumes provide storage to pods that outlives the container’s lifecycle (but not the pod’s lifecycle, except for PersistentVolumes).

emptyDir

volumes:
  - name: cache-volume
    emptyDir:
      medium: Memory  # Uses tmpfs (RAM-backed), limited by node memory
      sizeLimit: 256Mi

Default emptyDir uses the node’s disk. With medium: Memory, it uses RAM (tmpfs)—fast but counts against the pod’s memory limit. Common use case: a sidecar container writes to an emptyDir volume that the main container reads from (e.g., log shipper sidecar).

PersistentVolumes and PersistentVolumeClaims

The PV/PVC abstraction decouples storage provisioning from consumption.

┌──────────────────┐     ┌───────────────────────────┐     ┌──────────────────┐
│    Admin/         │     │  PV: 50Gi SSD, RWO       │     │  Pod             │
│  Dynamic          │────▶│  PVC: 10Gi, ReadWriteOnce│◀────│  mounts PVC       │
│  Provisioning     │     │  StorageClass: fast-ssd   │     │                  │
└──────────────────┘     └───────────────────────────┘     └──────────────────┘

Access modes:

Storage Classes

StorageClasses enable dynamic provisioning—PVCs are automatically fulfilled without pre-created PVs.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "5000"
  throughput: "250"
  encrypted: "true"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

WaitForFirstConsumer is critical for zone-aware provisioning (EBS, PD). Without it, the volume might be provisioned in us-east-1a while the pod gets scheduled in us-east-1b, causing a mount failure.

StatefulSet Volume Management

StatefulSets use volume claim templates to provision a unique PVC per pod:

apiVersion: apps/v1
kind: StatefulSet
spec:
  serviceName: "mysql-headless"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 100Gi

This creates PVCs named data-mysql-0, data-mysql-1, etc. When pod mysql-2 is deleted and recreated, it reclaims data-mysql-2. This identity is why StatefulSets are used for databases.

CSI (Container Storage Interface)

CSI is the standard interface for storage vendors to integrate with Kubernetes. Before CSI, storage drivers were compiled into the Kubernetes binary (in-tree plugins).

CSI architecture:

  Kubelet ──▶ CSI Node Plugin (on each node)
                  │
                  ▼
             CSI Controller Plugin (on control plane or as operator)
                  │
                  ▼
           Storage Backend (EBS, GCE PD, NFS, Ceph, etc.)

Major CSI drivers:

References

• Kubernetes Storage Concepts
• CSI Specification
• Dynamic Provisioning

Interview Questions

Q1: What is the difference between a Volume, a PersistentVolume, and a PersistentVolumeClaim?

Answer: A Volume is defined in the pod spec and has the same lifetime as the pod—when the pod is deleted, the volume’s data is gone (except for some types like hostPath). A PersistentVolume (PV) is a cluster-level resource representing actual storage (a disk, NFS share, etc.) with an independent lifecycle. A PersistentVolumeClaim (PVC) is a user’s request for storage—specifying size, access mode, and storage class. The PVC is bound to a PV (statically by admin or dynamically via StorageClass), and the PVC is referenced in the pod spec. The PVC can outlive pods that mount it.

Q2: Why is WaitForFirstConsumer important for StorageClasses?

Answer: Cloud block storage (EBS, GCE PD, Azure Disk) is zone-specific. With volumeBindingMode: Immediate, the PVC is bound to a PV before the pod is scheduled—so the volume might be created in us-east-1a while the scheduler places the pod in us-east-1b, causing a mount failure. WaitForFirstConsumer delays PVC binding until a pod that uses it is scheduled. The scheduler then selects a node, and the volume is provisioned in the same zone. This prevents zone mismatch failures.

Q3: How does StatefulSet volume management differ from Deployment?

Answer: StatefulSets use volumeClaimTemplates which create a unique, named PVC for each pod replica (e.g., data-mysql-0, data-mysql-1). When a pod is deleted and recreated, it reattaches to the same PVC, preserving data identity. Deployments share a PVC template but all pods could theoretically share one PVC (with RWX) or get ephemeral volumes. StatefulSets also guarantee ordered creation/deletion, so pod-1 isn’t created until pod-0 is running, and pod-0 isn’t deleted until all others are down.

Q4: What is CSI and why did Kubernetes move away from in-tree storage drivers?

Answer: CSI (Container Storage Interface) is a standardized gRPC interface that decouples storage drivers from Kubernetes core. Previously, storage drivers were compiled into the Kubernetes binary (in-tree), meaning any driver bug or update required a Kubernetes release. CSI moves drivers to external plugins, allowing vendors to release independently, support multiple K8s versions, and innovate without the K8s release cycle. CSI drivers run as DaemonSets (node plugin) and StatefulSets/Deployments (controller plugin) on the cluster.

Q5: When would you use emptyDir vs. a PVC?

Answer: Use emptyDir for temporary, ephemeral data that the pod needs during its lifetime but doesn’t need to persist across restarts: scratch space, caching, git clone workspaces, or shared data between containers in the same pod (sidecar pattern). Use a PVC when data must survive pod deletion/recreation: databases, file uploads, application state. An emptyDir with medium: Memory (tmpfs) is useful for high-speed temporary data like computation intermediaries. A common mistake is using emptyDir for data that should persist—when the pod reschedules, the data is lost.