Kubernetes Scheduling Deep Dive

Scheduler Architecture

The kube-scheduler is a control plane component responsible for assigning newly created Pods to nodes. It watches the API server for Pods with no nodeName assigned and makes placement decisions through a well-defined pipeline.

The scheduler runs as a single active instance (leader-elected) and does NOT place containers directly—it writes the nodeName field to the Pod spec via the API server, and the target node’s kubelet handles the actual container creation.

Scheduling Cycle

Unscheduled Pod
    │
    ▼
┌─────────────────────┐
│  Scheduling Queue    │  Pods waiting to be scheduled (priority sorted)
└────────┬────────────┘
         ▼
┌─────────────────────┐
│  Filtering Phase     │  Eliminate infeasible nodes
└────────┬────────────┘
         ▼
┌─────────────────────┐
│  Scoring Phase       │  Rank feasible nodes
└────────┬────────────┘
         ▼
┌─────────────────────┐
│  Binding Phase       │  Write nodeName to Pod spec
└─────────────────────┘

Filtering Phase

Each registered filter plugin evaluates every node. If ANY plugin rejects a node, it is removed from candidacy. Default filters include:

Scoring Phase

Feasible nodes are scored by each scoring plugin. Scores are normalized and summed. Default scorers:

The node with the highest total score is selected. Ties are broken randomly.

Taints and Tolerations

Taints are applied to nodes to repel pods; tolerations are applied to pods to absorb taints.

# Add a taint to a node
kubectl taint nodes node1 dedicated=gpu:NoSchedule

# Pod spec with toleration
tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

Built-in taints (automatically applied by controllers):

• node.kubernetes.io/not-ready — Node not ready (kubelet unhealthy)
• node.kubernetes.io/unreachable — Node unreachable from control plane
• node.kubernetes.io/memory-pressure — Node under memory pressure
• node.kubernetes.io/disk-pressure — Node under disk pressure

Node Affinity and Pod Affinity

Node Affinity

Constrains pod scheduling based on node labels. Two types:

# Required (hard constraint) - must match
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values: ["us-east-1a", "us-east-1b"]

# Preferred (soft constraint) - scheduler tries to satisfy
affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
            - key: node-type
              operator: In
              values: ["high-memory"]

Pod Affinity / Anti-Affinity

Constrains scheduling based on labels of already-running pods:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: web
        topologyKey: topology.kubernetes.io/zone
    # Ensures web pods spread across zones

Pod Topology Spread Constraints

Topology spread constraints provide more predictable spreading than anti-affinity. They enforce max skew across topology domains.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule  # hard constraint
    labelSelector:
      matchLabels:
        app: api
  - maxSkew: 2
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway  # soft constraint (best-effort)
    labelSelector:
      matchLabels:
        app: api

Max skew is the difference between the number of pods on any two topology domains. With maxSkew: 1 and 5 pods across 3 zones, the distribution would be 2-2-1 (max diff = 1), never 3-1-1.

Resource Requests and Limits

resources:
  requests:        # Used by scheduler for placement decisions
    cpu: "250m"     # 0.25 CPU cores
    memory: "256Mi"
  limits:          # Enforced by kubelet (hard cap)
    cpu: "500m"     # Throttled if exceeded
    memory: "512Mi" # OOMKilled if exceeded

QoS Classes (derived from requests/limits):

Scheduler Plugins

The scheduler uses a plugin architecture (since v1.19) with extension points:

Custom Schedulers

Run a second scheduler alongside the default one:

apiVersion: kubescheduler.config.k8s.io/v1beta3
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /etc/kubernetes/scheduler.conf
profiles:
  - schedulerName: my-custom-scheduler
    plugins:
      # Enable/disable/override plugins

Target pods at your custom scheduler:

spec:
  schedulerName: my-custom-scheduler

Use cases for custom schedulers: GPU bin-packing, cost-aware scheduling, latency-sensitive workloads, gang scheduling (Volcano), topology-aware scheduling.

References

• Kubernetes Scheduler Documentation
• Scheduler Framework
• Topology Spread Constraints

Interview Questions

Q1: How does the Kubernetes scheduler decide where to place a pod?

Answer: The scheduler runs a two-phase process. Filtering eliminates infeasible nodes (insufficient resources, taints not tolerated, node affinity mismatch, volume binding failure). Scoring ranks the remaining nodes using multiple plugins (resource fit, affinity, image locality). The highest-scoring node wins. Ties are broken randomly. The scheduler writes nodeName to the Pod spec—it never talks to kubelet directly.

Q2: What is the difference between node affinity and pod anti-affinity?

Answer: Node affinity constrains scheduling based on node labels (e.g., schedule on nodes in us-east-1a or on GPU nodes). Pod anti-affinity constrains scheduling based on labels of already-running pods (e.g., don’t schedule two replicas of the same app on the same node). Node affinity answers “where should this pod go?” Pod anti-affinity answers “where should this pod NOT go relative to other pods?”

Q3: What happens when a pod exceeds its memory limit?

Answer: The container is OOMKilled by the kernel’s OOM killer. The container is terminated with exit code 137. If the pod has a restart policy of Always or OnFailure, kubelet recreates it. CPU limits work differently—exceeding CPU limits causes throttling (CFS quota), not termination. This asymmetry is a common interview trap.

Q4: Explain topology spread constraints vs. pod anti-affinity.

Answer: Topology spread constraints enforce a maximum skew (difference) in pod count across topology domains (zones, nodes). They give predictable, balanced distribution. Pod anti-affinity prevents scheduling if any matching pod already exists on a node/zone, which can lead to scheduling failures as the cluster fills. Topology spread is generally preferred for production because it’s more flexible and provides better utilization while still ensuring spread.

Q5: When would you use a custom scheduler?

Answer: Use a custom scheduler when the default scheduler’s decisions are suboptimal for specialized workloads. Examples: GPU workloads needing bin-packing (pack GPU pods tightly to leave room for non-GPU work), gang scheduling (all pods of a job must start together, e.g., Spark, MPI via Volcano), cost-aware scheduling (prefer spot/preemptible instances), or latency-sensitive workloads that need NUMA-aware placement. You specify schedulerName in the pod spec to target the custom scheduler.