Chaos Engineering
Principles
- Build a hypothesis about steady-state behavior
- Vary real-world events (server failure, network latency, resource exhaustion)
- Run experiments in production (or staging)
- Automate experiments to run continuously
- Minimize blast radius (start small, expand gradually)
Steady State Hypothesis
"Under normal conditions, our API serves 1000 rps with p99 < 200ms
and error rate < 0.1%."
If this holds during chaos experiment → system is resilient.
If this breaks → found a weakness to fix.
Common Experiments
| Experiment | Tool | What It Tests |
|---|---|---|
| Kill a server | Chaos Monkey | Failover, redundancy |
| Add network latency | TC, Toxiproxy | Timeouts, retries |
| Fill disk | dd, chaos-mesh | Alerting, cleanup |
| CPU stress | stress-ng | Autoscaling, throttling |
| DNS failure | Block DNS | Fallback, caching |
| Dependency failure | Mock server | Circuit breakers |
Tools
| Tool | Platform | Approach |
|---|---|---|
| Chaos Monkey | AWS | Random instance termination |
| Litmus | Kubernetes | CRD-based experiments |
| chaos-mesh | Kubernetes | Pod/network/IO chaos |
| Gremlin | Multi-cloud | SaaS chaos platform |
| Toxiproxy | Network | Proxy with fault injection |
Game Days
Scheduled chaos experiments with the full team:
- Plan: Define experiment, expected outcome, rollback plan
- Execute: Run experiment while team observes
- Observe: Monitor dashboards, alerts, response time
- Discuss: What worked, what didn’t, what to improve
- Action items: Fix weaknesses found
Blast Radius Control
- Start in staging, graduate to production
- Start with single instance, expand gradually
- Use feature flags to control experiments
- Have automated rollback
- Time-box experiments
- Exclude critical paths initially
Interview Questions
Q: What is chaos engineering? A: Intentionally injecting failures into systems to find weaknesses before they cause real outages. The goal is to build confidence in the system’s ability to handle turbulent conditions. Netflix pioneered this with Chaos Monkey.
Q: How do you safely run chaos experiments in production? A: (1) Define steady-state metrics, (2) start with smallest blast radius, (3) have automated rollback, (4) time-box the experiment, (5) exclude critical paths initially, (6) run during low-traffic periods, (7) have the team watching dashboards.
Q: What is a Game Day? A: A scheduled chaos engineering exercise where the team practices responding to failures. Similar to fire drills. The team runs chaos experiments, observes system behavior, and identifies improvement opportunities. Builds muscle memory for incident response.