Follow-up Tree: Circuit Breakers

Overview

“What is a circuit breaker, and what does it protect that a timeout does not?” is the reliability interview’s sharpest escalation, because the honest answer — aggregate resource protection and fast failure — forces the candidate to distinguish three controls most answers blur together: timeouts (duration), bulkheads (concurrency), and breakers (invocation itself). The ladder escalates through the questions that separate textbook answers from operated systems: the state machine (closed, open, half-open) and what each state actually does to callers; what counts as a “failure” and how thresholds are counted over time rather than in a row; fallbacks and their silent-corruption danger (an open breaker serving stale data is a product decision disguised as a log line); and the interaction with retries — where retry storms and breakers amplify each other unless budgets and jitter are designed in. Each rung ties back to a chain that owns its foundation: the resource arithmetic that makes breakers necessary is Connection Pooling, the degradation ladder the fallback serves is Graceful Degradation, and the canonical book treatment is the Circuit Breakers chapter. This page owns the breaker itself: the state machine, the counting, the fallback contract, and the arithmetic of retries meeting open circuits in production.

The Opening Question

“What is a circuit breaker pattern, and what problem does it solve that a timeout doesn’t?”

The expected core answer: a circuit breaker is a per-dependency state machine wrapping calls to a downstream service — closed (calls pass; failures are counted), open (calls fail immediately without touching the downstream), half-open (a limited probe tests recovery). It solves what a timeout cannot: a timeout bounds each call’s duration, but under a dependency outage the service still makes every call, pays the timeout wait on each, and holds threads, connections, and memory while waiting — aggregate exhaustion by calls that are all going to fail. The breaker cuts invocation itself: once failures cross the threshold, subsequent calls fail in microseconds at the breaker, the downstream gets breathing room to recover, and the caller’s resources are freed for requests that can actually succeed. The pattern was popularized by Nygard’s Release It!; the Microsoft architecture pattern page is the standard web reference, and the Circuit Breakers chapter treats the implementation in depth.

Why this answer is correct: it states the unit of protection — invocation and aggregate resources, not individual call duration — and names all three states with their caller-visible behavior. The predictable follow-up is “how does it decide to open?”, and a candidate who already said “failures are counted” has put the counting question on the table, which is exactly where the ladder goes.

Level 2: Closed, Open, Half-Open — The State Machine and What It Promises Callers

What it tests: whether the states are a memorized diagram or a contract with observable behavior — what callers experience in each state, who transitions when, and why half-open exists at all.

Weak answer: “It has three states: normal, tripped, and testing” — vocabulary without contract: no model of what a call does in the open state (fails how fast? with what error? is it recorded?), no probe discipline, no answer for who decides recovery.

Strong answer: walks each state as behavior. Closed: calls pass through; the breaker records outcomes (success, timeout, 5xx, rejection) into a window; no added latency in the happy path — the pattern’s cost is a counter, not a hop. Open: calls fail fast with an error the caller can distinguish (“circuit open,” not a downstream 500); no network I/O occurs; this is the state that returns threads and connections to the pool instantly, converting a 5-second timeout economy into a microsecond one. Half-open: after a cooldown, the breaker admits a bounded number of probe calls (often exactly one) while rejecting the rest; probes succeed → close (reset the window); probes fail → reopen. The half-open discipline is the subtle part: without bounding probes, recovery testing is another thundering herd — every caller’s next request becomes a probe against a recovering service, which is why proper implementations serialize probes and hold other calls on the open-state failure path until the verdict.

Escalation trigger: “Your breaker opens on a dependency blip, closes 30 seconds later, then reopens — repeating for an hour. Diagnose the state machine.”

Answer. The flapping signature has three classic causes, each diagnosable from the transition log. Cause one: cooldown too short relative to the dependency’s actual recovery time — the probe fires while the downstream is still sick, fails, reopens; the fix is tuning cooldown to measured recovery time, not a default. Cause two: probe success does not predict sustained capacity — one probe succeeds (the dependency can serve one request) and the breaker closes, dumping the full queue onto a service that was overloaded to begin with; the fix is graduated re-admission (ramp allowed traffic over seconds) rather than a binary close. Cause three, the sneakiest: the failure counter is polluted by errors the breaker should not count (client 4xx, business rejections, timeouts from an unrelated slow dependency) — the window says “failures” but means “a mixed bag,” so the breaker opens on a dependency that is fine. The diagnosis discipline: log every transition with the window contents (counts by error class), because the transition history is the answer — the observability requirements of breakers are the debugging surface for exactly this incident. The senior framing: the state machine is a distributed failure detector, and every detector has false positives; the design question is not “can it be wrong” but “what does being wrong cost, and is that visible.”

Level 3: What Counts as a Failure — Thresholds, Windows, and Slow Calls

What it tests: whether you can design the counting: which outcomes count, over what window, at what rate — and articulate the failure modes of naive thresholds (consecutive counts, no slow-call detection).

Weak answer: “After 3 failures it opens” — the consecutive-count toy model, which a single alternating success resets forever, and which ignores that the dangerous failure is slowness, not exceptions.

Strong answer: specifies a windowed, rate-based policy: failures are counted over a sliding window (by volume or time) and the breaker opens when the failure rate crosses a threshold with a minimum sample size (avoiding “1 of 2 calls failed = 50 percent”); failures include timeouts, connection errors, and 5xx-class responses but exclude 4xx (client errors are the caller’s bug, and counting them opens circuits over your own defects); and — the rung’s differentiator — slow calls count as failures via a latency budget, because the overload signature is latency inflation, not crashes: a dependency answering at 9 s is more damaging than one answering with 500s, and a breaker that only opens on exceptions protects against nothing the timeouts were not already doing. Knows the tuning failure modes both ways: thresholds too tight cause flapping; too loose and the breaker is a spectator to the outage it was purchased for.

Escalation trigger: “Your breaker counts failures, but your dependency fails by getting slow — 4-second p99, zero errors. The breaker never opens. Why, and what do you change?”

Answer. The breaker watches the wrong signal: it counts exceptions and 5xx, and a dependency that degrades by latency alone never produces them — every call “succeeds” at 4 seconds, the timeout sits above the degradation band, so the breaker stays closed while the aggregate damage (threads parked, pools consumed) is exactly what it was installed to prevent. The changes, in order: add slow-call detection — define a latency budget per dependency (a call slower than budget counts as a failure in the window) and open on slow-call ratio; pair it with the pool arithmetic — the number of concurrent calls the dependency absorbs before latency inflates is measurable (Connection Pooling owns the Little’s-Law sizing), and the breaker’s concurrency limit should sit near that number, not at infinity; and make the timeout itself a counted failure (a timed-out call is a failure regardless of whether the downstream eventually answered — the Timeouts and Retries audit covers the assumption stack). The transferable lesson: “failure” is a definition, and the definition must include the failure mode the dependency actually exhibits — discovered by looking at its latency distribution under load, not by waiting for exceptions. A breaker tuned to the wrong definition is a smoke detector with the battery removed.

Level 4: Fallbacks — What the Open Circuit Serves, and What It Must Never Serve

What it tests: whether you treat the fallback as a product-and-correctness contract rather than a try/catch returning null — including the observability trap of silent degradation.

Weak answer: “When the circuit is open, return a default value” — no taxonomy of safe defaults, no staleness analysis, no monitoring story; the fallback becomes an unlogged correctness hazard.

Strong answer: enumerates the fallback classes and their contracts: cached/stale data (fine for browsable content, dangerous for prices and inventory — the staleness-tolerance audit in Graceful Degradation applies field by field); static defaults and empty results (safe where absence is a legal state — “no recommendations available” — a lie where it is not — “zero items in cart”); functional alternatives (a simpler downstream, a different provider); and no fallback — failing the request loudly, the correct answer for invariants: a payment capture with no fallback should fail visibly rather than “succeed” with a guessed amount. Insists on the observability contract: every fallback served is a metric, alertable (fallback rate per dependency), and user-visible where the business promises accuracy — because the failure mode of mature breaker deployments is silent partial correctness: nothing pages, revenue quietly degrades, and the open circuit hides in dashboards full of green.

Escalation trigger: “Your breaker opened on the pricing service during an incident. The fallback serves yesterday’s cached prices. Nothing paged for six hours. Walk through everything that went wrong and the design changes.”

Answer. The failure chain: the breaker did its job (stopped calling a failing dependency), the fallback violated its contract (served transaction-adjacent data — prices — beyond any staleness bound), and observability had no signal for any of it (fallback serving was a debug log line, not a metric with an alert). Three design changes, one per link. Contract: classify the dependency’s data before choosing the fallback — prices are invariant-adjacent, so the fallback is fail the read (checkout blocks with an explicit error) or serve stale with explicit labeling and a hard age bound, never silent substitution; cached prices belong to non-transactional surfaces (product browsing) only. Observability: fallback-served becomes a first-class metric with a rate alert — “pricing fallback rate > 2 percent for 5 minutes” pages, because a silent open circuit on a revenue path is an incident even when latency graphs look pristine. Review: every fallback gets an owner’s decision at design time, not an engineer’s reflex at incident time — the runbook states which fallback each dependency serves and what user-visible promise it makes. The senior framing: the breaker’s fallback is where reliability engineering and product correctness meet, and the pattern’s dangers are all silent ones — the opposite of the loud failures that motivated the pattern. This is the graceful degradation decision space seen from the call site: the breaker is the mechanism, the degradation ladder is the policy, and wiring mechanism to policy explicitly is what separates the two.

Level 5: Retry vs Breaker Interaction, and the Judgment Rung

What it tests: the system-level arithmetic: retries multiplying failures through open circuits, retry budgets, jitter, placement — plus the judgment about where breakers do not belong.

Weak answer: “Add retries for reliability, add breakers for protection” — the two patterns stapled together, which in an outage multiply each other: every caller retries every failure against a dependency that needs silence, turning an incident into an amplification loop.

Strong answer: derives the interaction arithmetic: with R retries per request and a downstream failing at rate f, observed load multiplies by roughly (1 + R × f) — under near-total failure with 2 retries, a dependency gets ~3x the load it cannot serve, which is why outages deepen before they recover. The controls: jittered exponential backoff (spread the herd; the reconnect-storm arithmetic in WebSockets is the same math), a retry budget (cap retries as a percentage of live traffic, so retry volume collapses as failure rate rises), idempotency as the precondition for any retry (the Message Queues chain’s at-least-once discipline applies to HTTP retries identically), and retry placement — retries belong at one layer (typically the edge of the call tree), never stacked at every hop, because stacked retries multiply (3 × 3 across two hops = up to 9 leaf calls per user request). Then the judgment: breakers belong on remote, fallible, non-critical dependencies where a fallback or fast failure is acceptable; they are wrong where a fallback would be a correctness lie (double-entry ledger writes), and they are a band-aid where the real fix is capacity or an async boundary. Placement is per-dependency, not global — one shared breaker defeats the isolation that motivated the pattern, and per-dependency partition is the Bulkheads chapter’s core claim.

Escalation trigger: “During the pricing incident, retry storms made recovery impossible: the dependency recovered for 20 seconds, then collapsed again under pent-up retries. Design the retry/recovery path so it doesn’t fight the breaker.”

Answer. The collapse-recover-collapse loop is retry amplification meeting eager recovery: while the dependency was down, every caller accumulated retryable work; the moment it steadied, the pent-up cohort arrived simultaneously, re-saturated it, and re-tripped the breaker. The design that ends the fight: retry budgets at every retrying layer — retries capped as a small percentage of current traffic, so as the dependency fails, total retry volume shrinks rather than compounding; decorrelated jitter on every backoff so the pent-up cohort arrives spread over seconds, not in one instant; the breaker’s close transition graduated (probe, then 10 percent, then 50 percent, then full) so admitted load matches demonstrated recovery — the binary close is what handed the recovering dependency its second outage; coordination at the caller of last resort: the edge applies its own shedding (Graceful Degradation) so user traffic cannot re-amplify what internal layers have damped; and the incident runbook step everyone forgets — throttle or pause non-critical producers (the Background Jobs fleet) during dependency incidents, because queued work draining at full rate is retry load by another name. The judgment content to close: the breaker and the retry are opposing forces — one stops invoking, the other insists — and the design work is making them compose: retries with budgets and jitter respect the breaker’s signal, and the breaker’s graduated close respects the retry backlog’s existence. Systems fail in exactly the seam where these two were designed independently.

How to Approach Unfamiliar Variations

1. Name the unit of protection first. Duration (timeout), concurrency (bulkhead), or invocation (breaker) — every unfamiliar variant (rate-based breakers, adaptive concurrency limiters, mesh outlier ejection) is one of these units with a policy attached. Identifying the unit tells you what the variant will and will not protect.
2. Ask how the detector defines failure. Any breaker-like mechanism is only as good as its failure definition: does it count slow calls, 4xx, its own timeouts? The definition determines whether it opens during the outage that motivates it.
3. Trace retries through every failure path. For any reliability variation, draw the failure case with retries at each layer and compute the worst-case call multiplier. The product of retry stacks is where outages deepen; budgets and jitter are the levers.
4. Apply the fallback invariant test. Whatever the variation serves when a dependency is unavailable, ask: if this answer is wrong, what breaks? Sort the fallback into fail, static, stale-with-label, or substitute — and require the observability signal for whichever is chosen.

Interview Questions

Q: Why does the half-open state admit only a limited probe instead of all queued traffic? A: Because recovery testing under full load is a thundering herd: a dependency that just recovered can serve a little, and admitting everything re-saturates it instantly, re-tripping the breaker — a loop. The bounded probe (often one concurrent call) gets a clean recovery signal while the rest stay on the fast-fail path; production designs then ramp admitted traffic gradually after the probe succeeds, matching admitted load to demonstrated recovery.

Q: Your circuit breaker tracks consecutive failures and opens at 5. In a flapping dependency that alternates success/failure, what happens — and what is the fix? A: The alternating success resets the counter before it reaches 5, so the breaker never opens while the dependency fails at a 50 percent rate — the worst of both worlds: half of all calls fail slowly through the network path, and no fast-fail protection ever engages. The fix is windowed rate-based counting with a minimum sample size: failures over a sliding window, opening on ratio rather than streak. Consecutive counting is the toy version of the idea.

Q: Where should a circuit breaker live — in every calling service, at the edge gateway, or in the service mesh? Defend. A: At the caller of the dependency, because the breaker’s resource protection is local: it releases this process’s threads, connections, and latency budget, and no remote layer can do that for it. The mesh or gateway adds coarse global protection (outlier ejection, edge shedding) as a second layer, but a service relying solely on the mesh still parks its own threads against dead dependencies — the resource arithmetic is per-process. Layered answer: local breakers for resource protection, mesh ejection for fleet-wide bad instances, edge control for user-facing posture.

Q: A reviewer rejects your PR: “retries with backoff make the breaker redundant.” Both-ways argument, please. A: Neither replaces the other. Retries with backoff and budgets improve the individual call’s chance across transient blips, but during sustained failure they amplify load (each request becomes several) and keep threads parked per attempt — no protection of caller resources. The breaker stops invocation on sustained failure and frees resources instantly, but without retries it converts every transient blip into user-visible failure. Together — retry for transients, breaker for sustained failure, budget and jitter so they do not fight — each covers the other’s gap. The reviewer’s claim holds only in the degenerate case where all failures are sustained.

Key Takeaways

• A timeout bounds one call’s duration; a breaker stops making calls at all — the protection is aggregate: threads, connections, and latency budget returned to the caller the moment the circuit opens.
• The state machine is a failure detector with a caller-facing contract: closed counts, open fails fast, half-open probes with a bound — and every transition must be logged with window contents, because the log is the debugging surface.
• “Failure” is a definition that must include slow calls: a dependency that degrades by latency alone never trips an exception-counting breaker.
• Fallbacks are product decisions with staleness bounds: silent substitution of transaction-adjacent data is a correctness incident, and fallback-served must be a metric with an alert, not a debug log.
• Retries and breakers are opposing forces — retries amplify, breakers suppress — and they compose only with jittered backoff, retry budgets, idempotency, and graduated re-admission after the circuit closes.

Cross-References

• Graceful Degradation — the sibling chain that owns the policy the breaker’s fallback serves: shedding, bulkheads, the degradation ladder.
• Connection Pooling — the resource arithmetic (Little’s Law, pool sizing) that determines where breaker concurrency limits belong.
• Load Balancing — health-check removal versus breaker invocation: two detectors with different vantage points and lag.
• Message Queues — the at-least-once and idempotency discipline that makes retries safe anywhere, including here.
• Background Jobs — the queued-work fleet whose drain rate becomes retry load during dependency incidents.
• API Gateway — the edge layer’s own breaker/shedding posture and its interaction with per-service breakers.
• Backpressure — the flow-control formulation of the same problem: what a slow consumer does to its producer’s resources.
• Timeouts and Retries — the assumption audit for every timeout and retry the breaker wraps.
• Circuit Breakers (Microservices) — the book chapter: the pattern in the microservices context, implementation-level detail.
• Bulkheads — the resource-partition pattern the breaker completes: concurrency limits plus invocation limits.
• Reliability Patterns — the catalog view: where breakers sit among timeouts, load shedding, and redundancy.
• Feature Flags — the mechanism for the fallback posture switches that degraded modes need.
• AWS S3, us-east-1, 2017 — the recovery-load case: why returning capacity needs graduated re-admission, not a switch.
• Circuit Breaker pattern (Microsoft Azure Architecture) — the standard reference sketch of the state machine and implementation notes (verified this session).
• Release It! (Michael Nygard) — the book that named the pattern, cited by title; the Microsoft pattern page is the stable web reference.