Circuit Breaking & Retry Budgets

Resilience middleware is the part of the request pipeline that decides what happens when an upstream is slow, saturated, or failing — and getting it wrong is how a single degraded backend becomes a full outage. A retry policy that looked prudent in isolation multiplies load at the exact moment the system can least absorb it; a circuit breaker sized for the average case never trips during the burst that actually matters. This layer sits alongside the rest of the middleware chain: after routing has chosen an upstream, resilience policy governs how the gateway forwards, how long it waits, whether it retries, and when it stops sending traffic to a host altogether. This page covers the four primitives that compose that behavior — timeouts, outlier detection, circuit breakers, and retry budgets — with runnable configuration for Envoy 1.32+, Kong 3.x, and NGINX Plus R32+, and the interactions between them that are the real source of production incidents.

Design invariants every resilience layer must satisfy:

• Retries are bounded by a budget expressed as a fraction of live traffic, not only by a per-request count — the total retry rate is what causes amplification, not any single request’s behavior.
• Only idempotent failure modes are retried (5xx, connect-failure, reset), and never a request that may have already mutated upstream state.
• per_try_timeout is strictly less than the global request timeout, sized so all attempts plus backoff fit inside the global budget.
• Backoff is exponential with jitter; a fixed retry interval synchronizes clients into coordinated retry waves.
• Circuit-breaker concurrency limits are set above expected peak concurrency so the breaker rejects overload rather than the connection pool silently exhausting first.
• Outlier detection ejects individual failing hosts before a full circuit opens, so one bad replica does not fail the whole cluster.

The mental model: four primitives, one control loop

Resilience at the gateway is not a single feature. It is four cooperating primitives, each answering a different question about a request to a degraded upstream:

1. Timeouts answer how long do we wait? A per-attempt timeout bounds one try; a global timeout bounds the whole request including retries. Without a per-try timeout, a single hung connection holds a worker and a pool slot indefinitely.
2. Outlier detection (passive health checking) answers which hosts are still healthy? It observes actual request outcomes and temporarily ejects a host from the load-balancing set after consecutive failures, letting the remaining healthy hosts absorb traffic.
3. Circuit breakers answer how much load may reach this upstream cluster at all? In Envoy these are concurrency ceilings — connections, pending requests, active requests, active retries — that reject overflow immediately rather than queueing it into collapse.
4. Retry budgets answer how much of our traffic may be retries right now? They cap the aggregate retry rate so a broad outage cannot turn one failure into a self-inflicted load multiplier.

The failure that ties these together is retry amplification: an upstream blips, every in-flight request retries, the retry traffic doubles offered load, the upstream — already struggling — falls further behind, and more requests fail and retry. This is the thundering herd, and it is why retries and breakers must be co-designed rather than tuned independently during an incident.

The clearest way to reason about circuit breaking specifically is as a three-state machine per upstream. Traffic flows while the circuit is closed; a failure threshold trips it open and traffic is short-circuited; after a cooldown it moves to half-open and admits a trickle of probe requests to test recovery.

Primary concept: composing retries, timeouts, and breakers

The single most important rule of this layer is that the primitives interact multiplicatively, and the interaction is what has to be configured — not each knob in isolation. A retry policy sits on top of a timeout budget, and both sit inside a circuit breaker’s concurrency ceiling.

Envoy 1.32+ — the reference model

Envoy separates the concerns cleanly, which makes it the clearest template even if you deploy a different gateway. circuit_breakers live on the upstream cluster and bound concurrency; outlier_detection lives on the upstream cluster and ejects hosts; retry_policy lives on the route and governs re-attempts with an aggregate budget.

# Envoy 1.32+ — full resilience stack on one upstream cluster
clusters:
  - name: payments_cluster
    connect_timeout: 0.25s
    lb_policy: LEAST_REQUEST
    # Concurrency ceilings: reject overload rather than queue it into collapse
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          max_connections: 1024      # size ABOVE peak concurrency, not average
          max_pending_requests: 256  # short queue; overflow returns 503 immediately
          max_requests: 1024         # active requests cap (HTTP/2 multiplexed)
          max_retries: 32            # cap on concurrent retries fleet-wide
    # Passive health: eject an individual failing host from the LB set
    outlier_detection:
      consecutive_5xx: 5
      consecutive_gateway_failure: 5   # count resets/timeouts, not just 5xx
      interval: 5s
      base_ejection_time: 30s
      max_ejection_percent: 50         # never eject more than half the hosts
      enforcing_consecutive_5xx: 100

route_config:
  virtual_hosts:
    - name: payments
      domains: ["*"]
      routes:
        - match: { prefix: "/v2/payments" }
          route:
            cluster: payments_cluster
            timeout: 1s                  # GLOBAL budget: whole request incl. retries
            retry_policy:
              retry_on: "5xx,reset,connect-failure"
              num_retries: 2
              per_try_timeout: 0.25s     # MUST be < global timeout
              retry_back_off:
                base_interval: 0.025s    # exponential: 25ms, 50ms, 100ms...
                max_interval: 0.25s
              # Aggregate cap: retries can never exceed 20% of active requests
              retry_budget:
                budget_percent: 20.0
                min_retry_concurrency: 3

Read that config as a budget hierarchy. The global timeout: 1s is the ceiling for the whole request. Inside it, per_try_timeout: 0.25s with num_retries: 2 means up to three attempts plus backoff must fit under one second — they do. The retry_budget guarantees that across every concurrent request, retry traffic never exceeds 20% of live load, so a broad outage cannot amplify offered load beyond 1.2x. outlier_detection runs orthogonally: while all this happens, hosts that return five consecutive failures are ejected for 30 seconds so healthy replicas carry the load.

Kong 3.x — passive health checks plus bounded retries

Kong expresses the same intent through the upstream object’s health checker and the service-level retries counter. Kong’s passive (circuit-breaker-style) health checking demotes an unhealthy target out of the balancer, which is the functional analog of outlier detection.

# Kong 3.x declarative — passive health checks + bounded retries
_format_version: "3.0"

upstreams:
  - name: payments-upstream
    # Kong retries are the balancer's re-pick count on a failed try
    healthchecks:
      passive:
        type: http
        healthy:
          successes: 3
          http_statuses: [200, 201, 202, 204]
        unhealthy:
          http_failures: 5          # eject target after 5 failed responses
          tcp_failures: 3
          timeouts: 3               # count timeouts, not only HTTP errors
          http_statuses: [500, 502, 503, 504]
      active:
        type: http
        http_path: /health
        healthy:   { interval: 5, successes: 2 }
        unhealthy: { interval: 5, http_failures: 2, timeouts: 2 }
    targets:
      - target: payments-a.internal:8080
        weight: 100
      - target: payments-b.internal:8080
        weight: 100

services:
  - name: payments-api
    host: payments-upstream        # points at the upstream, not a raw URL
    connect_timeout: 250           # per-connection attempt (ms)
    write_timeout: 1000
    read_timeout: 1000
    retries: 2                     # balancer re-picks a healthy target twice
    routes:
      - name: payments-v2
        paths: ["/v2/payments"]
        strip_path: true

Kong’s retries is subtly different from Envoy’s: it is a re-pick within the load balancer, so a retry goes to a different healthy target rather than the same one — which is exactly what you want during a partial outage. The critical gotcha is that Kong retries on any failed connection attempt including transient errors, and it has no aggregate budget, so a low retries value (2, not the default 5) is your only amplification guard. Pair it with tight timeouts: a 250 ms connect_timeout means a dead host is abandoned fast rather than holding a worker.

For counter-based state shared across data-plane replicas — the same problem rate limiting and throttling solves with a Redis backend — Kong’s passive health state is per-node by default. In a multi-node fleet each Kong node learns about an unhealthy target independently, which slows convergence; enable Kong’s cluster events propagation so an ejection on one node informs the others.

NGINX Plus R32+ — upstream health checks and proxy_next_upstream

NGINX Plus expresses the same shape with max_fails/fail_timeout for passive ejection, health_check for active probing, and proxy_next_upstream for bounded retries.

# NGINX Plus R32+ — passive + active health checks with bounded retries
upstream payments_pool {
  zone payments_zone 64k;
  server payments-a.internal:8080 max_fails=5 fail_timeout=30s;
  server payments-b.internal:8080 max_fails=5 fail_timeout=30s;
  least_conn;
  keepalive 32;
}

server {
  listen 443 ssl;

  location /v2/payments {
    proxy_pass http://payments_pool;

    proxy_connect_timeout 250ms;
    proxy_read_timeout    1s;

    # Retry only idempotent, safe failure conditions on the NEXT upstream
    proxy_next_upstream          error timeout http_502 http_503 http_504;
    proxy_next_upstream_tries    2;      # hard cap on re-picks
    proxy_next_upstream_timeout  1s;     # global budget across all tries

    # Active health check ejects a target before passive counting catches up
    health_check interval=5s fails=2 passes=3 uri=/health match=payments_ok;
  }
}

match payments_ok {
  status 200;
}

The proxy_next_upstream_tries and proxy_next_upstream_timeout pair is NGINX’s equivalent of num_retries plus a global timeout: the tries cap bounds count, the timeout caps total elapsed time across re-picks. Omit either and a slow upstream can walk the entire pool one 30-second timeout at a time.

Secondary concept: the knobs that decide whether the breaker actually fires

The default configurations of all three gateways will pass a smoke test and fail a real incident. Three advanced behaviors decide the difference.

Counting the right failures. A breaker or health checker that only counts HTTP 5xx is blind to the failure modes that dominate real outages: connection resets, connect failures, and timeouts. Envoy’s consecutive_5xx ignores a host that hangs and never returns a status — you must add consecutive_gateway_failure to catch resets and timeouts. Kong’s passive check needs timeouts and tcp_failures set alongside http_failures. NGINX’s proxy_next_upstream must list timeout explicitly.

Ejection ceilings prevent self-inflicted outages. If outlier detection can eject an unlimited fraction of a single cluster, a correlated failure — a bad deploy that makes every host return 500 — ejects the entire cluster and you have manufactured a total outage from a degradation. Envoy’s max_ejection_percent: 50 caps this; NGINX and Kong need enough targets that max_fails cannot demote all of them at once. Always leave a floor of healthy capacity.

Enforcing percentage vs. detection. Envoy distinguishes detecting an outlier from enforcing ejection: enforcing_consecutive_5xx is the probability (0–100) that a detected outlier is actually ejected. Set it to 100 in production; leaving it low is a common reason a host that trips the detector keeps serving traffic. This split lets you run detection in “monitor only” mode (enforcing_*: 0) to tune thresholds before enabling enforcement — do that on a new cluster before trusting the numbers.

Under sustained load, all of this interacts with capacity: a breaker that opens sheds load onto the remaining hosts, which raises their concurrency, which can trip their breakers next. Sizing the pool and the concurrency ceilings so the surviving hosts can absorb a partial ejection is a capacity planning exercise, and in multi-region deployments it is inseparable from high-availability topology design — where you want failover to a healthy region rather than cascading ejection within a failing one.

Comparative implementation

The pattern across all four: only Envoy gives you a real aggregate retry budget. On Kong, NGINX, and Tyk you approximate it by keeping the per-request retry count low (1–2) and leaning harder on host ejection so retries land on healthy targets rather than re-hammering a failing one.

Operational gotchas

The breaker that never opens because the pool exhausts first. If max_pending_requests and max_connections are too low, requests fail with pool-overflow errors before the failure detector sees enough 5xx samples to open the circuit. The breaker is armed but never fires. Fix: size concurrency ceilings above peak, and count gateway failures (timeouts, resets) not just HTTP 5xx.

Retry amplification from per-request-only limits. Setting num_retries without a retry_budget (Envoy) or setting Kong retries to the default 5 means a broad outage lets every request retry at once. Offered load jumps 2–6x precisely when the upstream is weakest. Fix: add a retry_budget, or cap per-request retries at 1–2, and always add jitter to backoff. This is covered in depth in tuning retry budgets to prevent thundering herd.

Per-try timeout ≥ global timeout, so retries are dead. If per_try_timeout equals or exceeds the route timeout, the first slow attempt burns the whole budget and no retry ever runs. The policy looks configured but is inert. Fix: per_try_timeout × (num_retries + 1) plus backoff must be less than the global timeout.

Retrying non-idempotent requests. Retrying a POST that already reached the upstream can double-charge a payment or double-create a resource. retry_on including reset will retry a request that may have been fully processed before the connection dropped. Fix: retry only idempotent methods, or require an idempotency key the upstream deduplicates on.

Full-cluster ejection from a correlated fault. A bad deploy makes every host fail; outlier detection ejects them all; the whole cluster is down. Fix: set max_ejection_percent (Envoy) and keep enough targets that max_fails cannot demote the whole pool.

Fixed backoff synchronizing clients. A constant retry interval makes all failed clients retry in lockstep, producing periodic load spikes. Fix: exponential backoff with jitter, never a fixed sleep.

Production Configuration Checklist

• Every retry policy has an aggregate budget (retry_budget) or a per-request cap of no more than 2 retries.
• retry_on includes only idempotent failure modes; non-idempotent methods are excluded or protected by an idempotency key.
• per_try_timeout is strictly less than the global route timeout, and all attempts plus backoff fit inside it.
• Backoff is exponential with jitter — no fixed retry interval anywhere in the fleet.
• Circuit-breaker concurrency ceilings (max_connections, max_pending_requests) are sized above peak concurrency, not average.
• Failure detection counts timeouts, resets, and connect failures — not only HTTP 5xx.
• Outlier detection / passive health checks have an ejection ceiling (max_ejection_percent) so a correlated fault cannot eject the whole cluster.
• enforcing_consecutive_5xx (or equivalent) is at 100 in production, not left in detect-only mode.
• Health/circuit state converges across data-plane replicas (Envoy per-host, Kong cluster events, Tyk external store).
• Breaker and retry behavior is validated with fault injection (tc netem delay, forced 503s) under representative load before production.
• Metrics for ejections, breaker-open events, and retry rate are dashboarded and alerted, not just logged.

FAQ

What is the difference between circuit breaking and outlier detection in Envoy?

Envoy circuit breakers are concurrency limits: they cap max_connections, max_pending_requests, max_requests, and max_retries per cluster, and reject overflow immediately with a 503 to protect the upstream from overload. Outlier detection is passive health checking: it observes real request results and ejects individual failing hosts from the load-balancing set for a base_ejection_time. Circuit breakers bound how much load reaches a given cluster; outlier detection decides which hosts inside that cluster still receive traffic. Production resilience needs both configured together.

Why does my circuit breaker never open under load?

The most common cause is that the connection pool or pending-request queue saturates before the failure detector accumulates enough samples. Requests fail with connection or queue-overflow errors rather than the 5xx responses the breaker counts, so the failure ratio never crosses the threshold. Size max_connections and max_pending_requests above expected concurrency, count timeouts and resets as failures with consecutive_gateway_failure, and lower the sample window so the detector reacts before the pool exhausts.

What is a retry budget and how is it different from num_retries?

num_retries caps retries for a single request; a retry budget caps retries across all concurrent requests as a percentage of active traffic. With only num_retries set, a broad upstream outage lets every in-flight request retry simultaneously, multiplying load 2–3x at the worst possible moment. Envoy retry_budget with budget_percent: 20 guarantees retries never exceed 20 percent of active requests regardless of how many individual requests want to retry, which is the primary defense against retry-driven thundering herd.

Should per-try timeout be shorter than the global route timeout?

Yes. per_try_timeout bounds a single attempt while the global timeout bounds the whole request including all retries. If per_try_timeout is greater than or equal to the global timeout, the first slow attempt consumes the entire budget and no retry ever runs, so the retry policy is silently dead. Set per_try_timeout so that num_retries attempts plus backoff fit inside the global timeout — for example a 0.25s per-try with two retries under a 1s global timeout.

Related

• Tuning Retry Budgets to Prevent Thundering Herd — step-by-step budget sizing, jitter, and load validation for the amplification failure mode.
• Rate Limiting & Throttling Strategies — shed load before it reaches a failing upstream, and share counter state across replicas the way health state must be shared.
• Scaling Limits & Capacity Planning — sizing pools and concurrency ceilings so surviving hosts absorb a partial ejection.
• High-Availability Topologies — regional failover as the alternative to cascading ejection inside one degraded cluster.
• Middleware Chains & Request Transformation — where resilience policy sits in the full request pipeline.

← Middleware Chains & Request Transformation