Tuning Retry Budgets to Prevent Thundering Herd

A retry policy is a load multiplier waiting for a trigger. When an upstream is healthy, retries are rare and invisible; when it blips, every in-flight request can decide to retry at the same instant, and the gateway hands the already-struggling upstream two or three times its normal load precisely when it can least absorb it. That is the thundering herd, and it turns a five-second transient into a multi-minute outage. This page is a concrete procedure for sizing retry budgets, adding jitter, capping total attempts, and proving under load that a blip stays a blip — with production config for Envoy 1.32+ and Kong 3.x.

Prerequisite concepts

This assumes you understand the four resilience primitives and how they interact, covered in circuit breaking and retry budgets — specifically why per_try_timeout must sit under the global timeout and why breakers and retries are co-designed. It also assumes familiarity with where resilience policy sits in the broader middleware chain, after routing has selected an upstream. If your goal is to shed excess load before it ever reaches the retry layer, pair this with rate limiting and throttling strategies.

Why per-request retry caps are not enough

The intuition that “at most 2 retries per request” bounds the damage is wrong at the fleet level. Consider a route serving 10,000 RPS with num_retries: 2. Under normal conditions the upstream succeeds on the first try and retries are near zero. Now the upstream returns 503 for a 10-second window. Every one of those 10,000 requests per second exhausts its retry allowance — the gateway sends the original plus two retries, offering 30,000 RPS to an upstream that just proved it cannot handle 10,000. The per-request cap held perfectly and the system still tripled its own load.

The fix is a cap on the aggregate retry rate, expressed as a fraction of active traffic. That is what a retry budget is: no matter how many individual requests want to retry, the total retry traffic is bounded to, say, 20% of live load — so worst-case offered load is 1.2x, not 3x.

Step-by-step: sizing the budget

1. Establish the baseline and headroom. Measure steady-state upstream RPS and its proven ceiling from capacity planning. The retry budget must fit inside the headroom between them. If the upstream runs at 70% of ceiling, a 20% retry budget briefly pushes it to ~84% — acceptable. If it runs at 90%, use 10%.

2. Set the aggregate budget, not just the count. In Envoy, retry_budget.budget_percent is the aggregate cap and min_retry_concurrency is the floor for low-traffic routes.

# Envoy 1.32+ — retry budget with jittered exponential backoff
route_config:
  virtual_hosts:
    - name: orders
      domains: ["*"]
      routes:
        - match: { prefix: "/v2/orders" }
          route:
            cluster: orders_cluster
            timeout: 1s                    # global budget for the whole request
            retry_policy:
              retry_on: "5xx,reset,connect-failure"
              num_retries: 2               # per-request ceiling
              per_try_timeout: 0.25s       # < global timeout; leaves room for retries
              retry_back_off:
                base_interval: 0.025s      # jittered: wait ∈ [0, min(25ms·2^n, 250ms)]
                max_interval: 0.25s
              retry_budget:
                budget_percent: 20.0       # retries ≤ 20% of active requests, fleet-wide
                min_retry_concurrency: 3   # allow ≥3 concurrent retries on quiet routes

Envoy applies full jitter automatically: the wait before retry n is a uniform random value between 0 and min(base_interval · 2ⁿ, max_interval). You never configure jitter explicitly — you only bound the exponential envelope with base_interval and max_interval. This is what desynchronizes clients so they do not retry in lockstep.

3. Cap total attempts against the global timeout. With per_try_timeout: 0.25s and num_retries: 2, worst case is three attempts of 250 ms plus backoff — comfortably under the 1s global timeout. If you raised num_retries to 5, the attempts alone would need 1.5s and the retries past the second would be silently unreachable. Always check that (num_retries + 1) × per_try_timeout plus max backoff fits under the global timeout.

4. On Kong, approximate the budget with a low count and fast ejection. Kong 3.x has no retry_budget, so the guard is a low retries plus aggressive health checks so a retry lands on a different healthy target.

# Kong 3.x — no aggregate budget, so keep retries low + eject fast
_format_version: "3.0"
upstreams:
  - name: orders-upstream
    healthchecks:
      passive:
        unhealthy:
          http_failures: 3       # eject a bad target quickly
          timeouts: 3
          http_statuses: [500, 502, 503, 504]
        healthy:
          successes: 3
    targets:
      - target: orders-a.internal:8080
        weight: 100
      - target: orders-b.internal:8080
        weight: 100
services:
  - name: orders-api
    host: orders-upstream
    connect_timeout: 250          # abandon a dead host fast (ms)
    read_timeout: 1000
    retries: 1                    # NOT the default 5 — the only amplification guard
    routes:
      - name: orders-v2
        paths: ["/v2/orders"]

With retries: 1 the worst-case multiplier is 2x, and because passive health checks eject the failing target after three failures, the single retry quickly starts landing on the healthy target rather than re-hitting the failing one — which is the closest Kong gets to a budget.

Decision matrix

Gotchas and failure signals

Budget rounds to zero on quiet routes. At 5 RPS, 20% of active requests may round below one, so retries never fire when you need them. min_retry_concurrency: 3 is the floor that prevents this — verify it is set on low-traffic routes.

retry_on: reset retrying a processed request. A connection reset can happen after the upstream fully handled the request. Retrying it double-applies the side effect. For non-idempotent methods, drop reset or gate retries behind an idempotency key the upstream deduplicates on.

Fixed backoff defeats the budget. A budget caps rate but not timing. If backoff is a constant sleep, the capped retries still arrive in a synchronized wave. Envoy’s automatic full jitter handles this; on any client-side or Kong-adjacent retry loop, implement decorrelated jitter (sleep = random(base, previous·3)) rather than a fixed interval.

The signal to watch: during a fault, plot upstream received RPS ÷ client sent RPS. A correct budget holds this at ≤ 1 + budget_percent. Envoy exposes upstream_rq_retry, upstream_rq_retry_success, and upstream_rq_retry_limit_exceeded — a rising retry_limit_exceeded means the budget is actively shedding retries, which is the budget working, not a bug.

Validation

• Fault injection (tc netem delay, forced 503 for 10–20s) run under representative steady RPS.
• Measured offered-load multiplier stays at or below 1 + budget_percent (≈1.2x at 20%).
• min_retry_concurrency verified to allow retries on low-traffic routes.
• (num_retries + 1) × per_try_timeout + max backoff confirmed less than the global timeout.
• Backoff jitter confirmed active — retries do not arrive in a synchronized wave.
• Non-idempotent routes have retries disabled or an idempotency key enforced.
• Retry-rate metrics (upstream_rq_retry, upstream_rq_retry_limit_exceeded) dashboarded and alerted.

FAQ

What budget_percent should I set for an Envoy retry budget?

Start at 20 percent, which is Envoy’s default and a good production baseline: retries can never exceed one fifth of active requests, capping worst-case amplification at 1.2x offered load. Lower it to 10 percent for fragile upstreams with little headroom, or raise it toward 25–30 percent only for upstreams with proven spare capacity. Always keep min_retry_concurrency (default 3) so low-traffic routes can still retry a few requests even when the percentage rounds down to zero.

How much jitter should I add to retry backoff?

Envoy applies full jitter automatically: each backoff is a random value between 0 and the current exponential ceiling, so you only configure base_interval and max_interval. The effective wait for attempt n is a uniform random pick up to min(base_interval · 2ⁿ, max_interval). For clients or gateways without built-in jitter, implement decorrelated jitter — sleep = random_between(base, previous_sleep · 3) — which spreads retries more widely and avoids synchronized waves better than fixed or equal jitter.

Kong has no retry budget field. How do I prevent amplification?

Kong’s retries value is a per-request re-pick count with no aggregate budget, so set it low: 1 or 2, never the default 5. Combine it with aggressive passive health checks so a retry lands on a healthy target rather than re-hitting the failing one, and tight connect_timeout and read_timeout so a dead host is abandoned in hundreds of milliseconds. For a true aggregate cap, front Kong with a rate limiter or move the retry-sensitive path to Envoy where retry_budget exists.

How do I verify a retry budget actually prevents thundering herd?

Inject a controlled upstream fault under representative load and watch the offered-load multiplier. Make the upstream return 503 or add latency for 10–20 seconds while a load generator holds steady RPS, and measure upstream received RPS versus client sent RPS. With a correct budget the multiplier stays at or below 1 + budget_percent (about 1.2x at 20 percent). Without it you will see a 2–6x spike. Envoy exposes the retry rate directly via the upstream_rq_retry and upstream_rq_retry_limit_exceeded counters.

Parent: Circuit Breaking & Retry Budgets

Related

• Circuit Breaking & Retry Budgets — the full resilience model this budget sits inside: timeouts, outlier detection, and breakers.
• Rate Limiting & Throttling Strategies — shed load before it reaches the retry layer during sustained degradation.
• Scaling Limits & Capacity Planning — measuring the upstream headroom a retry budget must fit inside.