Sizing Envoy Worker Threads and Connection Pools

Envoy’s default configuration adapts to the machine it lands on, which is convenient until the machine is a container with a CPU limit the runtime cannot see. The result is a proxy that starts as many worker threads as the host has cores, gets throttled by the cgroup, and produces latency spikes that look like upstream problems. This page covers the three numbers worth setting explicitly — worker concurrency, connection pool limits and buffer sizes — and how to derive each from something you can measure.

Prerequisite concepts

This assumes the throughput and memory model in scaling limits and capacity planning, and the execution model comparison in Kong vs Tyk vs Envoy for microservices for why Envoy’s thread model differs from a worker-process one.

Worker threads and the container limit

Envoy runs one event loop per worker thread and distributes accepted connections between them. Each worker is independent: its own connection pools, its own circuit-breaker counters, its own share of the listener’s connections.

The default counts host cores, not your CPU limit On a sixty-four core host with a container limited to four CPUs, the default concurrency starts sixty-four worker threads competing for four CPUs of quota. The scheduler throttles them at the end of each quota period, producing latency spikes with no corresponding rise in upstream time. default: concurrency = host cores 64 worker threads, each with its own pools and counters 4 CPUs of actual quota — the other 60 threads wait explicit: concurrency matched to the limit 4 worker threads, 4 CPUs — no quota throttling, no thundering wake-ups Symptom of getting it wrong: periodic p99 spikes at a fixed interval matching the scheduler quota period, with flat upstream latency underneath. It looks like a network problem and is entirely self-inflicted. 
# Envoy 1.32+ — set concurrency to the CPU limit, not the host core count
# Command line, since it is a bootstrap concern:
#   envoy -c /etc/envoy/envoy.yaml --concurrency 4
#
# In Kubernetes, derive it from the limit you already declared:
resources:
  limits:   { cpu: "4", memory: "4Gi" }
  requests: { cpu: "4", memory: "4Gi" }   # equal: avoid throttling under contention

Round up rather than down when the limit is fractional — a 1.5 CPU limit is better served by two workers than by one, because a single worker cannot use more than one core no matter how much quota is available.

Connection pools are per worker

The circuit-breaker thresholds that bound an upstream cluster are enforced per worker, which means the effective fleet-wide limit is the configured value multiplied by workers multiplied by pods. This catches teams out in both directions: too permissive against a fragile upstream, or so restrictive that a single worker queues while others idle.

Multiply before you decide the number is safe A max-requests threshold of two hundred, on four workers, across six pods, permits four thousand eight hundred concurrent requests against the upstream. The number that matters to the upstream is the product, not the value in the configuration file. max_requests: 200, as written in the config per worker × 4 workers = 800 per pod × 6 pods = 4,800 concurrent requests the upstream may receive Divide the upstream's proven concurrency ceiling by workers × pods to get the per-worker value, and re-derive it whenever either number changes — an autoscaler that doubles the pod count also doubles this ceiling silently. The same multiplication applies to max_pending_requests and to retry concurrency. 
clusters:
  - name: orders_cluster
    connect_timeout: 1s
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          max_connections: 512          # per worker, per pod
          max_pending_requests: 64      # queue depth before shedding
          max_requests: 200             # concurrent requests in flight
          max_retries: 8                # concurrent retries, not retries per request
    upstream_connection_options:
      tcp_keepalive: { keepalive_time: 60 }
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options:
            max_concurrent_streams: 100

max_pending_requests deserves particular attention: it is the queue in front of the upstream, and a large value converts a capacity problem into a latency problem. Keeping it small — tens rather than thousands — means the proxy sheds load quickly instead of holding requests that will time out anyway, which is the behaviour described in circuit breaking and retry budgets.

Buffers and the memory ceiling

Buffer size times concurrency is a hard memory floor At the 1 MiB default, ten thousand concurrent connections need roughly ten gigabytes of buffer alone. Reducing the per-connection buffer limit to thirty-two kilobytes brings the same concurrency down to about three hundred and twenty megabytes, at the cost of more read events for large bodies. per_connection_buffer_limit_bytes, at 10,000 concurrent connections 1 MiB (default) ≈ 10 GB of buffer — more than the pod has 256 KiB ≈ 2.5 GB 32 KiB ≈ 320 MB — the right order of magnitude for a proxy Smaller buffers mean more read events for large bodies, which costs CPU — so size against your real body sizes.

Decision matrix

Symptom Likely cause Change
Periodic p99 spikes, flat upstream latency worker count above the CPU limit set --concurrency explicitly
Upstream overwhelmed despite thresholds per-worker limits multiplied by fleet size divide the ceiling by workers × pods
Memory grows with connections then OOM default per-connection buffer set per_connection_buffer_limit_bytes
Requests queue while CPU is idle max_pending_requests too large shrink it and shed earlier
One worker hot, others idle long-lived connections pinned at accept shorten max connection duration

Gotchas and failure signals

Requests and limits should be equal for CPU on a latency-sensitive proxy. A pod that can burst above its request will be throttled unpredictably when neighbours claim their share, and the resulting jitter is indistinguishable from network variance on a dashboard.

Statistics are per worker too. A histogram that looks fine in aggregate can hide one worker at saturation, so alert on per-worker saturation rather than on the mean.

Connections are assigned at accept and never move. With long-lived connections, an imbalance persists for the life of those connections; a bounded connection duration is the only mechanism that rebalances them.

Raising concurrency does not raise throughput once CPU is saturated — it raises context switching. Prove the CPU is the bottleneck before changing the number, with the load-testing method in load testing an API gateway with k6.

Validation

  • --concurrency set explicitly and equal to the container CPU limit
  • CPU request equals CPU limit on latency-sensitive proxies
  • Circuit-breaker thresholds derived by dividing the upstream ceiling by workers × pods
  • per_connection_buffer_limit_bytes set, and memory headroom checked at peak concurrency
  • Per-worker saturation visible on a dashboard, not just the aggregate
  • Numbers re-derived whenever the pod count or CPU limit changes

FAQ

What should I set concurrency to?

The container CPU limit, rounded up if it is fractional — not the host core count, which is what the default uses. A sixty-four core host running a four-CPU container will otherwise start sixty-four worker threads competing for four CPUs of quota, producing periodic latency spikes at the scheduler quota interval with flat upstream latency underneath.

Are circuit-breaker thresholds per worker or per process?

Per worker. The effective ceiling seen by the upstream is the configured value multiplied by the worker count and again by the pod count, so a max_requests of two hundred across four workers and six pods permits four thousand eight hundred concurrent requests. Derive the per-worker value by dividing the upstream proven ceiling by workers times pods, and re-derive it whenever either number changes.

Why is my Envoy using so much memory?

Almost always the per-connection buffer limit, which defaults to one mebibyte. At ten thousand concurrent connections that is roughly ten gigabytes of buffer alone. Setting per_connection_buffer_limit_bytes to thirty-two kilobytes brings the same concurrency to a few hundred megabytes, at the cost of more read events for large request bodies.

Should CPU requests equal CPU limits?

On a latency-sensitive proxy, yes. A pod allowed to burst above its request gets throttled unpredictably when neighbours claim their share, and the resulting jitter is indistinguishable from network variance on a dashboard. Equal request and limit trades some bin-packing efficiency for latency you can reason about.

Parent: Scaling Limits & Capacity Planning