Debugging Upstream Timeout Cascades

Latency climbing while the error rate stays flat is the least dramatic and most misdiagnosed gateway incident. Nothing is failing, so no alert based on errors fires; requests are simply queueing somewhere, and the queue is usually created by a timeout that is longer than the one above it. This runbook finds the layer that is queueing and the mismatch that caused it.

Prerequisite concepts

This assumes the nested-timeout model in circuit breaking and retry budgets and the per-stage timing signals in gateway observability and operations.

The signature

Three flat lines and two rising ones Throughput and error rate stay flat while p99 latency and the in-flight request count rise together. That combination is queueing rather than failure: the same number of requests are arriving and succeeding, but each is spending longer waiting. error rate — flat throughput — flat p99 latency in-flight requests In-flight count rising with latency while throughput holds is Little's law in action: concurrency equals arrival rate times service time, so if arrivals are flat and concurrency is rising, service time is the variable. No error-based alert will fire for any of this until something finally exceeds a timeout.

Step 1 — Find the layer that is queueing

# gateway overhead vs upstream time — which half grew?
histogram_quantile(0.99, sum by (le) (rate(gateway_request_duration_bucket{route="orders"}[5m])))
histogram_quantile(0.99, sum by (le) (rate(gateway_upstream_duration_bucket{route="orders"}[5m])))

# queueing at the gateway shows here before it shows anywhere else
sum(envoy_cluster_upstream_rq_pending_active{cluster="orders_cluster"})
sum(rate(envoy_cluster_upstream_rq_pending_overflow{cluster="orders_cluster"}[2m]))

Pending requests are the definitive signal. A non-zero and rising pending_active means the gateway has more requests for that cluster than its connection limits allow in flight, and everything behind that limit is waiting rather than failing.

Step 2 — Check the timeout ladder

An inner timeout longer than the outer one manufactures waste The client gives up after two seconds, the gateway after ten and the upstream after thirty. Work continues on requests whose callers have already left, occupying connections and database sessions, and the resulting queue slows the requests whose callers are still waiting. timeout at each layer — inner must be shorter than outer client 2 s — gives up first gateway 10 s — still waiting for a caller who left upstream 30 s — still computing an answer nobody will receive Every second past the client's deadline is work that cannot help anyone and is occupying a connection, a thread and often a database session that a live request needs. Correct order: client ≥ gateway ≥ upstream, each with room for retries and backoff.

The cascade forms when this ordering is inverted anywhere in the chain. Because the upstream keeps working on abandoned requests, its effective capacity falls, which lengthens the queue, which pushes more requests past the client deadline — a loop that stabilises only at a much worse operating point.

Step 3 — The mitigations, in order of preference

Action Effect Risk
Shorten the upstream timeout below the gateway’s abandoned work stops accumulating some slow-but-valid requests now fail
Reduce max_pending_requests shed early instead of queueing visible 503s, which is usually better
Add or tighten a concurrency limit bounds in-flight work requires knowing the safe concurrency
Scale the upstream more capacity for the same queue slow, and does not fix the ordering
Disable retries temporarily removes amplification loses genuine transient recovery

Shedding load is the counter-intuitive one and usually correct: a fast 503 lets a client retry against a healthy node, while a request queued for eight seconds and then timed out has consumed capacity and helped nobody.

How much of the upstream is working on nobody Early in the cascade almost all upstream work serves a caller who is still waiting. As the queue lengthens past the client deadline the share of work whose caller has already given up grows, until most of the upstream capacity is producing answers that will be discarded. upstream capacity, by whether a caller is still waiting queue 0.5 s nearly all of it useful queue 3 s abandoned queue 8 s abandoned — most of the fleet is now computing discarded answers Shedding at the gateway converts the dashed region back into free capacity, which is why it recovers faster than scaling.

Gotchas and failure signals

Averages hide this entirely. The mean barely moves while the p99 triples, because the queue affects the tail first. Always look at a high percentile.

Retries multiply queued work. Each retry is another queued request against an upstream already behind, which is the amplification described in tuning retry budgets to prevent thundering herd.

A client that retries on timeout with no backoff turns its own timeout into a load multiplier, and the gateway sees the result as increased arrival rate rather than as the same users waiting.

The upstream often looks healthy from its own perspective, reporting normal error rates and CPU. It is doing exactly what it was asked, on requests nobody is waiting for.

It is also worth knowing that the same shape appears without any upstream problem at all, when a gateway-side dependency slows down. A quota store, an external authorisation service or a token introspection endpoint that gets slower produces rising gateway overhead with flat upstream time — the mirror image of the usual case, and one that sends teams to investigate a service that is behaving perfectly. The split between gateway and upstream time answers this in one panel, which is why it is the first thing to build.

Validation

  • Timeouts verified as non-increasing from client through gateway to upstream
  • Pending request count and pending overflow graphed per cluster
  • In-flight request count graphed alongside completed request rate
  • p99, not the mean, on every latency panel used for alerting
  • Load shedding preferred over unbounded queueing, and the threshold documented
  • This procedure exercised with an injected upstream delay
  • Gateway-side dependencies such as quota stores and authorisation services included in the same latency breakdown, since they produce the mirror image of this signature

FAQ

Why does no alert fire during a timeout cascade?

Because nothing is failing yet. Throughput and error rate stay flat while latency and in-flight request count rise together, which is queueing rather than failure. Alerts keyed on error rate see nothing until requests finally start exceeding a timeout, by which point the queue is deep. Alert on a latency percentile and on in-flight count, not only on errors.

How do I find which layer is queueing?

Compare gateway overhead against upstream time on the same panel, and look at the pending request count for that upstream cluster. A rising pending count means the gateway has more work for that upstream than its connection limits allow in flight, so requests are waiting rather than failing — which is the definitive signal and the one most dashboards lack.

What actually causes the cascade?

A timeout somewhere inside the chain that is longer than the one outside it. When the upstream keeps working for thirty seconds on a request whose caller gave up after two, that work occupies a connection and often a database session while helping nobody. Effective capacity falls, the queue lengthens, and more requests pass their client deadline — a loop that settles at a much worse operating point.

Is shedding load really better than queueing?

Usually, yes. A fast 503 lets a client retry against a healthy node or fail cleanly, while a request queued for eight seconds and then timed out has consumed capacity and helped nobody. Reducing the pending-request limit converts an invisible latency problem into visible errors, which is both easier to alert on and less damaging overall.

Parent: Incident Response & Gateway Runbooks