Triaging a 502 Storm at the Gateway

A wall of 502s is the most common gateway page and one of the least ambiguous, because the code already tells you the upstream did not answer usably. What it does not tell you is which upstream, on which nodes, and why — and there are only five realistic causes. This runbook narrows to one of them in about three minutes.

Prerequisite concepts

This uses the triage sequence from incident response and gateway runbooks and the health-check behaviour described in outlier detection vs active health checks.

Step 1 — Scope it in one query

# where are the 502s, by route and upstream cluster
sum by (route, cluster) (rate(gateway_requests_total{code="502"}[2m]))

# and are they on every gateway node?
sum by (instance) (rate(gateway_requests_total{code="502"}[2m]))

One route means one upstream. Every route means something shared: the gateway itself, the network between it and everything, or a resource limit. One node means that node, and the answer is usually to remove it from rotation and investigate afterwards.

Five causes, separated by scope If the 502s are on one route, the upstream for that route is refusing connections or has no healthy endpoints. If they are on every route but one node, that node has exhausted a resource. If they are on every route and every node, the cause is shared: a certificate, a DNS failure, or a network path. what is the scope of the 502s? one route — that upstream refuses connections, or has no healthy endpoints check its pods, its own errors, and whether health checks just ejected everything every route, one node — that node exhausted something file descriptors, memory, connection pool — remove it from rotation first every route, every node — something shared certificate expiry, DNS resolution, a network path, or a config change

Step 2 — Ask the gateway what it thinks

Every gateway can report its own view of upstream health, and that view is frequently different from what a human assumes.

# Envoy 1.32+ — which endpoints does it believe are healthy right now
curl -s localhost:9901/clusters | grep -E 'orders_cluster.*(health_flags|cx_active)'
# healthy hosts, ejections, and pending requests in one place:
curl -s localhost:9901/stats | grep -E 'orders_cluster.*(membership_healthy|outlier|pending_overflow)'

# Kong 3.x — the equivalent view
curl -s localhost:8001/upstreams/orders-upstream/health | jq '.data[] | {target, health}'

membership_healthy at zero with a non-zero total is the single most informative number in a 502 storm: the gateway has no endpoint it is willing to use, and the question becomes why they were all ejected rather than why the requests failed.

Read the healthy endpoint count before reading anything else Healthy endpoints matching the expected count means the upstream is accepting connections and failing at the application level. A count that has fallen partway means health checks or outlier detection ejected some. A count of zero with a non-zero total means everything was ejected, which is a different investigation entirely. healthy endpoints, out of 12 expected 12 healthy connections fine — the app is failing 5 healthy partial ejection — check the ejection counter 0 healthy everything ejected — why, and is the panic threshold engaged? The bottom row often means the health endpoint started failing rather than the service — a dependency it checks, a deploy that changed its path, or a timeout that is now shorter than the check takes.

Step 3 — The five causes, and the check for each

Cause Confirming check Immediate action
Upstream pods gone or crash-looping pod status, restart count scale up or roll back the upstream
Health endpoint failing, service fine call the endpoint directly and a real route relax or fix the check
Upstream TLS certificate expired openssl s_client against the upstream renew; check others expiring soon
Connection pool exhausted at the gateway pending_overflow, cx_active at the limit raise limits, shed load
DNS resolution failing or stale resolve the upstream name from a gateway pod fix resolution; check the cache TTL

The second row is the one that fools people most often: the service is serving traffic correctly and its health endpoint is failing, so the gateway ejects every healthy host. Always test both the health endpoint and a real route before concluding the upstream is down.

Step 4 — Mitigate before you finish diagnosing

What you can do in the first minute, without knowing the cause Shifting a canary weight to zero, reverting the last config change, relaxing an over-aggressive health check and failing over to a second region are all fast and reversible. Restarting the gateway is slower, drops in-flight requests and destroys the evidence, which is why it belongs last rather than first. mitigation effect reversible set any canary weight to zero seconds completely revert the last config change seconds to minutes completely relax the health check threshold one check interval yes restart the gateway drops in-flight requests destroys the evidence

Gotchas and failure signals

502 and 503 mean different things and are frequently conflated. 502 is a connection or parsing failure; 503 is usually no healthy endpoint or deliberate shedding. Splitting them in the query saves a step.

Retries hide the onset. With retries enabled, a partial upstream failure produces higher latency long before it produces 502s, so the storm appears suddenly at the point retries stop helping. Watch the retry rate as an early indicator.

A restart looks like it fixed it because it resets connection pools and health state — and it also destroys the evidence and often recurs within the hour.

Certificate expiry produces a clean, total, simultaneous failure across every node at once, which is otherwise a rare pattern. If everything failed at the same second, check certificates first.

Validation

  • The scope query is saved and linked from the alert
  • Healthy endpoint count per cluster is on a dashboard, not just an admin endpoint
  • Retry rate is graphed as an early indicator
  • Upstream certificate expiry is alerted on with weeks of warning
  • The runbook names who may relax a health check and who may fail over
  • This procedure exercised against an injected upstream failure

FAQ

What is the first thing to check in a 502 storm?

The scope, with one query: 502 rate broken down by route and by gateway instance. One route points at that upstream, every route on one node points at a resource limit on that node, and every route on every node points at something shared — a certificate, DNS, or a config change. Each answer eliminates most of the possibilities before anyone opens a log.

What does a healthy endpoint count of zero mean?

That the gateway has no upstream it is willing to use, so the investigation changes from “why are requests failing” to “why was everything ejected”. Very often the service is serving traffic correctly and its health endpoint started failing — a dependency it checks, a changed path, or a check timeout that is now shorter than the check takes. Always test the health endpoint and a real route separately.

Should I restart the gateway?

Last, not first. A restart resets connection pools and health state so it often appears to fix things, and it also drops every in-flight request and destroys the evidence you need to stop it recurring within the hour. Shifting a canary weight to zero, reverting the last config change, and relaxing an over-aggressive health check are all faster and reversible.

Why did the storm appear so suddenly?

Retries. While an upstream is partially failing, retries mask it as higher latency rather than as errors, so nothing crosses an error threshold until the failure rate rises past the point retries can cover. Graph the retry rate as an early indicator and the onset stops being a surprise.

Parent: Incident Response & Gateway Runbooks