Incident Response & Gateway Runbooks

The gateway is the first place an operator looks during an incident and the last place anyone has written a runbook for. It sits on the path of every request, which makes it the best vantage point in the system — and it also makes it the component most often blamed for failures it is merely reporting. This topic covers the discipline that turns that vantage point into fast answers: the four questions to ask in order, the signals that separate a gateway problem from an upstream one, and the runbooks worth writing before you need them. It sits under gateway observability and operations, which covers the telemetry these procedures consume.

Architectural baseline

The gateway reports failures it did not cause. A 502 means the upstream did not answer usefully; a 504 means it did not answer in time. Both are the gateway telling you about somebody else, and the first job of any gateway runbook is to establish which side of the proxy the problem is on before anyone starts changing gateway configuration.

Configuration changes are the leading self-inflicted cause. They reach production faster than code, skip the test suite, and are reviewed by fewer people. Any triage that has not asked “what changed in the gateway config in the last hour?” is incomplete.

Runbooks are for the middle of the night. A runbook that requires judgement, context or a second opinion is a document, not a runbook. The test is whether an engineer who has never seen this failure can execute it correctly at three in the morning.

The four questions, in order

Narrow the location before you touch anything Ask whether the gateway or the upstream is failing, then whether the failure is on every node or only some, then whether it is on every route or only some, then what changed recently. Each answer eliminates a class of cause, and the sequence takes about two minutes with the right dashboards. 1. Which side? — gateway time vs upstream time on the same panel eliminates half the system 2. Every node, or some? — error rate broken down by gateway instance some ⇒ config propagation 3. Every route, or some? — error rate broken down by route some ⇒ one upstream or plugin 4. What changed? — config version, deploys, certificate expiry, all on one timeline Two minutes if the dashboards exist; forty if they have to be assembled during the incident.

The dashboards that answer these four questions are the highest-value observability work available, and they are unglamorous: four panels, each a rate broken down by one label.

Separating gateway from upstream

The single most useful panel splits the request duration into the part the gateway spent and the part it waited. Every gateway can emit both; almost none do by default.

# gateway overhead: total time minus upstream time, p99
histogram_quantile(0.99, sum by (le, route) (rate(gateway_request_duration_bucket[5m])))
-
histogram_quantile(0.99, sum by (le, route) (rate(gateway_upstream_duration_bucket[5m])))

# and the fastest triage query of all — where are the errors, by code class
sum by (route, code_class) (rate(gateway_requests_total{code_class!="2xx"}[5m]))
What each status code is actually telling you A 502 means the upstream answered unusably or not at all. A 503 usually means no healthy upstream or load shedding. A 504 means the upstream exceeded the timeout. A 499 or client-close means the caller gave up first. Only 500 from the gateway itself, and 4xx from a policy, are the gateway's own decisions. code what it means whose problem 502 connection refused, reset, or an unparseable response upstream 503 no healthy endpoint, or the gateway is shedding load upstream, or capacity 504 upstream exceeded the configured timeout upstream, or the timeout 499 the client disconnected before the response client, or you were too slow 500 from the gateway, or 4xx from a policy the gateway's own decision Only the highlighted row is the gateway failing; the rest is the gateway reporting accurately.

The runbooks worth having

Three cover the overwhelming majority of gateway incidents, and each has its own page here.

A sudden rise in 502s. Almost always an upstream that has stopped accepting connections, occasionally a certificate that expired, rarely the gateway. Covered in triaging a 502 storm at the gateway.

A configuration change that made things worse. The fastest possible revert, followed by diagnosis. Covered in rolling back a bad gateway config safely.

Latency climbing with no errors. Queueing somewhere, usually a timeout mismatch between layers. Covered in debugging upstream timeout cascades.

The first five minutes

Most of the value in a gateway incident is created before anyone understands the cause, and it comes from three actions that are safe to take without a diagnosis.

Stop the bleeding if there is an obvious lever. If a canary is in progress, set its weight to zero. If a configuration change landed in the last fifteen minutes, revert it. Neither action requires being sure it was the cause — both are cheap, both are reversible, and both remove a variable from the investigation. The instinct to establish causation first is correct in engineering and wrong in incident response.

Capture evidence that will disappear. In-flight request counts, connection pool saturation, active connection age distribution and the current running config version are all point-in-time values that no dashboard retains at useful resolution. Take a snapshot into the incident channel in the first minutes; you will want it during the review and it will be gone.

Say what you know in one line. “5xx on the orders route only, all nodes, started 14:32, config version unchanged” is worth more than twenty minutes of investigation because it lets everyone else eliminate what they were about to check. The format matters less than the discipline of publishing it early and updating it as facts change.

# a snapshot worth taking before anything else
kubectl exec -n gateway "$POD" -- curl -s localhost:9901/stats   | grep -E 'downstream_cx_active|upstream_cx_active|upstream_rq_pending|_overflow'
kubectl exec -n gateway "$POD" -- curl -s localhost:9901/server_info | jq '.state, .uptime_current_epoch'
kubectl get pods -n gateway -o wide | tee /tmp/incident-pods.txt

Deciding whether the gateway is the cause or the messenger

The distinction is worth a moment of formal thought because getting it wrong sends a team down a long path. Three observations separate the cases reliably.

Look at gateway overhead, not total latency. If the time the gateway itself spends is flat and the upstream time has risen, nothing about the gateway has changed. That single comparison resolves the majority of “is it the gateway?” arguments, and it takes one panel.

Check whether the error rate is uniform across upstreams. A gateway problem — a bad plugin, an exhausted resource, a config error — affects routes broadly. An upstream problem affects the routes pointing at that upstream and no others. A dashboard broken down by route answers this instantly and is the reason the route label matters more than any other.

Compare nodes. A gateway fault that affects some nodes and not others is nearly always configuration that has not converged, or a resource limit reached on the affected nodes. A fault that affects every node uniformly is more likely to be something the gateway is downstream of.

After the incident: what actually prevents the next one

Post-incident work on a gateway tends to produce two kinds of action item, and they have very different value.

The low-value kind adds an alert on the specific symptom that occurred. That alert will fire the next time this exact failure happens, which is rarely, and it adds one more thing to tune. The high-value kind removes the reason the failure was hard to diagnose: a missing label, a metric that was not exported, a runbook step that turned out to be wrong, a dashboard that had to be built during the incident.

The second kind compounds. Every incident that ends with “we could not tell which node was on the old config” and produces a config-version label makes every future incident shorter, including ones with completely different causes. Prioritise the observability gap over the specific alert, and prefer changes that make a class of question answerable to ones that answer a single question you have already answered.

It is also worth recording explicitly when the gateway was not at fault. A pattern of incidents where the gateway was blamed, investigated and cleared is itself a finding: it means the dashboards do not distinguish gateway from upstream clearly enough, and the fix is a panel rather than a process.

What makes a runbook usable at 3am

The difference between a document and a runbook A document says to investigate the upstream and check whether it is healthy. A runbook gives the exact query to run, the number that means yes, the command to execute and the expected output. The second can be followed by someone who has never seen the failure before. a document "Check whether the upstream is healthy" "Consider rolling back if appropriate" "Escalate to the platform team" requires knowing what healthy looks like a runbook run: this exact query if the number is above 5: run this command expected output: this line appears executable without prior context Write runbooks immediately after an incident, while the exact commands are still in the terminal history — a week later they get written from memory and become documents again.

Practising, so the first execution is not during an outage

A runbook’s first real execution reveals its errors, and the only question is whether that happens during an incident or during an exercise you chose the time of.

Fault injection at the gateway makes these exercises cheap, because the gateway can produce most of the failure modes it will later report. A fault filter that returns 503 for a percentage of requests to one upstream reproduces a partial outage. Adding a fixed delay above the configured timeout reproduces a timeout cascade. Pointing a route at an upstream with no healthy endpoints reproduces the silent-canary failure. None of these require breaking a real service.

# Envoy 1.32+ — inject a fault on one route, for one game day
- name: envoy.filters.http.fault
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault
    delay:
      fixed_delay: 4s                      # above the 3s route timeout
      percentage: { numerator: 20, denominator: HUNDRED }
    abort:
      http_status: 503
      percentage: { numerator: 5, denominator: HUNDRED }
    headers:
      - name: x-chaos                      # only requests that opt in
        string_match: { exact: "gameday" }

Gating the fault on a header means the exercise runs in production against synthetic traffic without touching real users, which is what makes it possible to run monthly rather than annually.

Run the exercise with the person least familiar with the system holding the runbook, and with the people who wrote it staying quiet. Every place they hesitate is a defect in the document, and those hesitations are precisely the ones that will cost twenty minutes at three in the morning. Fix the runbook in the same session, while the hesitation is still fresh — a list of improvements written down for later becomes a list of improvements.

Operational gotchas

The gateway’s own dashboards go down with it. If the metrics pipeline runs through the gateway, an outage removes the evidence. Keep the operational telemetry path independent of the traffic path.

Access logs appear at request completion, so an incident consisting of very slow requests is invisible in logs until it resolves. Watch in-flight request count, not just completed request rate.

Config version is the label people forget. Without it, “did the change reach this node?” is answered by logging into nodes. With it, it is a panel.

A runbook that has never been executed is a hypothesis. Run each one during a game day against an injected fault, and fix the step that turns out to be wrong — there is always one.

Production configuration checklist

  • Gateway time and upstream time emitted as separate metrics on every route
  • Error rate breakable down by node, route and code class in one dashboard
  • Running config version exported as a metric label
  • In-flight request count exported, not only completed request rate
  • Telemetry path independent of the traffic path
  • One runbook each for 502 storms, bad config, and latency without errors
  • Every runbook contains exact commands and expected outputs
  • Each runbook executed at least once against an injected fault
  • The exercise was led by someone who had not written the runbook, and every hesitation was fixed in the same session rather than recorded for later

FAQ

Is a 502 the gateway’s fault?

Almost never. A 502 means the upstream refused the connection, reset it, or returned something the gateway could not parse — the gateway is reporting accurately about somebody else. The codes that indicate a gateway problem are a 500 generated by the gateway itself and 4xx responses produced by its own policies. Establishing which side of the proxy the fault is on is the first triage step for exactly this reason.

What should the first four questions be?

Which side of the proxy is failing, whether it affects every node or only some, whether it affects every route or only some, and what changed recently. Each answer eliminates a class of cause: some nodes points at config propagation, some routes points at one upstream or plugin, and a recent config change is the leading self-inflicted cause of gateway incidents.

Why do slow requests not show up in the logs?

Because access log lines are written when the request completes. An incident made of requests that each take ninety seconds produces no log lines at all until they finish, so the dashboard looks quiet at exactly the moment the system is worst. Export an in-flight request count as a gauge alongside the completed request rate.

What makes a runbook actually usable during an incident?

Exact commands, exact thresholds and expected outputs, so it can be followed by an engineer who has never seen the failure. A step that says “check whether the upstream is healthy” requires knowing what healthy looks like; a step that gives the query, the number that means trouble and the command to run next does not. Write them straight after an incident while the commands are still in the shell history.

Parent: Gateway Observability & Operations