Rolling Back a Bad Gateway Config Safely

A configuration change that made production worse is the one incident where the correct first action is known before the diagnosis: put it back. Everything difficult about this runbook is in making that possible — knowing what the previous version was, having it still available, and being able to apply it without the tooling that is currently mid-rollout.

Prerequisite concepts

This assumes the staged rollout model from gateway observability and operations and the triage order in incident response and gateway runbooks.

Know what is running before you change it

A partial rollout is the state most likely to confuse everyone Eight nodes run the new configuration version, three run the previous one and one failed to load either and is still on a version from last week. Without a config-version metric label this state is invisible, and the symptoms look intermittent rather than partial. gateway nodes by running config version v412 — new 8 nodes v411 — previous 3 nodes v396 — stale 1 node — failed to load anything since last week The stale node is the one worth finding: it has been serving old routing rules quietly, and it will keep doing so after the rollback because the rollback is also a config change it will fail to apply. 
# the single most useful query during a config incident
count by (config_version) (gateway_config_version_info)

# and errors by version — does the new one actually correlate?
sum by (config_version) (rate(gateway_requests_total{code_class="5xx"}[2m]))
  / sum by (config_version) (rate(gateway_requests_total[2m]))

If the error rate is identical on both versions, the configuration change is not the cause and reverting it will cost time without helping. That query is worth running before the revert, and it takes five seconds.

The revert itself

# Kong 3.x, declarative — apply the previous artefact, not a hand-edited file
deck gateway sync ./releases/v411/kong.yaml --select-tag managed

# Envoy with a control plane — pin the snapshot version
gwctl config pin --snapshot v411 --reason "incident-2026-08-03"

# verify convergence before declaring success
watch -n2 'curl -s $PROM/api/v1/query   --data-urlencode "query=count by (config_version) (gateway_config_version_info)" | jq -r ".data.result[].metric.config_version"'

Two rules make this safe. Apply a stored artefact rather than editing the current configuration by hand — a hand edit during an incident produces a fourth version that has never been reviewed and never existed before. And verify convergence with the same query used to detect the problem, rather than assuming the command that returned zero also finished the job.

Four steps, each verified before the next Confirm the new version correlates with the errors, apply the stored previous artefact, watch every node converge onto it, and confirm the error rate has fallen. Skipping the first step wastes the revert; skipping the third declares success while some nodes are still failing. 1. correlate errors by version 2. apply the artefact stored, never hand-edited 3. watch convergence every node, not most 4. confirm A node that does not converge in step three is a finding in its own right and will not be fixed by repeating the command — take it out of rotation and investigate it separately. Record the version numbers and timestamps as you go; the review will need them and nobody remembers afterwards.

When rollback is not available

Some changes cannot be reverted by reapplying the previous artefact: a schema migration the new config depends on, a deleted route whose consumers have already moved, a certificate that was rotated. For these, the runbook needs a forward fix and should say so explicitly rather than implying a revert that will fail.

Not every change has a revert Route weights, timeouts, plugin settings and matching rules all revert cleanly by reapplying the previous artefact. A deleted route whose consumers have already moved, a rotated certificate and a change that depends on a completed migration do not — those need a forward fix, written before the change ships. reverts cleanly route weights and matching rules timeouts, retries, circuit breakers plugin configuration and ordering rate limits and quotas apply the previous artefact and wait needs a forward fix a deleted route consumers already left a rotated or replaced certificate config depending on a finished migration anything that changed persisted state write the procedure before shipping it Flag the right-hand cases at review time. Discovering mid-incident that the revert will not work is the worst moment to start designing a forward fix.

The general principle is that any change whose rollback is not simply “apply the previous version” should be flagged as such at review time, and its forward-fix procedure written before it ships. That is a small amount of work for the author and an enormous amount of relief for whoever is on call.

Keep the artefact for at least the previous ten versions rather than only the last one — the change that caused the problem is not always the most recent, particularly when a partially applied rollout left some nodes several versions behind.

Gotchas and failure signals

Reverting a config change does not revert its effects. Cache entries populated under the new rules, quota counters keyed differently, and connections established with different timeouts all persist.

A rollback that requires the CI pipeline is unavailable when the pipeline is busy or broken. Keep a direct path to the configuration store.

Correlating by time alone is misleading when a rollout took twenty minutes to propagate. Correlate by config version, which the label makes possible.

Two people reverting simultaneously produces a race with an unpredictable winner. Say in the channel who is applying the change before applying it.

One more habit is worth building into the procedure: announce the intended action in the incident channel before executing it, in one line naming the version being applied. Two people reverting at once produces a race whose winner is unpredictable and whose loser has no idea their change was overwritten, and the only reliable defence is a convention that costs five seconds.

Validation

  • Config version exported as a metric label on every node
  • Previous configuration artefacts retained and directly applicable
  • Rollback path independent of the CI pipeline, and rehearsed
  • Convergence verified by query, not by the exit code of the apply command
  • Changes without a simple revert flagged at review, with a forward fix written
  • Version numbers and timestamps recorded during the incident, not after
  • Convention agreed that whoever applies a configuration change announces it in the channel first, so two people cannot revert simultaneously and overwrite each other

FAQ

Should I revert first and diagnose later?

Almost always — but spend five seconds first comparing the error rate between config versions. If both versions show the same error rate, the change is not the cause and reverting will cost time without helping. That query is only possible if the running config version is exported as a metric label, which is the single highest-value observability addition for config incidents.

Why not just edit the current configuration to fix it?

Because a hand edit during an incident creates a fourth version that has never been reviewed, never been tested, and does not exist in any repository. Apply the previous stored artefact instead. If the previous version genuinely cannot be reapplied, that is a finding to record — and a forward fix should have been written when the change was reviewed.

How do I know the rollback finished?

By querying the distribution of config versions across nodes, not by the exit code of the apply command. A node that does not converge is a separate problem and will not be fixed by running the command again — it has probably been failing to load configuration for some time, which is exactly the sort of thing that only becomes visible during an incident.

What does reverting the config not undo?

Its effects. Cache entries populated under the new rules stay populated, quota counters keyed differently keep their values, and connections established with the old timeouts persist until they close. Reverting restores the rules, not the state the rules produced, so check for lingering effects before declaring the incident over.

Parent: Incident Response & Gateway Runbooks