Automated Rollback on SLO Breach

Automated rollback is the difference between a bad release costing two minutes and costing however long it takes someone to notice, log in, and decide. The mechanism is simple — watch a signal, revert the weight — and every failure of it comes from the same three places: the wrong signal, too little data, or a rollback path that depends on the thing that is broken.

Prerequisite concepts

This assumes the comparison discipline in traffic splitting and progressive delivery and the burn-rate material in defining SLOs for gateway latency and error budgets.

The loop

Four steps, and a hold that stops it oscillating The controller queries per-release metrics, compares canary against stable over the same window, decides to advance, hold or revert, and applies a weight change. A hold period after each change lets the metrics settle before the next evaluation, which is what stops the loop reacting to its own effects. query metrics per release compare canary vs stable decide advance, hold, revert apply weight one API call hold for one evaluation window before looking again Without the hold the controller evaluates metrics that still reflect the previous weight, and a rollout can oscillate between advancing and reverting without ever being wrong about anything. 
# an analysis definition, expressed as thresholds against a comparison
analysis:
  interval: 60s
  iterations: 10                    # ten minutes at each weight
  threshold_failures: 2             # two consecutive breaches, not one blip
  metrics:
    - name: error-rate
      query: |
        sum(rate(gateway_requests_total{route="orders",release="canary",code=~"5.."}[2m]))
        / sum(rate(gateway_requests_total{route="orders",release="canary"}[2m]))
      compare_to: stable
      max_relative_increase: 0.25   # 25% worse than stable, not an absolute number
    - name: client-error-rate
      query: |
        sum(rate(gateway_requests_total{route="orders",release="canary",code=~"4.."}[2m]))
        / sum(rate(gateway_requests_total{route="orders",release="canary"}[2m]))
      compare_to: stable
      max_relative_increase: 0.5
    - name: latency-p99
      query: histogram_quantile(0.99, sum by (le) (rate(gateway_duration_bucket{release="canary"}[2m])))
      compare_to: stable
      max_relative_increase: 0.2
  min_request_rate: 5              # do not evaluate below this — no data is not success

min_request_rate is the guard that matters most. A canary receiving no traffic produces no errors, and a controller comparing zero against zero will happily advance a release that was never exercised. Treat insufficient data as a hold, never as a pass.

Choosing thresholds that do not fire on noise

One breach is noise; two consecutive breaches is a signal The measured difference between canary and stable fluctuates around zero and touches the threshold once by chance. Requiring two consecutive evaluation windows above the threshold ignores that single excursion and still reverts within two minutes when the difference is real and sustained. canary − stable threshold one excursion — ignored sustained — revert here Requiring consecutive breaches costs one evaluation window of extra exposure and removes almost all false reverts, which is what stops teams from disabling the automation after the third spurious rollback.

The rollback path must not depend on the rollout

If the weight change goes through the same pipeline that is currently blocked waiting for canary analysis, the rollback cannot run. If it goes through a control plane that the bad release has overloaded, it cannot run either.

Where the minutes actually go With a human in the loop the elapsed time is dominated by detection and by deciding whether the canary is at fault, not by the rollback itself. An automated controller removes both, leaving only the evaluation window and the configuration propagation time. human detector notice: 6-20 min decide: 5-15 min revert total: 12-40 minutes automated controller 2 windows revert total: about 2 minutes, and no judgement under pressure The saving is not in the revert command — it is in removing the two steps that require a person to be awake, looking at the right dashboard, and confident enough to act.

The requirement is a path to the gateway configuration that is independent of both: a direct write to the configuration store, executed by the controller and available as a documented one-line command a human can run identically. Test it during a game day by rolling back a healthy canary — if that is awkward, it will be impossible during an incident.

One further guard is worth adding before the loop is trusted with production: an absolute stop. However good the comparison logic is, a controller that has reverted three times in an hour is telling you something the thresholds cannot express, and the right response is to stop the rollout entirely and page a person rather than to try a fourth time.

Decision matrix

Signal Use for automated rollback?
5xx rate, canary vs stable yes — the primary signal
4xx rate, canary vs stable yes — catches validation regressions
p99 latency, canary vs stable yes, with a relative threshold
Absolute latency threshold no — fires on traffic changes, not releases
Business metric yes, if it moves within the evaluation window
Log-based error strings no — too slow and too noisy for a control loop

Gotchas and failure signals

No data is not success. Hold on insufficient traffic; never advance.

Comparing to a fixed threshold instead of to stable produces reverts during ordinary traffic peaks and confidence during quiet nights.

A controller with no rate limit on its own actions can flap the weight several times a minute, which is worse for users than either version alone.

Rolling back the weight does not roll back writes. If the canary wrote, the data recovery is a separate plan and it should exist before the rollout starts.

Validation

  • Every metric compared relatively against stable, over the same window
  • Minimum request rate enforced; insufficient data holds rather than advances
  • Consecutive-breach requirement configured and tuned against a week of history
  • Rollback path independent of the deployment pipeline and the rollout controller
  • Rollback rehearsed on a healthy canary in a game day
  • Data recovery plan exists for anything the canary may have written
  • The controller itself is rate limited, so it cannot change the weight more than once per evaluation window and cannot flap the rollout faster than the metrics it is reading can possibly settle
  • Every automated decision is written to the change record with the observed numbers attached, so a repeat attempt three days later starts from evidence rather than from memory

FAQ

Why compare against stable instead of an absolute threshold?

Because absolute thresholds encode assumptions about traffic that stop holding. A latency threshold that is comfortable at midnight fires during the morning peak, and an error-rate threshold tuned for peak grants a free pass at night. Comparing canary against stable over the same window removes both effects, and needs nothing more than a release label on every metric.

What stops the controller reverting on noise?

A consecutive-breach requirement. A single evaluation window above the threshold is usually chance; two in a row is a signal. That costs one extra window of exposure and removes almost all false reverts, which matters because a team that experiences three spurious rollbacks turns the automation off.

What should happen when the canary receives no traffic?

Hold, never advance. A canary with no requests produces no errors, and a controller comparing zero against zero will happily promote a release that was never exercised. Configure a minimum request rate below which the analysis refuses to draw a conclusion, and treat insufficient data as a reason to wait.

Why must the rollback path be independent of the pipeline?

Because the pipeline is often the thing that is stuck. A rollout waiting on a canary analysis step cannot also be used to cancel that rollout, and a control plane overloaded by the bad release cannot deliver the weight change that would stop it. Provide a direct write to the configuration store, and rehearse it by rolling back a healthy canary during a game day.

Parent: Traffic Splitting & Progressive Delivery