Traffic Splitting & Progressive Delivery

Progressive delivery is the practice of separating deployment from release: the new build ships to production and then earns its traffic gradually, under measurement, with an automatic way back. The gateway is where that happens, because it is the only component that sees every request before it is committed to a version. This topic covers the four mechanisms — weighted splitting, header-based targeting, shadow traffic and automated rollback — what each is actually for, and the measurement discipline without which all four are theatre. It sits under advanced routing and API versioning, and uses the matching primitives from path and header-based routing.

Architectural baseline

Deployment is not release. A build running in production with zero percent of traffic is deployed and not released. Keeping those two events separate is what makes the release reversible without a redeploy, and it is the whole reason to do any of this at the gateway rather than in a deployment pipeline.

Every mechanism needs a signal. Shifting weight without watching anything is a slower way to have the same outage. The signal has to be per-version, which means the version must be a label on every metric the gateway emits — a requirement that shapes the instrumentation more than the routing.

Rollback must be cheaper than diagnosis. During an incident nobody should be deciding whether the canary is at fault. Set the weight back to zero, then investigate. That is only possible when the previous version is still running and still routable.

Four mechanisms, four purposes

They are not interchangeable, and most teams need three of the four Weighted splitting exposes a random sample of real users. Header targeting exposes a chosen set, such as internal staff. Shadow traffic exercises the new version with real requests whose responses are discarded, so users see nothing. Automated rollback closes the loop by reverting on a signal rather than on a human noticing. mechanism who is exposed what it is good at weighted split a random slice of real users statistical confidence, real load header targeting a chosen set — staff, one tenant controlled first exposure shadow traffic nobody — responses discarded correctness and load, zero risk automated rollback reverts the above bounding the blast radius in time

The common sequence is shadow first to prove the new version does not fall over, then header targeting to expose people who can report problems articulately, then weighted splitting to gather statistics, with automated rollback watching throughout.

Weighted splitting

# Envoy 1.32+ — weights on one route, with the version as a response header
routes:
  - match: { prefix: "/v2/orders" }
    route:
      weighted_clusters:
        clusters:
          - name: orders_stable
            weight: 95
            response_headers_to_add:
              - header: { key: x-release, value: stable }
          - name: orders_canary
            weight: 5
            response_headers_to_add:
              - header: { key: x-release, value: canary }
      timeout: 3s
# Kong 3.x — the same shape, expressed as weighted upstream targets
upstreams:
  - name: orders-upstream
    algorithm: round-robin
    healthchecks:
      active: { http_path: "/healthz", healthy: { interval: 5 }, unhealthy: { interval: 5 } }
    targets:
      - { target: orders-stable.internal:8080, weight: 95 }
      - { target: orders-canary.internal:8080, weight: 5 }

Two properties are worth being explicit about. Weights are per request, not per user, so a client making ten requests will usually touch both versions — fine for a stateless read, wrong for a multi-step flow. Where a user must stay on one version, hash a stable attribute into the choice rather than sampling randomly. And weights are not percentages of users; a single heavy consumer can dominate a five percent slice entirely.

The measurement that makes it real

Compare the canary against the stable version, not against a threshold Absolute thresholds mislead because traffic varies. The comparison that matters is canary against stable over the same window on four signals: error rate, latency at the ninety-ninth percentile, saturation and a business signal such as conversion. A difference beyond a defined margin on any of them is the rollback trigger. signal stable canary verdict 5xx rate 0.08% 0.09% within margin p99 latency 180 ms 184 ms within margin 4xx rate 1.2% 4.7% roll back checkout completion 71% 70.6% within margin The 4xx row is the one most rollout tooling ignores: no 5xx, no latency change, and the new build is rejecting requests it should accept — a validation regression that looks like healthy traffic on every server-error dashboard.

Comparing against a fixed threshold rather than against the stable version produces false confidence during quiet periods and false alarms during busy ones. Comparing like for like removes both, and it requires only that every metric carries a version label.

# make the comparison possible: version as a label on everything
# Envoy: stats prefix per cluster gives you envoy_cluster_orders_canary_*
# Kong: tag upstream targets and export the tag
# and in the log format:
#   {"route":"orders","release":"$upstream_release","status":$status,"duration":$request_time}

Sample size, and how long a stage must run

A canary at five percent of a route serving fifty requests a second sees two and a half requests a second. Detecting a change in an error rate of one in a thousand needs tens of thousands of requests, which at that rate is hours rather than the ten minutes most pipelines allow.

A ten-minute canary at 1% detects almost nothing On a route serving fifty requests a second, a one percent canary needs several hours to accumulate the sample required to detect a change in a one-in-a-thousand error rate. At five percent it needs about an hour, and at twenty-five percent about ten minutes. Choosing the weight is therefore also choosing how long the stage must run. time to a usable sample, route at 50 req/s 1% canary about 5 hours — a ten-minute stage proves nothing 5% canary about 1 hour 25% canary about 10 minutes — but a quarter of users are exposed The trade is exposure against elapsed time, and there is no configuration that avoids it. For low-traffic routes, shadow traffic and header targeting are more informative than a weight that will never accumulate a sample.

Header targeting and shadow traffic

Weighted splitting answers “does this behave under real load?” It does not answer “does this behave for the person best placed to notice?” — and it exposes real users to the first execution of new code. Two mechanisms fill those gaps.

Header targeting routes a chosen set of callers to the new version. Internal staff, one friendly tenant, or a specific integration whose owner has agreed to look. The routing itself is an ordinary header match; the design question is where the header comes from, and the answer must be a verified claim or a value the gateway sets itself. A header the caller supplies means anyone can opt into the untested version, which is occasionally what you want and usually is not.

# Envoy 1.32+ — staff to the canary, everyone else to stable
routes:
  - match:
      prefix: "/v2/orders"
      headers:
        - name: x-employee                 # set by the gateway from a verified claim
          string_match: { exact: "true" }
    route: { cluster: orders_canary }
  - match: { prefix: "/v2/orders" }
    route: { cluster: orders_stable }

Order matters here in a way that catches people out: the more specific rule must be evaluated first, and in Envoy that means listed first, since routes are matched in order within a virtual host.

Shadow traffic sends a copy of each request to the new version and discards the response. The user is served entirely by the stable version and never waits for the shadow, so a shadow that is slow, wrong or on fire has no user-visible effect at all. It is the only mechanism that exercises new code with real production traffic at zero risk, and it is consistently under-used.

# Envoy 1.32+ — mirror 20% of traffic, discard the responses
route:
  cluster: orders_stable
  request_mirror_policies:
    - cluster: orders_canary
      runtime_fraction:
        default_value: { numerator: 20, denominator: HUNDRED }
      trace_sampled: false

The constraint is writes. A mirrored request that creates an order creates two orders, and a mirrored payment is a duplicate charge. Shadowing is safe for reads by construction and safe for writes only when the shadow target writes to a separate store, or when the requests being mirrored are known to be idempotent and the shadow’s writes are discarded downstream.

What to do when the canary fails

The failure path deserves as much design as the success path, because it is the one that runs under pressure.

Reverting the weight is the first action and should be a single, well-known command that anyone on call can run without reading anything. Everything else — reading logs, comparing traces, deciding whether the canary was actually at fault — happens afterwards, on a system that is no longer serving users from the suspect build.

Two details make that possible. The stable version must still be running: a rollout that scales down the old version as it scales up the new one has removed the thing you would revert to, and now the rollback is a deploy. And the weight change must not require the same pipeline that is currently mid-rollout, since a pipeline waiting on a canary analysis step is exactly the pipeline you cannot use to cancel it.

Recording what happened matters more than it seems. A canary that was rolled back and then re-attempted three days later with no change to the code is a common and expensive pattern: the second attempt fails in the same way, and by then nobody remembers the details of the first. Write the signal, the observed difference and the decision into the change record at the moment of rollback, while the numbers are still on screen.

Comparative implementation

Capability Envoy 1.32+ Kong 3.x Gateway API
Weighted routing weighted_clusters upstream target weights backendRefs[].weight
Header-based targeting route headers match route headers matches.headers
Request mirroring request_mirror_policies plugin or upstream config RequestMirror filter, Extended
Sticky assignment consistent hash on a header hash-on with a header implementation-specific
Per-version metrics per-cluster stats per-target tags per-backend, implementation-specific

Progressive delivery for configuration, not only code

The same discipline applies to gateway configuration itself, and it is applied far less often. A routing change, a new plugin, a tightened schema or an adjusted rate limit are all releases, and all of them can be rolled out to a fraction of traffic before they reach everyone.

The mechanism differs slightly: instead of two versions of an upstream, there are two versions of a route or a policy, selected by the same weighting or header matching. A new validation schema can run against five percent of traffic in report-only mode, a stricter rate limit can apply to one tenant first, and a rewritten route can take a small share while the original serves the rest.

The reason this matters is that configuration changes reach production faster than code changes and are reviewed less carefully — they skip the build, the test suite and often the staged deployment. That combination is why a disproportionate share of gateway incidents originate in a configuration change rather than in a service release, and why the rollout discipline described in rolling back a bad gateway config safely belongs in the same conversation as canary deployments rather than in a separate one.

Operational gotchas

A canary sharing the stable version’s database is not isolated. Most bad releases damage data rather than latency, and a five percent canary writing corrupt rows corrupts them for everyone. Decide explicitly whether the canary may write.

Weight changes are configuration changes and propagate on whatever schedule that implies — instantly on a watched store, on a poll interval elsewhere. During a rollback that difference is the difference between thirty seconds and five minutes.

Sticky assignment interacts badly with weights. Hashing a user attribute into the choice keeps a user on one version, and it also means the realised split will not match the configured one when the hash distribution is uneven.

Nobody looks at 4xx. Add it to the comparison; a validation regression is invisible on every server-error dashboard and is one of the most common canary failures.

Production configuration checklist

  • Every gateway metric and log line carries a version or release label
  • Canary and stable compared against each other, not against fixed thresholds
  • 4xx rate included in the comparison alongside 5xx and latency
  • Stage duration derived from the sample size the route can actually produce
  • Rollback is a weight change, with the previous version still running and routable
  • Whether the canary may write to shared state decided and documented
  • Sticky assignment used for multi-step flows, random sampling for stateless reads
  • Weight-change propagation time measured, not assumed

FAQ

What is the difference between deployment and release?

A build running in production with none of the traffic is deployed but not released. Separating the two is what makes a release reversible without a redeploy: the previous version is still running, still routable, and returning to it is a weight change rather than a pipeline run. That separation is the reason progressive delivery belongs at the gateway rather than in a deployment tool.

How small should the first canary slice be?

Small enough to bound the damage and large enough to produce a usable sample, and those two pull in opposite directions. On a route serving fifty requests a second, a one percent canary needs about five hours to accumulate the requests needed to detect a change in a one-in-a-thousand error rate. If that is longer than you will wait, either raise the weight and accept the exposure, or use shadow traffic and header targeting instead.

Which signals should trigger a rollback?

Error rate, tail latency, saturation and at least one business signal, each compared between canary and stable over the same window rather than against a fixed threshold. Include the 4xx rate: most rollout tooling watches only 5xx, and a validation regression that rejects requests it should accept looks perfectly healthy on a server-error dashboard.

Can a canary share a database with the stable version?

It can, and you should decide that deliberately rather than by default. Most damaging releases corrupt data rather than raising latency, and a five percent canary writing bad rows writes them for everyone. If the canary may write, the rollback plan needs a data story as well as a routing one — reverting the weight does not revert the writes.

Parent: Advanced Routing & API Versioning