Weighted Canary Routing with Envoy and Kong

The configuration for a weighted split is three lines on either gateway. Everything that makes it work in production is around those three lines: whether a user stays on one version, whether the realised split matches the configured one, and how quickly a weight change reaches every node. This page covers all of it for Envoy 1.32+ and Kong 3.x.

Prerequisite concepts

This assumes the mechanism comparison and measurement discipline in traffic splitting and progressive delivery.

Random sampling versus sticky assignment

Weights are per request unless you make them per user With random weighting a user making six requests touches both versions, which is fine for stateless reads and wrong for a multi-step flow where the two versions disagree. Hashing a stable attribute such as a user identifier assigns each user to one version for the whole rollout. random weighting — one user, six requests stable stable canary stable canary stable a multi-step flow breaks hash on user id — same user, same version all six requests → stable, for the whole rollout flows stay coherent The cost of hashing is that the realised split will not exactly match the configured weight — hash distributions are uneven at small percentages, so a 5% ring can serve 3% or 8% of requests depending on who is active. 
# Envoy 1.32+ — sticky assignment by hashing a header
routes:
  - match: { prefix: "/v2/orders" }
    route:
      weighted_clusters:
        clusters:
          - { name: orders_stable, weight: 95 }
          - { name: orders_canary, weight: 5 }
      hash_policy:
        - header: { header_name: "x-user-id" }    # set by the gateway from a claim
          terminal: true
      timeout: 3s
# Kong 3.x — hash-on at the upstream, weights on the targets
upstreams:
  - name: orders-upstream
    hash_on: header
    hash_on_header: x-user-id
    hash_fallback: none          # no fallback: a request without the header is rejected
    targets:
      - { target: orders-stable.internal:8080, weight: 95 }
      - { target: orders-canary.internal:8080, weight: 5 }

hash_fallback: none is deliberate. A fallback to round-robin means requests missing the header are distributed randomly, which quietly reintroduces the behaviour hashing was meant to remove — and it will be the anonymous traffic, which is often the traffic you least want split mid-flow.

Weight changes and how fast they arrive

Rollback speed is propagation speed A weight change pushed over a streaming control plane reaches every node in under a second. One written to a watched key-value store arrives in a few seconds. One that relies on a periodic database poll can take a full refresh interval, and one baked into a declarative file requires a redeploy. time from changing the weight to every node serving it streaming control plane sub-second watched key-value store a few seconds database poll up to the refresh interval declarative file a redeploy — rollback is no longer a weight change Measure it once, in production, and write the number in the runbook next to the rollback command.

Verifying the split is what you configured

The realised split is an observation, not an assumption, and the first thing to check when a canary shows no signal at all.

# share of requests served by each release, last 5 minutes
sum by (release) (rate(gateway_requests_total{route="orders"}[5m]))
  / ignoring(release) group_left
sum      (rate(gateway_requests_total{route="orders"}[5m]))

A canary at a configured five percent that measures at zero point three means either the hash is concentrating traffic elsewhere, or the canary cluster has no healthy endpoints and requests are silently falling back. The second case is common and produces a rollout that looks perfect because nothing was ever tested.

Three reasons the observed share is not the configured one A healthy rollout shows an observed share close to the configured weight. An uneven hash distribution shows a share that is off by a factor but non-zero. A canary with no healthy endpoints shows a share of zero while the rollout reports success, because every request silently went to the stable cluster. configured 5% — what the metric says 4.8% observed healthy — proceed 1.9% observed uneven hash distribution — the sample will take longer 0.0% observed no healthy canary endpoints — nothing was tested at all Gate the rollout on observed share, not on elapsed time: a stage that never received traffic must not advance. This single check catches the most common silent failure in automated canary pipelines.

Decision matrix

Situation Choose
Stateless reads random weighting
Multi-step user flow hash on a stable user attribute
Per-tenant rollout hash on the tenant, or route on the claim
Very low traffic route header targeting instead of weights
Rollback must be under a minute a control plane that pushes, not one that polls

Gotchas and failure signals

A canary cluster with no healthy endpoints absorbs no traffic and reports no errors. Always gate on observed share.

Sticky assignment plus a changing user population shifts the realised split over the day as different cohorts become active. Compare rates, never cumulative counts.

Weights on targets versus weights on clusters differ in what health checking does to them: an unhealthy target is skipped and its share redistributed, which can silently move the whole canary share to stable.

Response headers naming the release are worth adding — they make a support conversation a one-line answer, and they cost nothing.

Ramp shape, and why equal steps are the wrong default

The obvious ramp is a sequence of equal steps: five, ten, twenty, fifty, a hundred. It is also the one that spends the most exposure for the least information.

Each stage exists to answer a question, and the questions are not equally hard. The first stage asks whether the new version works at all under real traffic, which needs very little data — a broken build fails immediately. The middle stages ask whether it is subtly worse, which needs the most data and therefore the most time. The last stage asks whether it holds at full load, which needs the least analysis and the most caution.

A shape that reflects that spends a short stage at a small weight, a long stage in the middle where the statistics are gathered, and a short final step. In practice: five percent for ten minutes to catch the obvious, twenty-five percent for an hour to gather a sample, then a hundred. Three stages, one long one, rather than six equal ones that each prove a little and take all afternoon.

Validation

  • Observed traffic share matches the configured weight, checked before advancing
  • Sticky assignment used where flows span requests, with no random fallback
  • Propagation time for a weight change measured and written in the runbook
  • Canary cluster health verified independently of the split
  • Release identifier present on both metrics and responses

FAQ

Will a user stay on one version during a weighted rollout?

Not by default. Weights are evaluated per request, so a user making six requests will usually touch both versions. That is fine for stateless reads and wrong for a multi-step flow where the two versions disagree. Hash a stable attribute such as a user identifier into the choice to assign each user to one version, and set no random fallback for requests missing the attribute.

Why does my canary show no errors and no signal?

Check the observed traffic share before anything else. A canary cluster with no healthy endpoints absorbs no traffic, produces no errors and lets an automated rollout advance through every stage having tested nothing. Gate each stage on measured share rather than on elapsed time — it is the single most common silent failure in canary pipelines.

Why does the realised split not match the configured weight?

Usually hashing. Hash distributions are uneven at small percentages, so a five percent ring can serve three or eight percent depending on which users are active, and the share drifts through the day as cohorts change. Compare rates rather than cumulative counts, and expect the sample to take longer than the nominal weight suggests.

How fast can I roll back?

As fast as your configuration propagates, which ranges from sub-second on a streaming control plane to a full redeploy for a baked declarative file. Measure it once in production and write the number in the runbook beside the rollback command, because during an incident the difference between thirty seconds and five minutes is the whole conversation.

Parent: Traffic Splitting & Progressive Delivery