Header-Based Dark Launches and Shadow Traffic

Shadow traffic is the only way to exercise new code with real production requests at zero user risk, and dark launching is the only way to put a feature in front of a chosen set of people before anyone else. Both are cheap to configure and both have one sharp edge — writes for shadowing, header trust for dark launches. This page covers the configuration and the edges.

Prerequisite concepts

This assumes the mechanism overview in traffic splitting and progressive delivery and header-match semantics from path and header-based routing.

Shadowing: real requests, discarded responses

The user never waits for the shadow and never sees its answer The gateway forwards the request to the stable version and returns that response. A copy goes to the shadow target, whose response is read and thrown away. A shadow that is slow, wrong or failing has no effect on the user, which is what makes this safe. request gateway stable — serves the user shadow — response read and discarded user The gateway does not wait for the shadow before responding — its latency is invisible to the caller. 
# Envoy 1.32+ — mirror a fraction, mark it, and keep it out of traces
route:
  cluster: orders_stable
  request_mirror_policies:
    - cluster: orders_shadow
      runtime_fraction: { default_value: { numerator: 20, denominator: HUNDRED } }
      trace_sampled: false
  request_headers_to_add:
    - header: { key: x-shadow, value: "false" }
# and on the shadow cluster, the header the target uses to know what it is:
# Envoy appends -shadow to the Host header automatically, which most frameworks log

The shadow target should be able to tell that it is a shadow. Envoy appends a suffix to the authority header, which is enough for the service to log differently, skip side effects, or write to a separate store. A shadow that cannot distinguish itself will emit alerts, send emails and charge cards.

Writes are the constraint. Mirroring a POST /payments charges twice. Three approaches work: mirror only safe methods, point the shadow at a separate data store, or have the shadow detect itself and short-circuit its side effects. What does not work is remembering to be careful.

Comparing what the shadow produced

Most differences are not defects, which is why comparison needs rules Comparing shadow responses against stable produces three categories: byte-identical, differences in fields expected to vary such as timestamps and generated identifiers, and genuine differences in meaningful values. Only the third category is a finding, and separating it from the second is the work. identical byte for byte the expected majority expected difference timestamps, request ids, generated identifiers exclude by field path real regression a price, a status, a missing field this is the finding Without an exclusion list the middle box swamps the right one and the comparison is abandoned within a day. Build the list from the first hour of output, then treat any new field appearing in it as a question rather than as noise to be suppressed. Sample the comparison rather than diffing every response — a percent of shadow traffic is plenty.

Dark launching by header

# Envoy 1.32+ — the specific rule first; routes match in order
routes:
  - match:
      prefix: "/v2/checkout"
      headers:
        - name: x-feature-ring          # set by the gateway from a verified claim
          string_match: { exact: "internal" }
    route: { cluster: checkout_next }
  - match: { prefix: "/v2/checkout" }
    route: { cluster: checkout_stable }

The header must not be one the caller can set, or the dark launch is a public launch for anyone who reads a blog post. Derive it from a verified claim, from a consumer record, or from an internal allow-list the gateway resolves — and strip the inbound header at the edge so an assertion from outside cannot survive.

Rings of exposure, widening as confidence grows Internal staff find obvious breakage and can describe it. A friendly tenant finds integration problems in a real workflow. A random percentage produces statistics. Everyone is the end state. Each ring detects something the previous one could not, which is why skipping straight to a percentage misses the findings a person would have reported. internal staff obvious breakage, described in words one friendly tenant integration issues in a real workflow 5% of users statistics: error rate, latency, conversion everyone the release is done Going straight to the third ring skips the only stage where a human explains what looked wrong — statistics tell you something changed, not what a user found confusing. Shadow traffic sits before all four: it exercises the code with real requests while exposing nobody at all.

One practical note on ordering: put the dark-launch rule above the general one and keep the two rules adjacent in the configuration file. A specific rule that drifts away from the general rule it overrides is how a dark launch quietly becomes permanent.

Decision matrix

Goal Mechanism
Prove new code survives production load shadow traffic
Compare outputs before exposing anyone shadow with response comparison
Let staff try it first header targeting from a verified claim
Give one tenant early access header or claim match on the tenant
Test a write path safely shadow with a separate data store
Gather statistics weighted split, not either of these

Gotchas and failure signals

Shadow load is real load on shared dependencies. A shadow at a hundred percent doubles the traffic reaching a database both versions share.

Mirrored requests inherit the original’s headers, including authentication. The shadow target must be inside the same trust boundary, or you have just forwarded credentials somewhere new.

Tracing a shadow doubles span volume and makes traces confusing. Keep trace_sampled: false unless you are specifically debugging the shadow.

A dark-launch header left in place after the launch becomes an undocumented way to reach a code path nobody tests any more. Remove it as part of finishing the rollout.

Validation

  • Shadow target can identify itself and skips side effects accordingly
  • Only safe methods mirrored, or the shadow writes to a separate store
  • Shared dependency capacity checked against the added shadow load
  • Response comparison excludes expected-difference fields, by explicit list
  • Dark-launch header derived from a verified source and stripped inbound
  • The header and the shadow policy both removed when the rollout completes
  • Someone owns the exclusion list used by the response comparison, and a field appearing in it for the first time is treated as a question rather than as noise to be suppressed permanently

FAQ

Does shadow traffic slow down the user’s request?

No. The gateway forwards to the stable version and returns that response without waiting for the shadow, whose response is read and discarded. A shadow that is slow, failing or completely wrong has no user-visible effect, which is exactly what makes shadowing the safest way to exercise new code with real traffic.

How do I shadow a write endpoint safely?

Three approaches work: mirror only safe methods, point the shadow at a separate data store, or have the shadow detect that it is a shadow and short-circuit its side effects. Envoy appends a suffix to the authority header, which most frameworks log and can branch on. What does not work is intending to be careful — a mirrored payment is a duplicate charge.

How do I stop response comparison drowning in noise?

Build an exclusion list of fields expected to differ — timestamps, request identifiers, anything generated — from the first hour of comparison output, and sample rather than diffing every response. Without the list the expected differences swamp the real ones and the comparison gets switched off within a day.

Can the dark-launch header come from the client?

Only if you are content for anyone to opt into untested code, which is occasionally the intent and usually is not. Derive it from a verified claim or a consumer record the gateway resolves, and strip the inbound header at the edge so an assertion from outside cannot survive. Remove the header entirely once the rollout finishes, or it becomes an undocumented path to code nobody tests.

Parent: Traffic Splitting & Progressive Delivery