Middleware Chains & Request Transformation

Every HTTP request that enters a production API gateway travels through a sequence of discrete processing stages — each responsible for exactly one concern — before a byte reaches your backend service. When that sequence is poorly ordered, stateful middleware contaminates horizontal scale, heavy transformation logic runs before auth checks waste CPU on traffic that will be rejected, and latency compounds across the chain. Getting the pipeline architecture right is therefore not an optimization; it is a prerequisite for operating distributed APIs at scale.

This page covers the full execution model: how phases map to data-plane concerns, when synchronous versus asynchronous dispatch matters, how to share state across gateway nodes without sacrificing scale, and how security, caching, and transformation interact at the boundary layer. The same principles apply whether you are running Kong Gateway, Envoy Proxy, NGINX Plus, or AWS API Gateway.

Key design invariants:

• Auth and schema validation must execute before transformation to prevent wasted compute on unauthorized requests.
• Short-circuit logic must be explicitly configured per plugin — implicit pass-through is the wrong default for security-sensitive stages.
• Stateful middleware (session affinity, distributed counters) requires externalized stores; local in-memory state silently breaks horizontal scaling.
• Async telemetry and logging must run off the critical path to avoid blocking worker threads under peak load.
• Cache key normalization must happen after transformation so the cached representation is always consistent with the downstream payload.
• W3C Trace Context (traceparent, tracestate) headers must be propagated or injected at the earliest possible phase to ensure end-to-end observability.
• Plugin execution priority values — not arbitrary ordering fields — control phase sequencing in Kong; misreading this is the most common source of ordering bugs.

Control Plane vs Data Plane in a Middleware Pipeline

The control plane compiles plugin configurations, resolves service upstreams, distributes policy bundles across gateway nodes, and handles certificate rotation. The data plane executes those compiled policies per request, applying filters in strict priority order with deterministic latency.

This separation matters for middleware chains because configuration changes propagate asynchronously — a new rate-limiting policy may be active on some nodes before others. During the propagation window, inconsistent behaviour across the fleet is expected. Data-plane nodes must cache their last known good configuration so a control plane outage does not halt traffic. See high availability topologies for how to design the control plane itself to survive partial outages.

In Envoy, the xDS control plane pushes Listener, Cluster, Route, and Endpoint resources independently via the aggregated discovery service (ADS). HTTP filters run on the data plane in declaration order within the http_filters list:

# Envoy 1.32+ — HttpConnectionManager with ordered filter chain
static_resources:
  listeners:
    - name: ingress
      address:
        socket_address: { address: 0.0.0.0, port_value: 8080 }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                http_filters:
                  # 1. JWT authentication — must run before routing decisions
                  - name: envoy.filters.http.jwt_authn
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
                      providers:
                        platform_idp:
                          issuer: "https://auth.platform.internal"
                          remote_jwks:
                            http_uri:
                              uri: "https://auth.platform.internal/.well-known/jwks.json"
                              cluster: jwks_cluster
                              timeout: 1s
                            cache_duration: { seconds: 300 }
                      rules:
                        - match: { prefix: "/api/" }
                          requires: { provider_name: platform_idp }
                  # 2. Lua filter for custom header normalization
                  - name: envoy.filters.http.lua
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.LuaPerRoute
                  # 3. Router — must always be last
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Note: In Envoy, filter order within http_filters is the execution order — JWT auth before Lua before router. Placing the router before a security filter silently bypasses it for all traffic.

In Kong 3.x, the data plane executes plugins ordered by their integer PRIORITY value (highest runs first). The built-in pre-function plugin runs at priority 1000000, request-validator at 999, rate-limiting at 910, and request-transformer at 801. Operators cannot override these values in declarative config — only custom plugins may declare a custom priority. Understanding this ordering is essential before designing authentication proxying and token validation pipelines.

Request Lifecycle and Routing Decision Model

A request’s journey through the gateway data plane has four distinct phases. Understanding which middleware belongs in each phase prevents the most common performance and correctness bugs.

Phase 1 — Pre-routing: Authentication, schema validation, and rate limiting and throttling all belong here. Short-circuiting at this stage prevents downstream compute waste. An unauthorized or malformed request must never consume transformation resources.

Phase 2 — Routing: Upstream resolution, load balancing algorithm selection, and circuit breaker evaluation. The gateway resolves which backend handles this request based on weighted targets, consistent-hash rules, or service registry lookups.

Phase 3 — Post-routing: Payload normalization and header enrichment happen here. Request and response transformation — including protocol translation from gRPC to REST, JSON field remapping, or VTL template rendering — runs after routing because it may need to inspect upstream metadata injected by the router.

Phase 4 — Response: Caching and response optimization write to or serve from the cache layer after upstream response. Telemetry exporters flush asynchronously to avoid blocking the response path.

The following Kong 3.x declarative config shows this ordering via plugin priorities:

# Kong 3.x — Phase-ordered plugin pipeline
_format_version: "3.0"
services:
  - name: upstream-api
    url: https://backend.internal:8443
    routes:
      - name: v1-route
        paths: ["/api/v1"]
        strip_path: false
    plugins:
      # Phase 1 — Pre-routing: priority 999 (runs before rate-limiting at 910)
      - name: request-validator
        config:
          body_schema: '{"type":"object","required":["id"],"properties":{"id":{"type":"string"}}}'
          parameter_schema:
            - name: Authorization
              in: header
              required: true
              schema: '{"type":"string"}'
      # Phase 1 — Pre-routing: priority 910
      - name: rate-limiting
        config:
          minute: 1000
          policy: redis
          redis_host: redis.internal
          redis_port: 6379
          fault_tolerant: true
      # Phase 3 — Post-routing: priority 801
      - name: request-transformer
        config:
          add:
            headers:
              - "X-Gateway-Version: kong-3"
              - "X-Request-ID: $(uuid)"
          remove:
            headers:
              - "X-Internal-Debug"

Note: The body_schema value in Kong’s request-validator must be a JSON string — not an inline YAML object. Inline YAML will silently pass schema validation on config load but fail at runtime when Kong attempts to parse the string.

Policy Enforcement Patterns

Effective policy enforcement requires three classes of middleware to cooperate: authentication, access control, and payload governance. Each class has distinct state requirements and failure semantics.

Authentication and Token Validation

Implementing JWT validation in Kong plugins demonstrates the full validation pipeline, but the key invariants are gateway-independent: the gateway must verify the token signature using cached JWKS (re-fetched on 401 from upstream), validate exp, iss, and aud claims before forwarding, and inject downstream identity headers (X-Consumer-ID, X-Forwarded-User) for backend services that trust the gateway.

Avoid forwarding raw Authorization headers to backends that do not perform their own validation — strip and replace with a signed internal identity assertion instead. This boundary aligns with the zero-trust security model where no service implicitly trusts another based on network position alone.

Rate Limiting with Distributed Counters

Local in-memory counters fail silently in multi-node deployments: each gateway pod maintains its own counter, and the effective rate limit becomes limit × pod_count. Production rate limiting requires a shared backend — Redis or Valkey — with atomic increment operations. The dynamic rate limiting with Redis backends pattern uses INCR + EXPIRE with a Lua script to guarantee atomicity without a distributed lock:

-- Redis Lua script: atomic sliding-window counter
local key    = KEYS[1]
local window = tonumber(ARGV[1])   -- seconds
local limit  = tonumber(ARGV[2])
local now    = tonumber(ARGV[3])   -- Unix ms

redis.call('ZREMRANGEBYSCORE', key, '-inf', now - (window * 1000))
local count = redis.call('ZCARD', key)
if count < limit then
  redis.call('ZADD', key, now, now)
  redis.call('PEXPIRE', key, window * 1000)
  return 1   -- allowed
end
return 0     -- rejected

Configure fault_tolerant: true in Kong’s rate-limiting plugin so that Redis connectivity failures fail open (allow traffic) rather than taking down the gateway. Decide per environment whether failing open or closed is the right default.

CORS and Preflight Handling

Configuring CORS policies for multi-tenant APIs covers the full implementation, but one architectural point warrants emphasis here: CORS preflight (OPTIONS) requests must bypass heavy transformation chains entirely. Routing OPTIONS traffic through JWT validation, schema validation, and payload transformation adds latency and commonly produces 401 responses that the browser interprets as a CORS failure — when the real issue is a misplaced auth filter.

// AWS API Gateway — gateway response CORS + request validator
{
  "gatewayResponse": {
    "responseParameters": {
      "gatewayresponse.header.Access-Control-Allow-Origin": "'https://app.example.com'",
      "gatewayresponse.header.Access-Control-Allow-Headers": "'Content-Type,Authorization,X-Request-ID'",
      "gatewayresponse.header.Access-Control-Allow-Methods": "'GET,POST,PUT,DELETE,OPTIONS'"
    },
    "responseTemplates": {
      "application/json": "{\"message\":\"$context.error.messageString\"}"
    }
  },
  "requestValidator": {
    "validateRequestBody": true,
    "validateRequestParameters": true
  }
}

Note: Access-Control-Allow-Origin must be a specific origin — not * — whenever Access-Control-Allow-Credentials: true is set. Browsers reject the wildcard-plus-credentials combination with a hard CORS failure.

Deployment Topologies

The choice of gateway deployment topology directly determines which middleware patterns are viable.

For centralized ingress deployments, plugin state (rate-limit counters, session tokens) must be externalized to Redis or a similar store the moment you run more than one gateway node. For sidecar deployments, each filter chain is independent — shared state requires an explicit call to a sidecar-external store, adding network latency to the critical path. The scaling limits and capacity planning guidance covers how to size these external stores relative to peak request volume.

Observability and Operational Telemetry

A middleware chain that fails silently is operationally worse than one that fails loudly. Every phase boundary should emit a structured log entry and a span, so operators can reconstruct the exact execution path for any failing request.

OpenTelemetry integration: Configure the gateway to inject traceparent and tracestate headers (W3C Trace Context) at the pre-routing phase, before any fork into async processing. Downstream services then propagate these headers, and the collector assembles a complete distributed trace.

# Envoy 1.32+ — OpenTelemetry tracing provider
tracing:
  provider:
    name: envoy.tracers.opentelemetry
    typed_config:
      "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
      grpc_service:
        envoy_grpc:
          cluster_name: otel_collector
      service_name: api-gateway
      resource_detectors:
        - name: envoy.tracers.opentelemetry.resource_detectors.environment
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.tracers.opentelemetry.resource_detectors.v3.EnvironmentResourceDetectorConfig

Structured logging fields to emit at each phase boundary:

• phase — pre-routing, routing, post-routing, response
• plugin_name — identifies which middleware handler produced the log line
• duration_ms — wall time for this phase, not cumulative
• short_circuit — boolean, true if this plugin terminated the request
• upstream_cluster — populated after routing resolution
• trace_id — from the injected traceparent header

Key metrics per phase:

For Kong, the OpenTelemetry plugin exports spans per plugin execution. Combine it with the http-log plugin targeting a local Fluent Bit sidecar to achieve both trace and log correlation without adding network hops to the critical path.

Failure Modes and Resilience Patterns

Circuit Breaker

A circuit breaker prevents a degraded upstream from consuming thread pool capacity across the gateway fleet. When consecutive errors exceed the threshold, the circuit opens and the gateway short-circuits requests with a 503 without forwarding them, allowing the upstream time to recover.

# Envoy 1.32+ — Circuit breaker thresholds per priority
clusters:
  - name: payment_backend
    connect_timeout: 0.5s
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          max_connections: 500
          max_pending_requests: 200
          max_requests: 1000
          max_retries: 3
          track_remaining: true
    outlier_detection:
      consecutive_5xx: 5
      interval: 10s
      base_ejection_time: 30s
      max_ejection_percent: 50
      enforcing_consecutive_5xx: 100
      success_rate_minimum_hosts: 3

Retry Budget

Unbounded retries under failure amplify load on a degraded upstream — the thundering herd problem. A retry budget caps total retries across all replicas as a fraction of total requests:

# Envoy 1.32+ — retry policy with budget (prevents thundering herd)
routes:
  - match: { prefix: "/api/payments" }
    route:
      cluster: payment_backend
      retry_policy:
        retry_on: "5xx,reset,connect-failure"
        num_retries: 2
        per_try_timeout: 1s
        retry_back_off:
          base_interval: 100ms
          max_interval: 1s
        retry_host_predicate:
          - name: envoy.retry_host_predicates.previous_hosts
        host_selection_retry_max_attempts: 3

Set retry_on: "5xx,reset,connect-failure" rather than the broader "retriable-4xx" to avoid retrying 429 responses from rate-limited upstreams — which would double the load on an already-overloaded service.

Thundering Herd on Cache Miss

When the cache is cold (after a restart, a cache flush, or TTL expiration of a popular key), simultaneous requests for the same resource all miss the cache and simultaneously hit the upstream. Mutex coalescing — serializing cache-fill requests for the same key — prevents this. In Kong, the proxy-cache plugin handles this at the plugin level. In NGINX, proxy_cache_lock on with proxy_cache_lock_timeout achieves the same effect:

# NGINX — cache lock to prevent thundering herd on cache miss
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:10m max_size=1g;

server {
  location /api/ {
    proxy_cache          api_cache;
    proxy_cache_lock     on;
    proxy_cache_lock_timeout 5s;
    proxy_cache_valid    200 60s;
    proxy_cache_use_stale error timeout updating;
    proxy_pass           http://backend_pool;
  }
}

proxy_cache_use_stale updating serves the stale entry while the cache-fill request is in flight — eliminating the lock wait for subsequent requests when the upstream is slow.

Plugin Timeout Cascades

External plugin calls — JWKS fetches, external auth sidecar calls, remote policy evaluations — must have hard timeouts bounded well below the client’s own request timeout. A plugin that hangs for 30 seconds will exhaust worker threads even when the upstream is healthy. Configure per-plugin timeouts in Kong with the instance_name timeout config, or use Envoy’s per_try_timeout in ext_authz filter configuration.

Implementation Blueprint

Technical Validation Checklist

Use this checklist before promoting a middleware pipeline configuration to production:

• Auth plugins execute before transformation plugins — verified by checking Kong plugin priorities or Envoy http_filters order.
• OPTIONS preflight requests bypass JWT validation and schema validation.
• Rate-limit counters are backed by Redis (or equivalent) — not local memory — on multi-node deployments.
• Redis connectivity failures in rate-limiting plugins are configured to fail open (fault_tolerant: true), with the decision documented.
• body_schema in Kong request-validator is a JSON string, not an inline YAML object.
• CORS Access-Control-Allow-Origin is a specific origin — not * — whenever credentials are in use.
• W3C Trace Context headers (traceparent, tracestate) are injected or propagated at pre-routing phase.
• Circuit breaker thresholds are set per upstream, not globally.
• Retry policy excludes retriable-4xx to avoid re-hitting rate-limited or auth-rejected upstreams.
• Cache keys include all request attributes that affect the response — method, path, relevant headers, query params.
• proxy_cache_lock on (NGINX) or equivalent mutex coalescing is enabled to prevent thundering herd on cache miss.
• Plugin-level timeouts are set below the client-facing request timeout by at least 20%.
• Structured log entries include phase, plugin_name, duration_ms, and trace_id for every middleware boundary.
• Deployment topology (centralized ingress vs sidecar) matches the stated scaling and isolation requirements for this service.

FAQ

How does middleware execution order impact API gateway latency?

Execution order determines the critical path length. Security and routing checks should run first to short-circuit invalid requests early. Heavy transformations placed at the start of the chain increase baseline latency for all traffic — including traffic that will later be rejected. Deferring them to post-routing phases optimizes throughput but complicates error handling when the transformation itself fails.

Can stateful middleware scale horizontally in a distributed gateway?

Only with externalized state management. Local in-memory state breaks consistency across nodes — each pod maintains its own counter, session store, or cache, and they diverge immediately under real traffic. Production deployments require distributed stores (Redis, Valkey, Memcached) or consistent hashing to route state-bearing requests to the same node. Both introduce network latency and require careful timeout tuning.

What are the primary trade-offs between synchronous and asynchronous middleware chains?

Synchronous chains guarantee execution order and simplify error propagation — a plugin that returns an error terminates the chain and the caller gets a deterministic response. They block worker threads, reducing maximum concurrency per node. Asynchronous chains improve throughput and resource utilization by yielding the thread during I/O waits. They complicate debugging because errors propagate via callbacks or promises rather than stack unwinding.

How do you prevent cache poisoning when combining transformation and caching middleware?

Run transformation middleware before cache lookup so the cache stores the normalized, transformed representation. Include all request attributes that affect the output in the cache key — not just the URL. Explicitly list Vary headers and exclude sensitive fields (e.g., Authorization, Cookie) that should not partition the cache. Tie cache invalidation to upstream ETag or Cache-Control: no-store headers so stale transformed payloads cannot be served after a schema change.

Related

• Authentication Proxying & Token Validation — JWT validation pipeline, JWKS caching, and downstream identity header injection.
• Circuit Breaking & Retry Budgets — outlier detection, retry budgets, and backoff that keep a transient upstream blip from amplifying.
• Request & Response Transformation — JSON payload rewriting, header manipulation, and template-based transformation patterns.
• Caching & Response Optimization — cache-key design, TTL strategies, and stale-while-revalidate patterns at the gateway layer.
• Gateway Selection Criteria — capability matrix comparing Kong, Envoy, Tyk, and NGINX across plugin extensibility, performance, and operational overhead.
• Security Boundaries & Zero-Trust — mTLS enforcement, identity propagation, and perimeter-free network policy at the API layer.

← Home