Request & Response Transformation

Request and response transformation sits at the heart of the Middleware Chains & Request Transformation layer: it is the mechanism by which API gateways decouple what clients send from what backends expect, and what backends return from what clients understand. Without it, every schema change, protocol difference, or multi-tenant contract variation forces a code change in every service. With it, a declarative gateway configuration absorbs the mismatch at the edge, letting downstream services stay stable while the client-facing API evolves independently.

This page covers the core transformation patterns — header manipulation, body rewriting, JSON-to-XML conversion, schema validation — and shows runnable configurations for Kong 3.x, Envoy 1.32+, Tyk 5.x, and NGINX. It also covers the critical sequencing decisions, operational failure modes, and production configuration checklist that prevent transformation pipelines from becoming a reliability liability.

Architectural Baseline

Before writing a single transformation rule, engineers need a clear mental model of where transformation fits inside the gateway’s request lifecycle.

The diagram above shows the canonical gateway phase ordering. Authentication must run at phase 2 to establish the security boundary; the full details of JWT claim extraction live in authentication proxying and token validation. Rate limiting and throttling at phase 3 operates on the verified identity, not raw client IP. Transformation runs at phase 4 on an already-authorized, already-budgeted request.

Two failure modes arise from violating this ordering. First, placing transformation before authentication lets an attacker send a crafted body that manipulates routing keys or injects headers before any credential check occurs. Second, placing transformation before rate limiting means the quota counter sees the original path or tenant key, not the transformed one — quota attribution breaks silently.

Mental model for body transformation: treat the gateway worker as a stateless stream processor. The request body arrives as a byte stream, passes through a parse → mutate → serialize cycle, and exits as a new byte stream. Any step that requires full-document materialization — deep JSONPath traversal, schema validation, XML XPath — buffers the entire body and introduces a memory high-water mark proportional to the payload size, a key input into capacity planning for your gateway. Streaming parsers avoid this for append-only mutations (header injection, field appending) but cannot avoid it for field deletion or reordering.

Primary Deep-Dive: Header and Body Transformation

Kong 3.x — request-transformer and response-transformer

Kong exposes transformation through two complementary plugins. The request-transformer plugin handles the inbound side; response-transformer handles what the upstream returns. Both are available in Kong OSS.

# Kong 3.x — declarative config (kong.yml)
# Pin: Kong 3.6+ required for json_body support in response-transformer
_format_version: "3.0"

services:
  - name: payments-service
    url: http://payments-backend:8080

routes:
  - name: payments-route
    service: payments-service
    paths:
      - /v1/payments

plugins:
  # ── Inbound: remap, inject, remove ──────────────────────────────────────
  - name: request-transformer
    route: payments-route
    config:
      add:
        headers:
          - "X-Tenant-ID:$(headers.Authorization | jwt_claim.tenant)"
          - "X-Request-Source:api-gateway"
        querystring:
          - "version:2"
      rename:
        headers:
          - "X-Legacy-Auth:Authorization"
      remove:
        headers:
          - "X-Internal-Debug"
        querystring:
          - "debug"
      replace:
        body:
          # rename top-level JSON key (dot-notation only — OSS limit)
          - "userId:metadata.tenantUserId"

  # ── Outbound: strip internal headers, normalise response schema ──────────
  - name: response-transformer
    route: payments-route
    config:
      remove:
        headers:
          - "X-Backend-Server"
          - "X-Powered-By"
          - "Server"
      add:
        headers:
          - "Cache-Control:no-store"
          - "X-Content-Type-Options:nosniff"
      rename:
        headers:
          - "X-Correlation-Id:X-Request-ID"

For nested JSON field remapping — e.g., mapping $.user.profile.id to $.metadata.tenant_id — Kong OSS reaches its limit. The request-transformer-advanced plugin (Kong Enterprise 3.x) adds JSONPath support:

-- Kong Enterprise 3.x — custom Lua body transformer
-- Attach as a custom plugin in /usr/local/share/lua/5.1/kong/plugins/body-remap/handler.lua

local cjson = require "cjson.safe"

local BodyRemapHandler = {}
BodyRemapHandler.PRIORITY = 800  -- runs after auth (1000), before proxy (0)
BodyRemapHandler.VERSION  = "1.0.0"

function BodyRemapHandler:access(conf)
  kong.service.request.enable_buffering()
  local body, err = kong.request.get_raw_body()
  if not body then return end

  local data, decode_err = cjson.decode(body)
  if not data then
    kong.log.warn("body-remap: decode failed: ", decode_err)
    return
  end

  -- Move nested field: data.user.id → data.metadata.tenant_id
  if data.user and data.user.id then
    data.metadata = data.metadata or {}
    data.metadata.tenant_id = data.user.id
    data.user.id = nil
  end

  kong.service.request.set_raw_body(cjson.encode(data))
  kong.service.request.set_header("Content-Type", "application/json")
end

return BodyRemapHandler

Note that enable_buffering() must be called before any body read in the access phase. Skipping it causes Kong to return an empty string from get_raw_body() with no error logged — a silent data-loss bug.

Envoy 1.32+ — ext_proc and Lua HTTP filter

Envoy’s native transformation story is split: header manipulation lives in the HeaderToMetadataFilter and LuaFilter; body transformation is best handled via the External Processing (ext_proc) filter or a Lua filter.

# Envoy 1.32+ — listener filter chain for request/response transformation
# Inline Lua filter for header normalization
static_resources:
  listeners:
    - name: main
      address:
        socket_address: { address: 0.0.0.0, port_value: 8080 }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                http_filters:
                  # Phase 1: extract JWT claim into metadata for downstream filters
                  - name: envoy.filters.http.jwt_authn
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
                      providers:
                        tenant_jwt:
                          remote_jwks:
                            http_uri:
                              uri: https://auth.internal/jwks.json
                              cluster: jwks_cluster
                              timeout: 2s
                      rules:
                        - match: { prefix: "/v1/" }
                          requires: { provider_name: tenant_jwt }

                  # Phase 2: Lua header transformation — runs after JWT auth
                  - name: envoy.filters.http.lua
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.LuaPerRoute
                      source_code:
                        inline_string: |
                          function envoy_on_request(request_handle)
                            -- Inject tenant ID from JWT metadata into upstream header
                            local meta = request_handle:streamInfo():dynamicMetadata()
                            local tenant = meta:get("envoy.filters.http.jwt_authn", "tenant_id")
                            if tenant then
                              request_handle:headers():add("X-Tenant-ID", tenant)
                            end
                            -- Strip client-side debug headers
                            request_handle:headers():remove("X-Debug-Flags")
                          end

                          function envoy_on_response(response_handle)
                            -- Remove internal tracing headers before client delivery
                            response_handle:headers():remove("X-Envoy-Upstream-Service-Time")
                            response_handle:headers():remove("server")
                            response_handle:headers():add("X-Content-Type-Options", "nosniff")
                          end

                  # Phase 3: ext_proc for body transformation (separate gRPC service)
                  - name: envoy.filters.http.ext_proc
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
                      grpc_service:
                        envoy_grpc:
                          cluster_name: body_transform_cluster
                      processing_mode:
                        request_body_mode: BUFFERED
                        response_body_mode: STREAMED
                      failure_mode_allow: false  # reject request on processor failure
                      message_timeout: 0.015s    # 15 ms budget

                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager

The failure_mode_allow: false flag is critical: when set to true, a failure in the external processor silently passes the original unmodified request through. In most transformation scenarios this is wrong — a failed body remap should be a hard error, not a silent passthrough that leaks unmapped fields to the backend.

Tyk 5.x — Virtual Endpoints and Go Plugin Transform

Tyk exposes transformation through its API definition’s extended_paths. The transform and transform_response keys use Go template syntax for header and body mutation:

// Tyk 5.x API definition — request and response body transformation
// Version: Tyk Gateway OSS 5.3+
{
  "api_definition": {
    "name": "Payments API",
    "version_data": {
      "not_versioned": true,
      "versions": {
        "Default": {
          "name": "Default",
          "extended_paths": {
            "transform": [
              {
                "path": "/v1/payments",
                "method": "POST",
                "template_data": {
                  "input_type": "json",
                  "enable_session": true,
                  "template_mode": "blob",
                  "blob": "eyJ0ZW5hbnRJZCI6Int7LlNlc3Npb25NZXRhRGF0YS50ZW5hbnRfaWR9fSIsInVzZXJJZCI6Int7LkRhdGEudXNlci5pZH19In0="
                }
              }
            ],
            "transform_response": [
              {
                "path": "/v1/payments",
                "method": "POST",
                "template_data": {
                  "input_type": "json",
                  "template_mode": "blob",
                  "blob": "eyJwYXltZW50SWQiOiJ7ey5EYXRhLmlkfX0iLCJzdGF0dXMiOiJ7ey5EYXRhLnN0YXR1c319In0="
                }
              }
            ],
            "transform_headers": [
              {
                "path": "/v1/payments",
                "method": "POST",
                "act_on": false,
                "add_headers": {
                  "X-Tenant-ID": "$tyk_context.jwt_claims_tenant_id",
                  "X-Request-Source": "tyk-gateway"
                },
                "delete_headers": ["X-Internal-Debug", "X-Forwarded-Host"]
              }
            ],
            "transform_response_headers": [
              {
                "path": "/v1/payments",
                "method": "POST",
                "act_on": true,
                "add_headers": {
                  "Cache-Control": "no-store",
                  "X-Content-Type-Options": "nosniff"
                },
                "delete_headers": ["X-Powered-By", "Server", "X-Backend-Node"]
              }
            ]
          }
        }
      }
    }
  }
}

The base64-encoded blob is a Go template string. Decoded, the request template maps {{.Data.user.id}} to a new userId field and injects the session-level tenant_id. The enable_session: true flag makes Tyk session metadata (populated from JWT claims during auth) available inside the template — without it, $tyk_context.* variables are empty strings.

Secondary Concept: Schema Validation and JSON-to-XML Conversion

Schema Validation at the Gateway Edge

Validating payloads against an OpenAPI 3.x or JSON Schema definition at the gateway edge prevents invalid data from ever reaching backend services. This shifts the schema enforcement left, reduces backend error noise, and allows the gateway to return a structured 400 response with field-level error detail.

Kong 3.x provides the request-validator plugin:

# Kong 3.x — request-validator plugin
# Requires Kong 3.4+ for full draft-07 JSON Schema support
- name: request-validator
  route: payments-route
  config:
    body_schema: |
      {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "type": "object",
        "required": ["amount", "currency", "recipient"],
        "properties": {
          "amount": { "type": "number", "minimum": 0.01, "maximum": 1000000 },
          "currency": { "type": "string", "enum": ["USD", "EUR", "GBP"] },
          "recipient": {
            "type": "object",
            "required": ["account_id"],
            "properties": {
              "account_id": { "type": "string", "pattern": "^[A-Z]{2}[0-9]{16}$" }
            }
          }
        },
        "additionalProperties": false
      }
    parameter_schema:
      - name: X-Idempotency-Key
        in: header
        required: true
        schema: { "type": "string", "minLength": 16, "maxLength": 64 }
    verbose_response: true  # include field path in 400 response body
    allowed_content_types:
      - "application/json"

With verbose_response: true, a payload missing the currency field returns:

{ "message": "request body validation failed: schema violation (currency is required)" }

This level of detail is useful in development but can leak schema internals in production. Consider setting verbose_response: false for externally exposed routes and relying on a generic 400 Bad Request to clients.

JSON-to-XML and Protocol Translation

Legacy backends that speak SOAP or XML are a common driver for protocol translation patterns. A thin JSON-to-XML transformation at the gateway lets REST clients consume these backends without modification.

NGINX with the ngx_http_xslt_module provides XSLT-based response transformation:

# NGINX 1.25+ — JSON-to-XML request conversion via Lua (OpenResty)
# Requires openresty/lua-cjson and openresty/lua-resty-http

http {
    lua_package_path "/usr/local/openresty/lualib/?.lua;;";

    upstream legacy_soap_backend {
        server legacy-payments.internal:8443;
    }

    server {
        listen 443 ssl http2;
        ssl_certificate     /etc/nginx/certs/gateway.crt;
        ssl_certificate_key /etc/nginx/certs/gateway.key;

        location /v1/payments {
            # Buffer request body for Lua transformation
            lua_need_request_body on;
            client_max_body_size 1m;

            # Transform JSON → SOAP XML before proxying
            rewrite_by_lua_block {
                local cjson = require "cjson.safe"
                local body = ngx.req.get_body_data()
                if not body then
                    ngx.status = 400
                    ngx.say('{"error":"empty request body"}')
                    return ngx.exit(400)
                end

                local data, err = cjson.decode(body)
                if not data then
                    ngx.status = 400
                    ngx.say('{"error":"invalid JSON"}')
                    return ngx.exit(400)
                end

                -- Build SOAP envelope
                local soap = string.format([[
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                  xmlns:pay="http://payments.internal/v1">
  <soapenv:Body>
    <pay:ProcessPayment>
      <pay:Amount>%s</pay:Amount>
      <pay:Currency>%s</pay:Currency>
      <pay:RecipientId>%s</pay:RecipientId>
    </pay:ProcessPayment>
  </soapenv:Body>
</soapenv:Envelope>]],
                    data.amount or 0,
                    data.currency or "USD",
                    (data.recipient and data.recipient.account_id) or "")

                ngx.req.set_body_data(soap)
                ngx.req.set_header("Content-Type", "text/xml; charset=utf-8")
                ngx.req.set_header("SOAPAction", "ProcessPayment")
                ngx.req.set_header("Content-Length", #soap)
            }

            proxy_pass https://legacy_soap_backend;
            proxy_set_header Host legacy-payments.internal;

            # Transform SOAP XML response → JSON after proxying
            header_filter_by_lua_block {
                ngx.header["Content-Type"] = "application/json"
                ngx.header["content-length"] = nil  -- re-set after body transform
            }

            body_filter_by_lua_block {
                -- For production: use lua-resty-simplexml or a full XML parser
                local chunk = ngx.arg[1]
                if chunk and ngx.arg[2] then
                    -- Extract value between <pay:PaymentId> tags
                    local id = chunk:match("<pay:PaymentId>([^<]+)</pay:PaymentId>") or "unknown"
                    local status = chunk:match("<pay:Status>([^<]+)</pay:Status>") or "unknown"
                    ngx.arg[1] = string.format('{"paymentId":"%s","status":"%s"}', id, status)
                end
            }
        }
    }
}

For rewriting JSON payloads on the fly in high-throughput environments, prefer streaming approaches over full-body materialization — the linked page covers memory management and GC trade-offs in detail.

Comparative Implementation Table

Operational Gotchas

1. Silent passthrough on processor failure (Envoy ext_proc)

Setting failure_mode_allow: true on an ext_proc filter causes Envoy to forward the original, untransformed request when the external processor is unreachable or returns an error. In a schema-remapping scenario, the upstream service receives unmapped fields and may store corrupt data with no log entry at the gateway. Always set failure_mode_allow: false and handle processor unavailability with a circuit breaker on the body_transform_cluster.

Remediation: Monitor ext_proc health with the envoy_cluster_upstream_cx_connect_fail_total metric. Alert when the error rate on the transform cluster exceeds 0.1% over a 1-minute window.

2. Body buffering unexpectedly enabled in Kong

Kong’s request-transformer plugin implicitly enables body buffering for application/json routes. If a route serves large file uploads via multipart/form-data and a transformation plugin is attached, Kong will attempt to buffer the entire multipart body into memory. The symptom is sudden RSS growth in Kong worker processes under upload load.

Remediation: Attach transformation plugins only to routes that genuinely need them. Use config.content_type_whitelist: ["application/json"] if the plugin supports it, or split upload routes from API routes and apply plugins selectively.

3. Go template injection in Tyk transform blobs

Tyk’s transform blob uses Go’s text/template package, not html/template. Values interpolated from JWT claims or request body fields are not HTML-escaped. An attacker who controls a JWT claim value can inject arbitrary characters into the transformed body, including double-quotes that break JSON structure or XML special characters.

Remediation: Always validate and sanitize values before interpolation. Use a Go plugin instead of a template blob for any field that originates from client-controlled input.

4. Content-Length mismatch after body mutation

When a plugin mutates the request body but does not update the Content-Length header, the upstream service may truncate the body (if the new length is shorter) or hang waiting for more bytes (if the new length is longer). Kong and Tyk handle this automatically for their built-in transform plugins. With custom Lua or NGINX body_filter_by_lua_block, you must explicitly set Content-Length or use Transfer-Encoding: chunked.

Remediation: In NGINX Lua body filters, set ngx.header["content-length"] = nil in the header_filter_by_lua_block to force chunked transfer, then let the upstream connection handle framing.

5. CORS preflight interception before transformation

Browsers send OPTIONS preflight requests that must be handled by the CORS and cross-origin security layer before reaching the transformation phase. If transformation plugins run on OPTIONS routes and attempt to parse an empty body, they throw decode errors and the preflight fails, breaking every cross-origin request from that client.

Remediation: Add a route-level exception for the OPTIONS method that bypasses body transformation plugins entirely, or configure transformation plugins with config.http_method: ["POST","PUT","PATCH"] where supported.

FAQ

What is the difference between request transformation and response transformation at an API gateway?

Request transformation mutates inbound payloads, headers, and query parameters before they reach the upstream service. Response transformation mutates what the upstream service returns before it reaches the client. Both operate within the gateway’s middleware pipeline and can be applied independently per route.

When should transformation run relative to authentication and rate limiting?

Authentication must run first to establish the security context, then rate limiting consumes the verified identity, then transformation runs on an already-authorized request. Transformation before auth allows untrusted data to influence routing decisions and can bypass quota attribution.

How do streaming parsers help with large payload transformation?

Streaming parsers process a payload incrementally as bytes arrive rather than buffering the entire body into memory. This keeps peak memory consumption low for large JSON or multipart bodies and reduces head-of-line blocking in the gateway worker pool.

Can Kong’s request-transformer plugin modify nested JSON fields?

The built-in request-transformer plugin supports dot-notation for top-level key renaming but not deep JSONPath traversal. For nested field remapping you need the request-transformer-advanced plugin (Kong Enterprise) or a custom Lua plugin that calls cjson.decode / cjson.encode around the body.

Production Configuration Checklist

• Transformation plugins run after auth plugins in the execution priority chain (Kong: plugin priority < 1000; Envoy: Lua filter defined after jwt_authn in the filter chain; Tyk: middleware order set in extended_paths)
• failure_mode_allow is set to false on all Envoy ext_proc filters used for body transformation
• Content-Length is either stripped (chunked transfer) or recalculated after every body mutation
• Schema validation is enabled on all externally exposed POST/PUT/PATCH routes with additionalProperties: false to reject undeclared fields
• OPTIONS preflight routes are explicitly excluded from body-parsing transformation plugins
• Go template blobs in Tyk do not interpolate client-controlled values without prior validation
• Response transformation strips internal headers (X-Powered-By, Server, X-Backend-Node) on every route
• Buffer limits are set explicitly (client_max_body_size in NGINX; max_buffer_size in Kong’s global config) to cap per-request memory under transformation
• Transformation latency is tracked as a named span in OpenTelemetry traces (span name: gateway.transform.request / gateway.transform.response)
• Circuit breakers are configured on external transform processor clusters (ext_proc, WASM runtime) with a failure threshold ≤ 5% over 30 seconds

Related

• Rewriting JSON Payloads on the Fly — memory management, streaming parse strategies, and GC trade-offs for high-throughput body mutation
• Migrate a SOAP Endpoint to REST via Gateway Transformation — an end-to-end runbook for fronting a legacy SOAP backend with a REST/JSON facade
• Caching & Response Optimization — how response transformation interacts with cache key derivation and Vary header management
• Kong vs Tyk vs Envoy for Microservices — capability comparison when transformation depth is a deciding factor
• High Availability Topologies — how transformation pipeline failures propagate in active-active vs active-passive gateway deployments
• Implementing JWT Validation in Kong Plugins — extracting JWT claims that feed into transformation header injection

Up: Middleware Chains & Request Transformation