mTLS Handoff Between Gateway and Mesh

Running a gateway in front of a mesh means a request is encrypted, decrypted and re-encrypted at least twice before it reaches an application. That is normal and correct — but only if each hop’s identity is the right one and the receiving side knows what it is trusting. This page covers the handoff at the gateway-to-sidecar boundary, the double-encryption trap, and how to make an upstream’s authorisation decision meaningful rather than decorative.

Prerequisite concepts

This assumes implementing mTLS at the gateway edge for the client-facing handshake and API gateway vs service mesh for the layer split.

Three identities on one request

Three identities, verified in three different places The end user is identified by a bearer token the gateway validates. The gateway itself is identified to the sidecar by a workload certificate. The application then authorises using the propagated user identity, trusting the gateway assertion only because the sidecar verified which workload sent it. client bearer token gateway validates token, holds a cert sidecar verifies the workload service user identity travels the whole way as a claim; workload identity is re-established at each hop The service can trust the forwarded user claim precisely because the sidecar proved which workload asserted it — that chain is what separates a real authorisation decision from one based on a header anyone could set. Break any link and the remaining checks become theatre rather than security. 
# Envoy 1.32+ gateway — present a workload certificate to the mesh
clusters:
  - name: orders_service
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          tls_certificate_sds_secret_configs:
            - name: "spiffe://cluster.local/ns/gateway/sa/edge"   # the gateway's own identity
              sds_config: { api_config_source: { api_type: GRPC } }
          combined_validation_context:
            default_validation_context:
              match_typed_subject_alt_names:
                - san_type: URI
                  matcher: { exact: "spiffe://cluster.local/ns/orders/sa/orders" }

The match_typed_subject_alt_names block is the part that is usually missing. Without it the gateway accepts any certificate the mesh certificate authority issued — which is every workload in that Kubernetes cluster, including one an attacker got a foothold in.

The double-encryption trap

Two layers both originating TLS produces a tunnel, not security If the gateway originates mutual TLS to the pod address while the sidecar also expects to originate mesh TLS, the outer session becomes opaque to the sidecar. Mesh policy, telemetry and routing all stop applying because the sidecar sees encrypted bytes rather than an HTTP request. wrong: gateway originates TLS past the sidecar gateway → (encrypted to the app) → sidecar sees opaque bytes → no policy, no telemetry, no routing symptom: mesh dashboards show no traffic for a service that is plainly serving requests right: gateway terminates at the sidecar, sidecar re-originates gateway → mesh mTLS to the sidecar → sidecar decrypts, applies policy, forwards on loopback one encryption per hop, and every hop can do its job Address the Service, not the pod IP: the sidecar interception depends on the destination the connection targets.

The practical rule is that each hop terminates and re-originates. A gateway that opens a TLS session to the application port rather than to the mesh-managed port has built a tunnel through the mesh, and the mesh will faithfully carry bytes it cannot see.

Making the assertion trustworthy

Once the gateway is a known workload, the user claim it forwards can be trusted — provided the same header cannot be set by anyone else. That requires two things: the gateway strips the header from inbound requests, and the receiving service accepts it only from the gateway’s identity.

Accept the assertion only from the workload allowed to make it The service's authorisation policy names the gateway's SPIFFE identity as the only principal permitted to call it. Any other workload presenting the same forwarded user header is rejected at the sidecar, so a compromised service elsewhere in the mesh cannot impersonate a user by setting a header. accepted principal: spiffe://…/ns/gateway/sa/edge x-user-sub: 8f21 — trusted the gateway verified the token itself rejected at the sidecar principal: spiffe://…/ns/reports/sa/batch x-user-sub: 8f21 — ignored not permitted to assert a user identity Without this policy the header is an assertion anyone in the mesh can make, and the gateway's token validation protects the front door while leaving every internal path open.

Decision matrix

Situation Choice
Gateway inside the mesh run it with a sidecar, or as a mesh-native gateway
Gateway outside the mesh issue it a workload identity from the mesh CA explicitly
Upstream not yet meshed terminate at the gateway, plan the sidecar as the next step
Compliance requires end-to-end encryption per-hop mTLS satisfies it; nested tunnels usually do not
Multiple clusters federate trust domains rather than sharing a CA key

Gotchas and failure signals

Mesh dashboards showing no traffic for a service that is clearly serving requests is the double-encryption signature.

Certificate rotation is on a different clock at each layer. A gateway holding a long-lived certificate against a mesh rotating hourly will fail at a boundary nobody scheduled.

Trust-domain federation is not certificate sharing. Copying a CA key between clusters creates one trust domain with two operators and no way to revoke half of it.

A gateway that accepts any mesh-issued certificate has authenticated the trust domain, not the service. Always match the expected subject alternative name.

Rotation, and the failure it causes at 03:00

Certificate lifetimes are the part of this design most likely to be inherited rather than chosen. A mesh typically issues workload certificates with lifetimes measured in hours and rotates them automatically; a gateway provisioned outside the mesh often holds a certificate measured in months, issued by a different process, renewed by a different team.

That asymmetry is survivable while both sides are healthy and becomes an outage the first time one of them is not. The specific failure is a gateway whose certificate expires between deploys: nothing in the deployment pipeline touched it, no alert fired because the monitoring watched the public-facing certificate rather than the internal one, and every upstream call fails at once with a TLS error that reads like a network fault.

Two habits prevent it. Issue the gateway’s workload identity from the same authority and on the same rotation schedule as everything else in the mesh, even when the gateway itself runs outside the mesh — the point is a single expiry regime, not a single deployment model. And alert on remaining certificate lifetime for every certificate in the path, not merely the one clients see, with a threshold well above one rotation interval so a failed renewal has time to be noticed by a person.

Validation

  • Gateway presents a workload identity and the upstream matches its exact SAN
  • Sidecar reports the gateway as the source principal in its telemetry
  • Forwarded user headers stripped from inbound requests at the edge
  • Authorisation policy names the gateway identity as the only permitted asserter
  • Certificate lifetimes and rotation intervals aligned across both layers

FAQ

Should the gateway terminate TLS and re-originate to the mesh?

Yes. Each hop terminates and re-originates, so the gateway decrypts the client session, applies policy, and opens a new mutual TLS session to the sidecar using its own workload identity. A gateway that instead opens a session straight through to the application port has tunnelled through the mesh, and the sidecar can no longer apply policy, emit telemetry or route.

How do I know if I have accidentally double-encrypted?

The mesh dashboards show little or no traffic for a service that is plainly serving requests, because the sidecar sees opaque bytes rather than HTTP. Check that the gateway addresses the Service rather than a pod IP, since sidecar interception depends on the destination the connection targets.

Is validating that the certificate came from the mesh CA enough?

No. Every workload in that Kubernetes cluster holds a certificate from that CA, so accepting any of them authenticates the trust domain rather than the peer. Match the exact subject alternative name of the workload you expect, so a foothold in an unrelated service does not become a valid caller.

How does the upstream know it can trust a forwarded user header?

Because the sidecar proved which workload sent it. Strip the header from inbound requests at the edge so no client can set it, then write an authorisation policy naming the gateway identity as the only principal permitted to assert a user. Without that policy the header is an assertion any workload in the mesh can make.

Parent: API Gateway vs Service Mesh