API Gateway vs Service Mesh

The question “should we use an API gateway or a service mesh?” is almost always a false binary. The two components govern different traffic axes and different trust boundaries, and the moment a platform grows past a handful of services the honest answer becomes “both, at different layers.” This page sits inside API gateway fundamentals and architecture and draws the line precisely: where an edge L7 gateway belongs, where a sidecar mesh belongs, how their overlapping capabilities — mTLS, retries, routing, observability — differ in intent even when they look identical on a config diff, and how to compose the two without paying twice for the same policy. The failure mode this page exists to prevent is the accidental double-proxy: two layers retrying the same call, two layers terminating and re-terminating TLS with no added guarantee, and two control planes disagreeing about which one owns a routing decision.

The mental model: two traffic axes, two trust boundaries

The cleanest way to reason about this is to stop thinking about products and start thinking about traffic direction.

North-south traffic crosses the perimeter of your system. It originates outside your trust boundary — a mobile app, a partner integration, a public API consumer — and terminates at a service you own. North-south is where you enforce consumer-facing concerns: who is this caller, do they have an API key, are they within their quota, does their request match your published contract, and should their payload be reshaped before it touches an internal service. This is the API gateway’s home. The gateway is a small number of fleet-managed L7 processes at the edge, and it is the one place where a request from an untrusted network becomes a request the rest of the platform is willing to trust.

East-west traffic flows between services you already own, inside the trust boundary. Service A calls service B calls service C. Here the concerns are different: establish cryptographic workload identity between peers, load-balance across healthy replicas, retry transient failures close to the source, eject sick hosts, and emit a coherent trace across every hop. These concerns apply uniformly to every service and should not require each team to reimplement them in application code. This is the service mesh’s home. The mesh data plane is a sidecar proxy co-located with every service instance, transparently intercepting inbound and outbound connections, driven by a central control plane that distributes identity, routing, and policy.

The trust boundaries map onto the axes. A gateway sits on the boundary and its job is to be a hard, auditable checkpoint between untrusted and trusted networks — the natural place to implement security boundaries and zero-trust enforcement at ingress. A mesh operates entirely within the boundary and its job is to make “trusted internal network” mean something cryptographic rather than positional: every hop authenticates the identity of its peer instead of assuming that anything on the internal network is friendly.

Topology: gateway at the edge, mesh for east-west

The diagram below shows the combined topology. External clients enter through the gateway (north-south), which authenticates them, applies edge policy, and hands the request to the mesh. Inside the mesh, every service-to-service call (east-west) traverses a pair of sidecars that enforce mTLS and resilience policy.

Overlapping capabilities that mean different things

The reason this comparison confuses teams is that both layers advertise the same feature list — mTLS, routing, retries, load balancing, observability — yet each capability has a different intent depending on where it runs.

mTLS. At the gateway, mTLS authenticates an external caller: a partner presents a client certificate and the gateway validates it against a trusted CA before admitting the request. In the mesh, mTLS establishes workload identity between internal peers using short-lived certificates the control plane rotates automatically — the mesh’s own identity system, not your partner PKI. These are not redundant. The detailed patterns for the edge case belong to implementing mTLS at the gateway edge; the mesh case is about giving every hop a cryptographic name.

Routing. The gateway routes by consumer-facing contract: API version, path, host, API key, or JWT claim. It is the correct place for multi-tenant routing strategies because tenant identity is a consumer concept resolved at the edge. The mesh routes by internal concerns: subset selection for canaries, locality-aware load balancing, and traffic shifting between service versions. Mesh routing does not know or care about your API keys.

Retries and resilience. Both can retry, and this overlap is the single most dangerous one — retries compound multiplicatively across layers. The mesh sidecar is the right owner for east-west retries and outlier detection because it sits next to the upstream. The gateway should retry only its immediate upstream, if at all. Get this wrong and one upstream blip becomes an amplified storm, which is exactly the failure mode covered in circuit breaking and retry budgets.

Observability. The gateway is the source of truth for external request metrics — consumer, quota, edge latency. The mesh is the source of truth for the internal call graph — the service dependency map and per-hop latency. They must share a trace context so a single traceparent stitches the edge span to every mesh span, a concern owned end to end by gateway observability and operations.

The gateway also owns the middleware chain — API-key auth, rate limiting, CORS, request and response transformation — that has no natural home in a mesh, which is transparent to application-level payload semantics by design.

Config in each layer: mTLS and retries side by side

The clearest way to see the split is to configure the same two concerns — mutual TLS and a retry policy — in an edge gateway and in a mesh, and notice how the intent differs even where the syntax rhymes.

Kong 3.x at the edge (north-south)

At the edge, Kong terminates external TLS, optionally requires a client certificate from partner callers, and applies a conservative retry to its immediate upstream. The following declarative config attaches the mtls-auth plugin and sets service-level retry and timeout behavior.

# Kong 3.x — edge gateway: client-cert auth + conservative upstream retry
_format_version: "3.0"

certificates:
  - cert: /etc/kong/partner-ca.crt   # CA that signs partner client certs
    id: partner-ca

ca_certificates:
  - cert: /etc/kong/partner-ca.crt
    id: partner-ca

services:
  - name: orders-api
    url: https://orders.mesh.internal:443
    # Kong retries the immediate upstream only; NOT the whole call graph
    retries: 1
    connect_timeout: 2000
    write_timeout: 5000
    read_timeout: 5000

    routes:
      - name: orders-v2
        paths: ["/v2/orders"]
        strip_path: true

    plugins:
      # Require and validate a partner client certificate at the edge
      - name: mtls-auth
        config:
          ca_certificates: ["partner-ca"]
          revocation_check_mode: STRICT
          skip_consumer_lookup: false

      # Consumer-facing rate limit — a mesh has no equivalent of this
      - name: rate-limiting
        config:
          minute: 600
          policy: redis
          redis_host: redis.internal
          fault_tolerant: true

The retries: 1 value is deliberate: the gateway makes at most one extra attempt against its immediate upstream and delegates deeper resilience to the mesh. mtls-auth here authenticates an external party — it is a perimeter control, not workload identity.

Istio / Envoy 1.32+ inside the mesh (east-west)

Inside the mesh, mTLS is workload-to-workload and issued by the control plane. A PeerAuthentication resource sets the mode to strict so unauthenticated plaintext is rejected, and a DestinationRule plus VirtualService own east-west retries and outlier detection close to the upstream.

# Istio 1.32+ — mesh: enforce strict workload mTLS for east-west traffic
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT          # reject any non-mTLS east-west connection
---
# DestinationRule — mesh mTLS origination + host ejection near the upstream
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: ledger
  namespace: payments
spec:
  host: ledger.payments.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL  # use mesh-issued, auto-rotated workload certs
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
---
# VirtualService — east-west retries owned by the sidecar, close to the callee
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: ledger
  namespace: payments
spec:
  hosts: ["ledger.payments.svc.cluster.local"]
  http:
    - route:
        - destination:
            host: ledger.payments.svc.cluster.local
      timeout: 2s
      retries:
        attempts: 2
        perTryTimeout: 0.5s
        retryOn: "5xx,reset,connect-failure"

ISTIO_MUTUAL tells the sidecar to present the mesh-issued workload certificate — no PKI material appears in the config because the control plane provisions and rotates it. The retry policy lives here, next to ledger, precisely because the sidecar has accurate local health signals for its upstream. The gateway’s retries: 1 and this attempts: 2 are owned by different components for different hops; they are not duplicated policy for the same hop.

Comparative implementation

Operational gotchas

Double-proxy latency and re-termination. Every layer is a hop. A request that crosses the gateway and then two sidecars per east-west call pays for three proxies. The crypto cost is amortized by connection reuse, but the hop count is not — measure end-to-end P99 with the full chain, not each component in isolation. Terminating external TLS at the gateway and re-originating mesh mTLS is correct; terminating and re-terminating TLS with no change in trust guarantee is pure overhead you should remove.

Duplicated policy with no single owner. The archetypal incident: the gateway retries a call twice, each mesh sidecar retries it twice, and one upstream hiccup becomes an 8x traffic multiplier that keeps the upstream down. Assign exactly one owner per hop for retries and one for circuit breaking. Write the ownership down; do not let it emerge accidentally from two teams each “just adding resilience.”

Control-plane sprawl. A gateway control plane plus a mesh control plane means two config models, two RBAC systems, two upgrade cadences, and two blast radii. During an incident, responders must know which plane owns the routing decision that broke. Standardize on GitOps for both, keep their config in the same review pipeline, and document a decision tree for “which plane do I change to fix X.”

mTLS mode mismatch during rollout. Flipping PeerAuthentication to STRICT before every workload has a sidecar injected will hard-fail plaintext callers. Roll out mesh mTLS in PERMISSIVE mode first, confirm every peer is speaking mTLS in telemetry, then switch to STRICT. The gateway’s re-originated connection must itself be a mesh member, or the first hop into the mesh will be rejected once strict mode is on.

Header and identity confusion at the seam. The gateway strips and re-signs external headers before the request enters the mesh; the mesh then attaches workload identity. If the gateway forwards raw client-supplied X-Forwarded-* or spoofable identity headers, the mesh may treat them as trusted. Sanitize at the edge so the only identity crossing the seam is the one the gateway minted.

For deciding whether you even need the mesh yet, weigh it against a simpler tiered gateway topology described in high-availability topologies — a small service count with modest east-west traffic is often better served by a gateway plus disciplined client libraries than by adopting a full mesh control plane prematurely.

Decision matrix: gateway, mesh, or both

Read the matrix as a progression, not three walled options. Almost every platform starts in the left column — a gateway is the first thing you need the moment you expose an API. The mesh becomes worthwhile when east-west traffic grows dense enough that reimplementing mTLS, retries, and tracing in every service’s application code is more expensive than operating a control plane. The right column is where mature platforms land: a product gateway at the edge for the consumer contract, and a mesh inside for workload identity and internal resilience.

Production Configuration Checklist

• North-south and east-west ownership is written down: the gateway owns edge/consumer concerns, the mesh owns internal identity and resilience.
• External TLS terminates at the gateway; the gateway re-originates as a mesh member rather than forwarding plaintext into the mesh.
• Mesh mTLS is rolled out in PERMISSIVE mode and confirmed in telemetry before switching to STRICT.
• Retry policy has exactly one owner per hop — the gateway retries only its immediate upstream; the mesh owns east-west retries.
• End-to-end retry budget is capped so gateway + mesh attempts cannot multiply into a thundering herd.
• Outlier detection / circuit breaking lives in the mesh sidecar, close to the upstream, not duplicated at the edge for the same hop.
• Consumer quotas and rate limits live only at the gateway; they are not attempted in the mesh.
• A single traceparent is propagated from the edge span through every mesh hop; edge and mesh export to the same collector.
• The gateway sanitizes and re-signs client-supplied identity headers before the request crosses into the mesh.
• Both control planes are driven by GitOps in the same review pipeline, with a documented “which plane owns this” decision tree.
• End-to-end P99 latency is measured across the full gateway + sidecar chain, not per component in isolation.
• Any TLS re-termination that does not change the trust guarantee is identified and removed.

FAQ

Is a service mesh a replacement for an API gateway?

No. They solve adjacent but distinct problems. An API gateway is an edge component that terminates external north-south traffic and applies consumer-facing concerns: authentication, API keys, rate limiting, request transformation, and a developer-facing contract. A service mesh governs east-west traffic between internal services, providing transparent mTLS, load balancing, retries, and observability without changing application code. Most mature platforms run both: a gateway at the perimeter and a mesh for service-to-service traffic behind it.

Do I need mTLS in both the gateway and the mesh?

The two mTLS layers protect different trust boundaries. Gateway mTLS authenticates external clients or partner systems presenting client certificates at the edge. Mesh mTLS establishes workload identity between internal services using short-lived, automatically rotated certificates issued by the mesh control plane. When both run, the gateway terminates external TLS, then re-originates mTLS as a mesh member so the request carries a verifiable workload identity from the edge inward. Do not disable mesh mTLS just because the gateway already terminated TLS — that would leave east-west traffic unauthenticated.

Where should retry and circuit-breaking policy live when I run both?

Put retry and outlier-detection policy in the layer closest to the failing dependency. The mesh sidecar should own east-west retries and per-host ejection because it sits next to the upstream and has accurate local health signals. The gateway should own edge-level retries only for its immediate upstream and should generally not retry deep into the call graph, or retries multiply at each hop and cause a thundering herd. Define a single owner per dependency and cap total retry budget end to end.

What extra latency does running a gateway and a mesh together add?

A combined topology adds one gateway hop at the edge (typically 1 to 5 ms) plus two sidecar hops per east-west call (roughly 0.3 to 1 ms each for the client-side and server-side proxy). The mTLS handshake is amortized by connection pooling and session reuse, so steady-state overhead is dominated by the extra proxy hops rather than crypto. The larger cost is usually operational: two control planes, two policy models, and two places to look during an incident.

Can I use Envoy for both the gateway and the mesh?

Yes, and it is a common pattern. Envoy is the data plane for most meshes (including Istio) and also runs as a standalone edge gateway. Sharing one proxy technology reduces the number of runtimes to operate and lets edge and mesh share filter and telemetry conventions. The trade-off is that a standalone Envoy edge lacks the developer portal, API-key management, and consumer OAuth flows that dedicated gateway products like Kong ship out of the box, so many teams pair a product gateway at the edge with an Envoy-based mesh inside.

Related

• Security Boundaries & Zero-Trust — how the gateway enforces the perimeter trust boundary that the mesh extends inward.
• Circuit Breaking & Retry Budgets — the resilience mechanics that make split retry ownership between edge and mesh safe.
• High-Availability Topologies — deciding between a tiered gateway and a full mesh as service count grows.
• Multi-Tenant Routing Strategies — why tenant routing is a consumer concept resolved at the edge, not in the mesh.
• Gateway Observability & Operations — stitching edge and mesh spans into a single trace across the full request path.

← API Gateway Fundamentals & Architecture