How API Gateways Differ from Load Balancers

Platform architects frequently conflate traffic distribution with application routing. Both components sit at the network edge and both forward client requests to upstream services — but their operational boundaries are strictly defined by the OSI layer they inspect and the policies they can enforce. Deploying only a load balancer when the workload demands fine-grained authentication or protocol translation leaves policy gaps that are invisible until a breach or a cascading failure exposes them. This page maps the exact boundaries between the two components so you can deploy them in the correct roles.

Prerequisite Concepts

Before reading further, make sure you are grounded in the following:

  • API Gateway Fundamentals & Architecture — the data-plane and control-plane model, request lifecycle, and deployment topologies that provide context for the comparison below.
  • Gateway Selection Criteria — the capability matrix used to evaluate Kong, Tyk, Envoy, and NGINX against a workload’s SLO requirements, including throughput and latency budgets.

OSI Layer Boundary: The Core Distinction

The single most important dividing line is which OSI layer each component fully parses.

OSI Layer Boundary: Load Balancer vs API Gateway A diagram showing the OSI stack layers L3 through L7. Load balancers operate at L3 and L4 (Network and Transport). L7 load balancers optionally read HTTP paths and headers. API gateways operate fully at L7, parsing JWT claims, gRPC method names, GraphQL operations, and custom headers. OSI Layer Load Balancer API Gateway L3 — Network IP routing, CIDR L4 — Transport TCP/UDP, ports L5 — Session TLS handshake L6 — Presentation Encoding, compression L7 — Application HTTP, gRPC, JWT GraphQL, headers Custom claims Primary scope Connection distribution Health checking, IP hash Round-robin, least-conn TLS termination (optional) L7 LB: path/host only No payload parse No policy enforcement Primary scope mTLS termination JWT / OIDC validation Rate limiting per route Protocol translation Declarative routing rules Request/response transforms GraphQL / gRPC routing Developer portal / catalog Load Balancer scope API Gateway scope Dashed = optional / partial coverage

Load balancers distribute connections across a pool of upstream instances using transport-layer metrics: round-robin, least connections, IP hash, or weighted random. They operate at L3/L4 and forward packets without parsing the application payload. Layer-7 load balancers can inspect HTTP host headers and URL paths to route between backend pools, but they do not parse JWT claims, decode gRPC frames, or execute per-route policy logic.

API gateways operate exclusively at L7. They terminate the client connection, buffer the full request body, and parse application-level semantics before dispatching upstream. This enables routing decisions based on JWT claims, gRPC method names, GraphQL operation types, or arbitrary header values — and policy enforcement (authentication, rate limiting per tenant, transformation) that a load balancer cannot provide.

Side-by-Side: Traffic Distribution vs. Application-Aware Routing

Load Balancer Routing Logic

A layer-4 load balancer routes by TCP 5-tuple (src IP, dst IP, src port, dst port, protocol). A layer-7 load balancer can additionally match on Host and URL prefix. Neither can condition routing on the value of a JWT claim, a gRPC service name, or a custom X-Tenant-ID header — those require full payload parsing.

AWS ALB (L7 load balancer) — host and path rules only:

# ALB listener rule — matches host + path, no payload inspection
Rules:
  - Priority: 10
    Conditions:
      - Field: host-header
        Values: ["api.example.com"]
      - Field: path-pattern
        Values: ["/v2/*"]
    Actions:
      - Type: forward
        TargetGroupArn: !Ref V2ServiceTargetGroup

This is the ceiling of what an L7 LB can express. There is no condition type for arbitrary header values, decoded JWT fields, gRPC method names, or GraphQL operation types.

API Gateway Routing Logic (Envoy 1.32+)

An API gateway evaluates a routing table only after parsing the full request. The Envoy virtual host below routes on both a URL prefix and a custom header value — a combination impossible in an ALB rule:

# Envoy 1.32+ route config: path + header-value routing
virtual_hosts:
  - name: api_service
    domains: ["api.example.com"]
    routes:
      - match:
          prefix: "/admin"
          headers:
            - name: "x-api-version"
              string_match:
                exact: "v2"
        route:
          cluster: admin_service_v2
          timeout: 0.5s
      - match:
          prefix: "/admin"
        route:
          cluster: admin_service_v1
          timeout: 0.5s

Because Envoy buffers and parses the request before selecting a route, it can combine any number of header, path, query-string, and JWT claim conditions in a single match block. The routing engine evaluates conditions top-to-bottom and dispatches to the first matching upstream cluster.

API Gateway Routing Logic (Kong 3.x)

Kong expresses the same intent through its Admin API or declarative kong.yaml. The headers match fields are evaluated at the plugin runner level — after the Kong core has decoded the HTTP/2 or HTTP/1.1 frame:

# kong.yaml (deck format) — Kong 3.x
services:
  - name: admin-service-v2
    url: http://admin-v2.internal:8080
    routes:
      - name: admin-v2-route
        paths: ["/admin"]
        headers:
          x-api-version: ["v2"]
        plugins:
          - name: jwt
            config:
              claims_to_verify: ["exp", "nbf"]

Kong evaluates the headers map and runs the jwt plugin sequentially in the same request pass — the load balancer upstream of Kong has already forwarded the connection without any knowledge of these conditions.

Decision Matrix: When to Use Each Component

Requirement L4 Load Balancer L7 Load Balancer API Gateway
Distribute TCP connections across pods ✅ ✅ —
Route HTTP traffic by host/path — ✅ ✅
Route by JWT claim or custom header — — ✅
Terminate mTLS and validate client cert — Limited ✅
Validate OIDC/JWT tokens — — ✅
Rate limiting per client_id or tenant — — ✅
Protocol translation (REST → gRPC) — — ✅
DDoS mitigation at connection layer ✅ ✅ —
Multi-AZ failover with health checks ✅ ✅ ✅ (upstream health)
Developer portal / API catalogue — — ✅
Sub-millisecond forwarding overhead ✅ ✅ — (adds ~1–5 ms per policy)

In production these are typically chained: an L4/L7 load balancer handles TCP distribution, DDoS mitigation, and multi-AZ failover, then forwards pre-filtered traffic to an API gateway that handles all application-layer concerns. The high-availability topologies page details how to size and place each tier.

Protocol Translation: A Gateway-Only Capability

A load balancer maintains protocol parity. An HTTP/1.1 request forwarded through an L7 LB arrives at the backend as HTTP/1.1. Gateways actively translate protocols: HTTP/1.1 → HTTP/2, REST → gRPC, or synchronous HTTP → async message queue. During translation the gateway mutates headers (Host, X-Forwarded-Proto, Content-Length), manages connection pooling per protocol, and handles chunked-encoding differences. For a deeper look at this capability, see handling gRPC to REST translation at scale.

NGINX Plus 1.25+ — gRPC proxying with header normalization:

# Gateway acts as protocol adapter, not a transparent pipe
location /api/v1/ {
    grpc_pass grpc://upstream_grpc_cluster;
    grpc_set_header Host            $host;
    grpc_set_header X-Forwarded-Proto $scheme;
    grpc_set_header X-Request-ID    $request_id;
}

A load balancer cannot do this. It would forward the raw HTTP/1.1 bytes to the gRPC backend, producing an immediate 400 Bad Request because the backend expects the binary framing of HTTP/2.

Policy Enforcement: Security Boundaries

Load balancers rely on external WAFs, IAM proxies, or bespoke middleware for security policy. API gateways embed policy execution in the data plane. They terminate mTLS at the gateway edge, validate OIDC tokens, enforce token-bucket rate limits, and apply request/response transformation rules before traffic reaches upstream services. This shifts the security perimeter inward, enabling zero-trust architectures where identity verification precedes routing.

Policy enforcement order (API gateway data plane):

  1. Terminate TLS: Validate client certificate chain; extract SPIFFE/SAN identity.
  2. Authenticate: Verify OIDC/JWT signature against JWKS endpoint; cache validation results to reduce latency.
  3. Authorize: Match extracted claims (sub, scope, roles) against route-level ACLs.
  4. Rate limit: Apply sliding-window or token-bucket limits keyed by client_id or IP.
  5. Transform: Strip sensitive headers, inject tracing spans, normalize payloads.
  6. Dispatch: Establish upstream connection only after all policy checks pass.

A load balancer skips steps 1–5 entirely (or, at best, handles step 1 without steps 2–5). That means every upstream service behind a plain LB must implement its own auth, rate-limit, and transformation logic — which duplicates code, drifts across teams, and makes centralized policy change impossible.

Gotchas and Failure Signals

502 after mTLS termination at the gateway

  • Signal: Upstream services return 502 Bad Gateway immediately after you enable mTLS at the gateway.
  • Root cause: Upstream service expects client certificates that the gateway has already consumed and not forwarded, or the X-Forwarded-Client-Cert (XFCC) header is missing.
  • Fix: Configure the gateway to inject X-Forwarded-Client-Cert (PEM-encoded) or set ssl_verify_client off on the upstream if cert validation is fully handled at the edge.

429 false positives when a transparent LB sits in front

  • Signal: Legitimate clients are rate-limited immediately; all clients appear to share a single IP.
  • Root cause: The gateway reads X-Real-IP or the first entry in X-Forwarded-For but the LB has overwritten those headers with its own IP.
  • Fix: Set trusted_proxies CIDR ranges on the gateway so it walks the X-Forwarded-For chain to the actual client IP.

gRPC DEADLINE_EXCEEDED behind a load balancer + gateway chain

  • Signal: gRPC clients report deadline exceeded; the upstream service logs show requests never arrived.
  • Root cause: The L4 LB or gateway is not propagating the grpc-timeout header, so the gateway’s own timeout fires before the upstream’s deadline.
  • Fix: Ensure grpc-timeout is forwarded or mapped to the upstream deadline metadata. Increase max_connections_per_host if the gateway connection pool is exhausted first.

Mismatched Content-Length after body transformation

  • Signal: Downstream clients receive a 200 OK but the body is truncated; or upstream returns 400 Bad Request.
  • Root cause: The gateway modified the request body (e.g., injected a field) but did not recalculate Content-Length.
  • Fix: Use Transfer-Encoding: chunked on requests the gateway mutates, or ensure the transformation plugin recalculates the header. In Kong 3.x, the request-transformer plugin handles this automatically when config.replace.body is set.

Validation Checklist

  • Load balancer trusted_proxies CIDR list matches all gateway node IPs.
  • Gateway X-Forwarded-Client-Cert injection is confirmed via curl -v against a protected route.
  • gRPC routes have grpc-timeout propagation enabled; upstream deadline is ≥ gateway route timeout.
  • All body-mutating transformation plugins recalculate or drop Content-Length.
  • Rate-limit counters are keyed by client_id (not raw IP) when clients share NAT.
  • Health check intervals on the load balancer are shorter than the circuit-breaker open threshold on the gateway to avoid double-failure during partial outages.
  • Protocol translation routes (grpc_pass, grpc_set_header) are pinned to the expected upstream HTTP/2 port — not the HTTP/1.1 plaintext port.

FAQ

Can a load balancer replace an API gateway?

A layer-7 load balancer can handle basic HTTP routing and TLS termination, but it cannot perform JWT validation, protocol translation (REST to gRPC), or declarative per-route policy enforcement. For anything beyond connection distribution and health checking, a dedicated API gateway is required.

Do I need both a load balancer and an API gateway?

Yes, in most production topologies. The load balancer handles DDoS mitigation, TCP/UDP distribution, and multi-AZ failover at the front. Behind it, the API gateway handles authentication, rate limiting, protocol translation, and fine-grained routing. Each component does what it does best.

Which OSI layers do load balancers and API gateways operate at?

Load balancers operate at L3 (IP) and L4 (TCP/UDP), with L7 load balancers optionally inspecting HTTP host headers and paths. API gateways always operate at L7 and above — they parse the full application payload including JWT claims, gRPC method names, GraphQL operation types, and custom headers.

Parent: Gateway Selection Criteria