Streaming, WebSocket & Realtime Gateways

Every assumption an API gateway makes about a request — that it is short, that it has a body of bounded size, that policy can be evaluated once before forwarding, that a worker is free again in milliseconds — stops being true when the connection is long-lived. A WebSocket that stays open for four hours, a server-sent event stream that trickles a message a minute, and a bidirectional gRPC channel are all requests as far as the proxy’s accounting is concerned, and they consume a slot for their whole lifetime. This topic covers what actually changes: the connection budget, where policy can still be enforced, how idle timeouts interact with heartbeats, and what a deploy does to clients that were mid-stream. It sits under API gateway fundamentals and architecture, and it revisits several assumptions made in scaling limits and capacity planning that only hold for short requests.

Architectural baseline

A long-lived connection is a held resource, not a completed request. Throughput planning for request-response traffic works in requests per second against a service time of milliseconds. Streaming planning works in concurrent connections against a memory cost per connection and a hard ceiling on file descriptors. The two models share almost no arithmetic, and a gateway sized for one will fail in a way that looks inexplicable under the other.

Policy is enforced at the start, or not at all. Middleware runs when the request is admitted. After an upgrade or after the response headers of a stream are sent, there is no further per-request hook, so quotas, payload validation and authorisation decisions have to be made up front and re-checked, if at all, by the application that owns the socket. This is the single largest architectural consequence and it is not a limitation of any particular product.

Three transports, three failure modes. WebSocket is a protocol upgrade that leaves HTTP behind. Server-sent events remain an ordinary HTTP response that never ends. gRPC streaming is HTTP/2 frames multiplexed over one connection. They are often discussed together because they solve the same problem, but they break differently, and buffering behaviour is where they diverge most.

What holding a connection actually costs

The number that matters is not requests per second, it is concurrent connections multiplied by the per-connection memory, checked against the process file-descriptor limit.

The same request rate, two completely different resource profiles At two thousand requests per second with a twenty millisecond service time, only about forty requests are in flight at once. At the same arrival rate with connections held for thirty minutes, three and a half million connections would be concurrent, which is why streaming capacity is planned in connections rather than in requests per second. 2,000 new requests per second, two service-time regimes 20 ms request-response ≈ 40 concurrent — a rounding error against any limit 30 s poll connection ≈ 60,000 concurrent — file descriptors now matter 30 min WebSocket ≈ 3.6 M concurrent — not a single-node problem at all Concurrency is arrival rate multiplied by holding time. The third row is not a tuning exercise; it is a different architecture, with connection sharding and an explicit connection budget per node. Budget per node = min(file descriptor limit, memory ÷ per-connection buffers) — and leave headroom for a failover.

Per connection, expect on the order of 16–64 KiB of proxy buffers, plus whatever the upstream holds. A node with 4 GiB available to the proxy and 32 KiB per connection tops out near 130,000 connections before memory, and rather earlier than that if the file-descriptor limit was left at its default. Both limits are worth setting explicitly rather than discovering.

# Envoy 1.32+ — bound the connection count deliberately
static_resources:
  listeners:
    - name: realtime
      address: { socket_address: { address: 0.0.0.0, port_value: 8443 } }
      per_connection_buffer_limit_bytes: 32768      # cap per-connection memory
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: realtime
                upgrade_configs:
                  - upgrade_type: websocket                 # allow the upgrade at all
                stream_idle_timeout: 0s                     # disable the per-stream idle timer
                common_http_protocol_options:
                  idle_timeout: 3600s                       # connection-level ceiling
                  max_connection_duration: 14400s           # force reconnection every 4 h
                route_config:
                  virtual_hosts:
                    - name: realtime
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/ws" }
                          route:
                            cluster: realtime_upstream
                            timeout: 0s                     # no overall request timeout
                            idle_timeout: 300s              # but do reap dead sockets
overload_manager:
  resource_monitors:
    - name: "envoy.resource_monitors.global_downstream_max_connections"
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig
        max_active_downstream_connections: 120000

Setting timeout: 0s is mandatory for a stream and dangerous on its own: it removes the only bound on how long a request may run. Pair it with a route-level idle_timeout so a connection that has stopped exchanging bytes is still reaped, and with max_connection_duration so no connection outlives a certificate rotation or a config generation indefinitely.

Where policy can still be enforced

The middleware chain runs once. Everything you want to enforce has to happen in that window, which changes how quotas and authorisation are designed for realtime endpoints.

One enforcement window, then hours of unpoliced frames Authentication, authorisation, quota accounting and route selection all happen during the initial handshake. Once the stream is established the proxy relays bytes: a token that expires, a quota that is exhausted or a permission that is revoked has no effect until the connection is closed and re-established. handshake — policy runs authn, authz, quota, route selection established — bytes relayed, no middleware token expiry, quota exhaustion and permission changes have no effect here the only lever left is closing the connection Three consequences worth designing for: rate limit the handshake rather than the messages; keep stream lifetimes shorter than token lifetimes so re-authentication happens naturally on reconnect; and give the application a way to close a session, because revocation cannot be enforced at the proxy after admission. A four-hour max_connection_duration against a fifteen-minute token is not a security control — it is a four-hour window in which a revoked credential keeps working.

Rate limiting therefore targets the handshake. A limit of a few connections per minute per consumer is both effective and cheap, and it is the control that stops a reconnect storm after a deploy from being indistinguishable from an attack. The mechanics are the same as in rate limiting and throttling strategies; only the key changes, from request to connection attempt.

# Kong 3.x — limit connection attempts, since messages cannot be counted
services:
  - name: realtime
    url: http://realtime.internal:8080
    read_timeout: 3600000        # ms; the socket must be allowed to idle
    write_timeout: 3600000
    routes:
      - name: ws
        paths: ["/ws"]
        protocols: ["http", "https"]
    plugins:
      - name: rate-limiting
        config:
          minute: 12             # connection attempts per consumer per minute
          policy: redis
          fault_tolerant: true
      - name: jwt
        config:
          claims_to_verify: ["exp"]

Idle timeouts and heartbeats

The most common realtime incident is a connection that both ends believe is alive being reaped by something in the middle. Every hop has its own idle timer — client, gateway, cloud load balancer, upstream — and the effective timeout is the smallest of them.

The shortest timer in the path wins, and it is rarely yours The client waits sixty seconds, the cloud load balancer three hundred and fifty, the gateway three thousand six hundred and the upstream nine hundred. The effective idle timeout is sixty seconds, so a heartbeat must be more frequent than that. Tuning only the gateway changes nothing. idle timer at each hop client library 60 s — the effective limit cloud load balancer 350 s — the default nobody set upstream server 900 s gateway 3600 s — the one that was carefully tuned, and never applies Set the heartbeat below the smallest timer, not below yours: a ping every 25 seconds survives the 60-second hop. Then measure — an idle drop shows up as a connection closed with no close frame and no error on either end.

WebSocket has protocol-level ping and pong frames for exactly this. Server-sent events have no framing, so the convention is a comment line — a line beginning with a colon — sent on a timer, which the client ignores but which keeps every intermediary’s idle timer fed. gRPC uses HTTP/2 PING frames, configured by keepalive parameters on both ends, and a mismatch there produces ENHANCE_YOUR_CALM errors rather than silent drops, which is at least easier to diagnose.

Comparative implementation

Concern WebSocket Server-sent events gRPC streaming
Transport HTTP upgrade, then framed ordinary HTTP response HTTP/2 frames
Direction bidirectional server to client only either or both
Proxy buffering risk low, framing is explicit high — response buffering breaks it low, framing is explicit
Keepalive mechanism ping and pong frames comment lines on a timer HTTP/2 PING
Reconnection client-implemented built into the browser API client-implemented
Works through strict proxies sometimes almost always needs end-to-end HTTP/2
Per-message policy none after upgrade none after headers none after headers

Server-sent events are consistently underrated for one-directional feeds. They are ordinary HTTP, they survive intermediaries that mangle upgrades, and browsers reconnect automatically with a Last-Event-ID header. The single configuration requirement is disabling response buffering on every hop, which is covered in server-sent events vs WebSockets at the edge.

Scaling past one node

Once the connection count exceeds what a single node can hold, two things have to be decided that never come up with short requests.

The first is how connections are distributed. A plain round-robin load balancer distributes new connections evenly, which is not the same as distributing held connections evenly — a node that restarts comes back empty and stays comparatively empty until enough churn has occurred to refill it. Least-connection balancing fixes the steady state and makes a restarted node absorb the next burst, which is usually what you want. A maximum connection duration then provides the churn that keeps the distribution honest over hours.

The second is whether the upstream needs affinity. A chat room, a collaborative document or any feature where connected clients must see each other’s messages requires either that related clients land on the same upstream instance, or a shared bus behind the instances. Affinity at the gateway is the tempting answer and the fragile one: it makes the routing decision dependent on state the gateway cannot rebuild after a restart, and it turns a single hot room into a hot node. A message bus behind stateless stream servers costs more to run and does not develop these failure modes, which is why almost every realtime platform ends up there eventually.

Both decisions interact with the capacity arithmetic. Affinity means a node cannot shed load by rejecting new connections, because the connections it must accept are determined by which room the client is joining rather than by how busy it is. Without affinity, a node at its connection ceiling can simply refuse and let the load balancer place the client elsewhere — which is the behaviour you want during a reconnect storm, and the reason a connection ceiling should reject cleanly rather than accept and degrade.

Observability for connections rather than requests

The dashboards that serve request-response traffic answer questions streams do not ask. Requests per second is near-constant and uninteresting; what matters is how many connections are open, how long they have been open, how many closed in the last minute and why, and how many bytes are moving in each direction. Age distribution in particular is the one to add first: a healthy fleet shows a broad spread of connection ages, while a spike of very young connections means something just closed a lot of them and every client came back at once.

Operational gotchas

A deploy is a reconnect storm. Draining a node closes every connection it holds, and every client reconnects within its backoff window — which for many client libraries is “immediately”. A fleet holding 100,000 connections across ten nodes sends 10,000 reconnections in a burst when one node restarts. Stagger drains, give clients jittered backoff, and cap connection attempts per consumer, using the same reasoning as tuning retry budgets to prevent thundering herd.

Load balancing happens once per connection, not per message. A client that connects during a low-traffic window keeps its node for hours regardless of how loaded that node becomes. Rebalancing requires closing connections, so a maximum connection duration is a load-distribution mechanism as much as a security one.

Access logs are written at connection close. An hours-long connection produces no log line until it ends, so a dashboard built on access logs shows nothing while thousands of streams are active. Emit a connection-open event, or a periodic gauge of active connections, or accept a blind spot exactly where you need visibility.

Health checks that open a connection and immediately close it will, at sufficient frequency, dominate your connection metrics and can trip connection-rate limits. Give probes their own route with its own limits.

Production configuration checklist

  • Connection ceiling set explicitly per node, below both the file-descriptor and memory limits
  • Per-connection buffer limit configured, not left at the default
  • Overall request timeout disabled on streaming routes, with an idle timeout still in place
  • max_connection_duration set so no connection outlives a certificate or config generation
  • Heartbeat interval below the smallest idle timer in the whole path, verified by measurement
  • Connection-attempt rate limit per consumer, sized for a reconnect storm
  • Active-connection gauge exported, since access logs only appear at close
  • Drain procedure staggers node restarts rather than restarting in parallel

FAQ

Why does a streaming route need timeout set to zero?

Because the overall request timeout counts from when the request was accepted, and a stream is one request that lasts for hours. Leaving the default in place closes every stream at that boundary regardless of how healthy it is. Disabling it removes the only bound on request duration, so always pair it with an idle timeout that reaps sockets exchanging no bytes, and a maximum connection duration that forces periodic reconnection.

Can I rate limit individual WebSocket messages at the gateway?

Not with standard middleware. After the upgrade completes the proxy is relaying frames and no per-request hook runs, so there is nothing to count. Rate limit connection attempts per consumer instead, which is both cheap and effective, and enforce per-message limits in the application that owns the socket if you need them.

Why do my streams die after exactly 60 seconds?

A timer somewhere in the path is shorter than your heartbeat interval, and it is usually not the gateway. Client libraries, cloud load balancers and upstream servers each have their own idle timeout, and the effective value is the smallest of them. Find every hop, take the minimum, and set the heartbeat comfortably below it — a ping every 25 seconds survives a 60-second timer.

Should I use WebSockets or server-sent events?

If the data only flows from server to client, server-sent events are usually the better choice: they are ordinary HTTP, they pass through intermediaries that mangle upgrades, and browsers reconnect automatically with a Last-Event-ID header. Choose WebSockets when the client genuinely needs to push messages back on the same connection, and be prepared to implement reconnection yourself.

Parent: API Gateway Fundamentals & Architecture