gRPC Streaming Keepalives and Timeouts

gRPC streams fail differently from WebSockets: instead of dying silently, they usually produce an error code, and the code is often ENHANCE_YOUR_CALM — which means one side decided the other was sending keepalive pings too aggressively and closed the connection as a defensive measure. That error is a negotiation failure between two independently configured keepalive policies, and it is entirely avoidable once the parameters are understood. This page covers the settings on client, gateway and server, and how they must relate.

Prerequisite concepts

This assumes the long-lived-connection model from streaming, WebSocket and realtime gateways and the HTTP/2 framing described in protocol translation patterns. Familiarity with the four call shapes — unary, server streaming, client streaming and bidirectional — is assumed.

Three timers, on three machines

The rule: nobody pings faster than the other end is willing to permit The client sends a ping every thirty seconds. The gateway permits a minimum interval of twenty seconds and itself pings the upstream every sixty. The server permits a minimum of thirty. Every ping interval is at or above the permitted minimum on the receiving side, which is the condition that avoids ENHANCE_YOUR_CALM. hop pings every permits pings no faster than client 30 s — (it is the one pinging) gateway 60 s to the upstream 20 s — so the client at 30 s is fine upstream server — 30 s — so the gateway at 60 s is fine Break either relationship and the receiving side answers with a GOAWAY carrying ENHANCE_YOUR_CALM — which reads like a rate limit and is in fact a keepalive disagreement between two config files nobody diffed together. The defaults do not satisfy this: many servers permit no pings at all on connections with no active calls. Four call shapes, and what each one does to a proxy Unary sends one message each way and behaves like an ordinary request. Server streaming holds the response open. Client streaming holds the request open. Bidirectional holds both, and is the shape that makes buffering, retries and load balancing all behave unlike anything in a request-response world. call shape client → server server → client proxy treats it as unary 1 message 1 message an ordinary request server streaming 1 message many a long response client streaming many 1 message a long upload bidirectional many many a held connection 
# Envoy 1.32+ — keepalive toward the upstream, and tolerance for the downstream
clusters:
  - name: grpc_upstream
    type: STRICT_DNS
    http2_protocol_options:
      connection_keepalive:
        interval: 60s              # ping the upstream this often
        timeout: 20s               # and close if no ack within this
      max_concurrent_streams: 500
    typed_extension_protocol_options: {}
# The downstream side: what the gateway tolerates from clients
typed_config:
  "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
  stat_prefix: grpc_edge
  http2_protocol_options:
    max_concurrent_streams: 1000
    initial_stream_window_size: 262144      # 256 KiB — flow control per stream
    initial_connection_window_size: 1048576 # 1 MiB per connection
  stream_idle_timeout: 0s                   # streaming calls are not idle-timed out
  common_http_protocol_options:
    idle_timeout: 3600s

On the Go server side the matching parameters are KeepaliveEnforcementPolicy.MinTime, which must be at or below the client’s ping interval, and PermitWithoutStream, which must be true if clients ping while no call is active.

Flow control is the other half

HTTP/2 has per-stream and per-connection flow-control windows, and a server-streaming call that produces data faster than the client consumes it will stall when the window is exhausted rather than buffering without limit. That is the desired behaviour, but it means the window size sets the throughput ceiling of a fast stream over a high-latency link.

Choosing the numbers rather than copying them

There are only three quantities to pick, and each has a defensible derivation rather than a folk default.

The ping interval should be shorter than the shortest idle timeout in the path, with enough margin that one lost ping does not close the connection — a third of the smallest timer is a reasonable rule. The ping timeout, which is how long the sender waits for an acknowledgement before declaring the connection dead, should be comfortably above the worst round-trip time you actually observe, not the median: setting it to two seconds on a link whose tail latency reaches three will close healthy connections during exactly the congestion events where you want them to survive.

The permitted minimum on the receiving side then has to sit at or below the interval every legitimate client uses. This is the value that most often ends up wrong, because it is configured by the team running the server while the interval is configured by whoever wrote the client — frequently a different organisation. Publish the expected client keepalive settings alongside your service definition and treat them as part of the contract, in the same way that a maximum message size is part of the contract.

Window divided by round-trip time is your throughput ceiling With the default sixty-five kilobyte window, a five millisecond link supports about thirteen megabytes per second per stream, a fifty millisecond link about one and a third, and a two hundred millisecond link about three hundred kilobytes. Raising the window to two hundred and fifty-six kilobytes multiplies each of those by four. per-stream throughput ceiling with the default 65 KiB window 5 ms RTT — same datacentre ≈ 13 MB/s 50 ms RTT — cross-region ≈ 1.3 MB/s 200 ms RTT — mobile, distant ≈ 0.3 MB/s — and no error is reported Raising initial_stream_window_size to 256 KiB multiplies every row by four; the cost is memory per concurrent stream.

Decision matrix

Symptom Cause Fix
ENHANCE_YOUR_CALM on connect client pings faster than the server permits raise the client interval or lower MinTime
Streams die after ~5 minutes stream_idle_timeout default set it to 0s for streaming routes
Slow streams over a distant link flow-control window too small raise initial_stream_window_size
UNAVAILABLE after a deploy connection drained mid-call client retry with backoff on that code only
Only the first message arrives an intermediary is buffering or is HTTP/1.1 ensure end-to-end HTTP/2

Gotchas and failure signals

Any HTTP/1.1 hop in the path breaks streaming entirely. gRPC requires HTTP/2 end to end; a load balancer that terminates HTTP/2 and re-originates HTTP/1.1 turns a bidirectional stream into a unary call that never completes.

Deadlines propagate, keepalives do not. A gRPC deadline set by the caller is carried through the metadata and respected by each hop. Keepalive settings are per hop and configured independently, which is exactly why they drift apart.

max_concurrent_streams limits calls per connection, not connections. A client multiplexing more calls than the limit queues them silently, appearing as latency with no error, and the metric to watch is stream queue depth rather than connection count.

Retrying a streaming call is not like retrying a unary one. A server-streaming call that failed halfway has already delivered messages the client processed, so a blind retry duplicates them. Retry only from a checkpoint the application understands, as with any at-least-once delivery.

Validation

  • Client ping interval is at or above the server’s permitted minimum, on every hop
  • PermitWithoutStream is true wherever clients ping on idle connections
  • stream_idle_timeout disabled for streaming routes, connection idle timeout retained
  • Flow-control window sized against the real round-trip time of the worst client
  • End-to-end HTTP/2 confirmed with no HTTP/1.1 downgrade at any hop
  • Retry policy distinguishes streaming from unary calls

FAQ

What causes ENHANCE_YOUR_CALM on a gRPC stream?

One side decided the other was sending keepalive pings more often than it permits, and closed the connection with a GOAWAY. It reads like a rate limit and is actually a disagreement between two independently configured keepalive policies. Make every ping interval at or above the permitted minimum on the receiving side, and set PermitWithoutStream where clients ping on idle connections.

The HTTP/2 flow-control window is the ceiling: throughput per stream is roughly window size divided by round-trip time. With the default sixty-five kilobyte window a two hundred millisecond link tops out near three hundred kilobytes per second, and nothing reports an error — the stream simply stalls waiting for window updates. Raising the initial stream window multiplies the ceiling proportionally, at the cost of memory per stream.

Do gRPC deadlines propagate through the gateway?

Yes. The deadline travels in request metadata and each hop is expected to respect it, which is what makes a caller-set deadline meaningful across several services. Keepalive settings do not propagate — they are configured per hop and per direction, which is precisely why they drift out of agreement and produce connection-level errors.

Can I retry a failed streaming call?

Not the way you retry a unary one. A server-streaming call that failed partway has already delivered messages the client processed, so replaying it duplicates work. Retry only from a checkpoint the application maintains, and treat the stream as at-least-once delivery rather than expecting the transport to make it exactly-once.

Parent: Streaming, WebSocket & Realtime Gateways