Rate Limiting & Throttling Strategies

Rate limiting is the enforcement boundary that keeps a single misbehaving consumer — or an unexpected traffic spike — from propagating backpressure through every upstream service in your stack. Within the middleware chain pipeline, the throttling engine sits after identity validation and before payload parsing, so the gateway can reject over-quota requests with minimal CPU cost. Getting the algorithm, the consumer identification strategy, and the distributed state model right determines whether your limits are accurate, fair, and operationally transparent.

Architectural Baseline

Before configuring any plugin, three design decisions must be locked down:

Algorithm. Fixed window, sliding window, token bucket, or leaky bucket each make different trade-offs between accuracy, burst tolerance, and state complexity. The choice is effectively irreversible once consumers build client-side retry logic around a published Retry-After contract.

Consumer key. IP address is the weakest identifier — shared NAT gateways and corporate proxies aggregate many users behind a single address. API key or JWT sub claim is the correct grain for per-tenant fairness. The identity layer must resolve the true consumer key before the counter is incremented, which is why authentication proxying and token validation must always execute earlier in the middleware chain than the rate limiter.

State backend. Local in-memory counters are fast but fragment quota across nodes. A centralised Redis backend with atomic operations is the only correct choice for horizontally scaled gateways. The dynamic rate limiting with Redis backends guide covers this in depth, including atomic counter scripts and hot-reload quota maps.

The diagram below shows where the throttling engine fits in the request lifecycle and how the state lookup path differs between local and distributed modes.

Algorithm Selection and Windowing Mechanics

Fixed Window

The simplest implementation: a counter resets every window_size seconds. It offers O(1) lookup but is vulnerable to a boundary spike — a client can fire the full quota in the last second of window N and again in the first second of window N+1, producing a burst of 2× the nominal limit within a two-second window. Acceptable for coarse metering of non-critical endpoints; not suitable where burst behaviour could overload a stateful backend.

Sliding Window

Two variants exist in practice:

• Sliding window log: stores a timestamped log per consumer in Redis, counts entries within now - window_size. Perfectly accurate but O(N) memory per consumer.
• Sliding window counter: blends the previous window’s count (weighted by overlap fraction) with the current window’s count. One Redis key per consumer, sub-millisecond lookup, ~0.1% worst-case error rate. Kong’s rate-limiting-advanced plugin implements this variant.

Token Bucket

Each consumer accumulates tokens at tokens_per_second up to a ceiling of burst_size. A request costs one token; if the bucket is empty, the request is rejected. Burst capacity is controllable without changing the average throughput limit, which makes token bucket the canonical choice for API products where plan tiers have defined burst allowances.

Leaky Bucket

Requests enter a fixed-size FIFO queue and drain at a constant output rate. Effective burst absorption is zero — the queue either accepts or overflows. Use this when the protected backend has a strict concurrency ceiling (e.g., a legacy SOAP service with a fixed thread pool) and any deviation from the steady-state rate causes cascading failures.

The diagram below maps each algorithm’s behaviour across two key axes: burst tolerance and counter accuracy.

Primary Deep-Dive: Kong 3.x Configuration

Kong’s built-in rate-limiting plugin implements fixed and sliding window counters; the rate-limiting-advanced plugin (Enterprise) adds token bucket, sliding window counter, and Redis Sentinel support.

# Kong 3.x — rate-limiting plugin (OSS), service-scoped
# Sliding window, consumer-key identified, Redis backend
_format_version: "3.0"
services:
  - name: payments-api
    url: http://payments-backend:8080
    plugins:
      - name: rate-limiting
        config:
          minute: 1000          # requests per minute ceiling
          hour: 20000           # secondary hourly ceiling
          policy: redis         # local | cluster | redis
          limit_by: consumer    # consumer | credential | ip | header | path
          redis_host: redis-primary.internal
          redis_port: 6379
          redis_password: "${REDIS_AUTH_TOKEN}"
          redis_timeout: 2000   # ms — fail-open if exceeded
          fault_tolerant: true  # pass requests when Redis is unreachable
          hide_client_headers: false
          error_code: 429
          error_message: "Rate limit exceeded. Retry-After header indicates next window."

For token bucket semantics with burst control, the Enterprise plugin adds:

# Kong 3.x — rate-limiting-advanced (Enterprise), sliding window + sync
      - name: rate-limiting-advanced
        config:
          limit: [1000]
          window_size: [60]
          window_type: sliding   # fixed | sliding
          identifier: consumer
          strategy: redis
          sync_rate: 0.25        # sync local counter to Redis every 0.25 s
          namespace: "payments:v2"
          retry_after_jitter_max: 5
          redis:
            host: redis-primary.internal
            port: 6379
            ssl: true
            connect_timeout: 1000
            send_timeout: 1000
            read_timeout: 1000

sync_rate is the critical knob: a value of -1 forces synchronous Redis writes on every request (most accurate, highest latency); values between 0 and 1 batch updates to Redis at that fractional-second interval, trading slight over-count tolerance for significantly lower p99 latency.

Secondary Deep-Dive: Envoy, Tyk, and NGINX

Envoy 1.32+ Global Rate Limit Filter

Envoy delegates rate limit decisions to an external gRPC service (typically Lyft’s ratelimit service or a compatible implementation). The HttpFilter adds a descriptor per request; the service increments counters and returns OK or OVER_LIMIT. This delegation model is what allows Envoy to support any counter algorithm — the logic lives in the external service, not Envoy itself.

# Envoy 1.32+ — global rate limit filter
# envoy.yaml (relevant HttpConnectionManager snippet)
http_filters:
  - name: envoy.filters.http.ratelimit
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
      domain: payments-api
      failure_mode_deny: false       # fail-open when gRPC service is down
      rate_limit_service:
        grpc_service:
          envoy_grpc:
            cluster_name: rate_limit_cluster
          timeout: 0.025s            # 25 ms hard budget for rate limit decision
        transport_api_version: V3
      request_headers_to_add_when_not_present:
        - header:
            key: X-RateLimit-Limit
            value: "%DYNAMIC_METADATA(envoy.filters.http.ratelimit:quota)%"

# Route-level descriptor — adds consumer ID and path to the counter key
virtual_hosts:
  - name: payments
    routes:
      - match: { prefix: "/v1/payments" }
        route:
          cluster: payments-backend
        typed_per_filter_config:
          envoy.filters.http.ratelimit:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimitPerRoute
            rate_limits:
              - actions:
                  - request_headers:
                      header_name: X-Consumer-ID
                      descriptor_key: consumer_id
                  - generic_key:
                      descriptor_value: payments-v1

Tyk 5.x Rate Limiting

Tyk applies rate limiting at the API key or policy level without requiring Lua or a separate service. The per_api and per_endpoint scoping is built into the API definition JSON, which makes it straightforward to apply different limits to different routes within the same service.

// Tyk 5.x — API definition (rate limiting section)
{
  "auth": { "auth_header_name": "Authorization" },
  "global_rate_limit": {
    "rate": 1000,
    "per": 60
  },
  "extended_paths": {
    "rate_limit": [
      {
        "path": "/v1/payments",
        "method": "POST",
        "rate": 100,
        "per": 60
      }
    ]
  },
  "enable_rate_limiting": true,
  "use_keyless": false
}

At the key policy level, the rate and per fields on the Policy object override the API-level defaults — useful for tiered plans where premium keys receive higher burst allowances without requiring separate API configurations.

NGINX with limit_req

For teams running open-source NGINX as a gateway, the limit_req_zone / limit_req directives implement a leaky bucket queue. There is no native sliding window or token bucket; for those, either the Lua-based lua-resty-limit-traffic library or a commercial NGINX Plus Advanced Rate Limiting module is required.

# nginx.conf — leaky bucket via limit_req (NGINX 1.25+)
# Shared memory zone: 10 MB holds ~160,000 consumer states
limit_req_zone $http_x_api_key zone=per_consumer:10m rate=100r/m;

server {
    listen 443 ssl;

    location /v1/payments {
        # burst: queue up to 20 excess requests; nodelay: serve burst immediately
        limit_req zone=per_consumer burst=20 nodelay;
        limit_req_status 429;

        # Inject standard rate limit headers via Lua (ngx_http_lua_module)
        header_filter_by_lua_block {
            ngx.header["X-RateLimit-Limit"] = "100"
            ngx.header["Retry-After"] = "60"
        }

        proxy_pass http://payments_backend;
    }
}

Comparative Implementation Table

Operational Gotchas

Boundary spikes with fixed windows. If you observe 2× burst at the top of each minute in metrics, clients are exploiting the window reset boundary. Switch to sliding window counter or add a short burst allowance that drains across the boundary.

fault_tolerant: true creates a silent quota bypass. When the Redis primary is unreachable and fault_tolerant is enabled, all consumers pass without limit. This is the correct default during Redis restarts but should trigger an alert immediately — sustained Redis absence means unlimited traffic reaches your backends. Instrument kong_data_store_errors_total in Prometheus and page on non-zero values.

sync_rate over-count at high RPS. A sync_rate of 0.5 s with 10 Kong nodes can allow up to 10 × requests_within_sync_interval over the nominal limit before the counter syncs. At 1,000 req/s per node that is a meaningful overshoot. Keep sync_rate below 0.1 s for tiers where accuracy is contractual, or use synchronous mode (sync_rate: -1).

Consumer key mismatch after middleware reorder. If the authentication plugin is moved after the rate limiter in the middleware chain, the consumer entity is not yet populated and limit_by: consumer silently falls back to IP. The rate limiter must always execute after token validation, never before — this is an invariant of the middleware ordering contract.

Envoy gRPC timeout budget exhaustion. Under Redis pressure, the external rate limit service can exceed the timeout budget, causing Envoy to fail-open (or fail-closed if failure_mode_deny: true). Export ratelimit_service_response_time_ms from the gRPC service and alert at the 95th percentile approaching the timeout ceiling.

NGINX limit_req_zone key cardinality. Using a high-entropy key (e.g., full JWT string rather than extracted sub claim) can exhaust the shared memory zone. Always extract a bounded-length key via $http_x_api_key or a map directive; set the zone size to support your maximum expected consumer count (1 MB ≈ 16,000 states at 64-byte keys).

Missing Retry-After header. Client SDKs that do not receive Retry-After on 429 responses will either hammer the gateway with immediate retries or implement arbitrary exponential back-off. Both outcomes worsen the overload situation. Always emit Retry-After alongside X-RateLimit-Reset.

Tyk policy inheritance gaps. When a key is not attached to a Tyk policy, it inherits the API-level global_rate_limit. If that global limit is higher than the intended tier ceiling, newly provisioned keys briefly receive excessive quota until the policy assignment is applied. Automate key-policy binding in your provisioning pipeline to close this window.

Production Configuration Checklist

• Algorithm chosen matches the burst profile and backend tolerance of each service tier
• Consumer key is consumer / API key / JWT sub — not raw IP address — for all authenticated endpoints
• Redis backend uses a replicated or Sentinel pair; no single-node Redis in production
• fault_tolerant: true (Kong) or failure_mode_deny: false (Envoy) is set, and Redis unavailability fires a P1 alert within 60 seconds
• X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, and Retry-After headers are present on every 429 response
• Rate limit counters are namespaced per environment (prod:payments:v2, not payments) to prevent quota bleed across staging and production
• sync_rate ≤ 0.1 s for contractual SLA tiers; synchronous mode for financial or compliance-sensitive routes
• Circuit breaker is configured on the Redis connection pool with exponential back-off and a local-cache fallback
• 429 ratio is exported as a Prometheus metric and alerts fire above 5% over a 5-minute rolling window
• Health check and internal service-to-service routes are bypassed via IP allowlist or a dedicated bypass header
• Window size and limits are documented in the API’s public developer portal and reviewed against consumer usage data quarterly
• Tyk keys are attached to a policy at provisioning time — never left with API-level global defaults as the active quota

FAQ

What is the difference between rate limiting and throttling?

Rate limiting enforces a hard ceiling on the number of requests a consumer can make within a time window — exceeding it returns HTTP 429 immediately. Throttling shapes the flow by queuing or delaying excess requests rather than rejecting them outright, trading latency for availability. Most gateway implementations support both modes via a single plugin, with rejection or queuing as a configurable fallback_action.

Which algorithm should I choose: token bucket, leaky bucket, or sliding window?

Use token bucket when you want to absorb short bursts without penalising bursty-but-infrequent clients — it stores credit up to a configurable burst_size and refills at a steady rate. Use leaky bucket when you need strict output pacing to protect a backend with no tolerance for sudden load spikes. Use sliding window counter when quota accuracy matters more than burst tolerance and you can afford the extra Redis state per consumer.

How do I prevent quota fragmentation across multiple gateway nodes?

Local in-memory counters give each node an independent view of quota, so a consumer can exhaust N times their limit across N nodes. The fix is a shared Redis backend using atomic INCR or Lua scripts, ideally with consistent hashing so a given consumer key always routes to the same Redis shard. Kong’s rate-limiting-advanced plugin and Envoy’s global rate limit service both support this pattern natively — see dynamic rate limiting with Redis backends for the full configuration.

What response headers should a rate-limited API return?

Return X-RateLimit-Limit (ceiling), X-RateLimit-Remaining (quota left in current window), X-RateLimit-Reset (Unix timestamp when the window resets), and Retry-After (seconds until the client may retry) on every 429 response. This gives client SDKs enough information to implement proactive back-off without polling.

Parent: Middleware Chains & Request Transformation

Related

• Dynamic Rate Limiting with Redis Backends — atomic counters, Lua scripts, and hot-reload quota maps for distributed gateway deployments
• NGINX vs Kong for Rate Limiting — head-to-head on counter accuracy, distributed state, and burst handling
• Caching & Response Optimization — deflect traffic at the cache layer to reduce effective load on rate limit counters under sustained high RPS
• Scaling Limits & Capacity Planning — sizing gateway node counts and Redis shard topology for your throughput targets
• Kong vs Tyk vs Envoy for Microservices — capability comparison covering rate limit algorithm support and operational model across vendors
• Security Boundaries & Zero Trust — combining rate limiting with identity-aware policy enforcement to protect against credential-stuffing and API abuse patterns