Defining SLOs for Gateway Latency and Error Budgets

The gateway is the one place in a distributed system that sees every user-facing request, which makes it the natural point to define the reliability contract a platform team commits to. But an SLO that is picked from a round number — “let’s do four nines” — without connecting it to real traffic and a working alert is theatre. This how-to walks the concrete sequence: choose the SLIs, set defensible targets and windows, turn the target into an error budget, and wire multi-window burn-rate alerts to the gateway’s own metrics so that when the budget drains, someone is paged for the right reason. The stakes are direct — an SLO that is too tight pages on-call for noise until they mute it, and one that is too loose lets a slow reliability leak run for weeks before anyone notices.

Prerequisites

This page assumes you have already instrumented the gateway as described in gateway metrics and SLOs: a request_total counter labelled on matched route and status, and a request_duration_seconds histogram with buckets aligned to your latency thresholds. It also assumes the broader operational context of gateway observability and operations — that metrics sit alongside tracing and structured logs, and that you have a Prometheus instance scraping the gateway. If your latency is currently exposed as an average rather than a histogram, stop and fix the exposition first: you cannot build a latency SLO on a mean.

Step 1 — Pick the SLIs

An SLI is a ratio of good events to total events. For a gateway, define exactly two to start:

Availability SLI — the fraction of requests the gateway served without a server-side failure. Good events are all non-5xx responses; total is all responses. Exclude 4xx, which are client faults.

# Availability SLI: fraction of requests that were NOT server errors, 5m window
sum(rate(kong_http_requests_total{code!~"5.."}[5m]))
/
sum(rate(kong_http_requests_total[5m]))

Latency SLI — the fraction of requests served faster than a threshold. This is computed from the histogram as a good-over-total count, not by computing a percentile value. If your target is “P99 under 300 ms,” the SLI is the fraction of requests in buckets with le="0.3" or below:

# Latency SLI: fraction of requests served in <= 300ms, 5m window
sum(rate(kong_request_latency_ms_bucket{le="300"}[5m]))
/
sum(rate(kong_request_latency_ms_count[5m]))

Expressing the latency SLI as a threshold ratio (rather than histogram_quantile) is what makes the burn-rate math below work, because burn rate operates on a fraction of bad events. Save histogram_quantile for the dashboard panel that shows the actual P99 value over time.

Step 2 — Set targets and windows

Pick the target from the value users actually need, then check it against what the current data shows you can sustain. Query the trailing 30 days of each SLI before committing — an aspirational target you are already missing every week only trains on-call to ignore the pager.

Use a rolling 30-day window as the SLO period that governs alerting. A rolling window moves continuously and avoids the artificial budget reset a calendar month creates. Keep a calendar-quarter view purely for stakeholder reporting.

The decision matrix below maps common gateway tiers to defensible targets. Match a route to a tier by its blast radius, not by wishful thinking.

Monthly error budget shrinks sharply as the availability target adds nines Four horizontal bars showing minutes of allowed downtime per 30-day window for four SLO targets. 99 percent allows about 432 minutes. 99.9 percent allows about 43 minutes. 99.95 percent allows about 21 minutes. 99.99 percent allows about 4 minutes. Each added nine cuts the budget by roughly ten times. 99% 432 min / 30d 99.9% 43 min / 30d 99.95% 21 min / 30d 99.99% 4 min / 30d each added nine cuts the monthly downtime budget by roughly 10x
Tier Example traffic Availability target Latency SLO Rationale
Critical / revenue Checkout, payments, auth issuance 99.95% P99 < 250 ms User-visible, money-losing; tight budget justifies on-call cost
Standard user-facing Product APIs, dashboards 99.9% P99 < 400 ms The default for most public routes; ~43 min/month budget
Internal / best-effort Admin tooling, batch triggers 99.5% P95 < 800 ms Failures annoy staff, not customers; looser budget saves toil
Async / background Webhook fan-out, report jobs 99% P95 < 2 s Latency largely invisible to end users; generous budget

Step 3 — Compute the error budget

The error budget is the complement of the target. For a 99.9% availability SLO the budget is 0.1% of requests. Over a 30-day window that is about 43 minutes of total unavailability, or proportionally more time at partial degradation. Express remaining budget as a recording rule so both a dashboard gauge and the burn alerts can read it:

# Prometheus recording rules — error budget remaining for a 99.9% SLO
groups:
  - name: gateway-slo-budget
    rules:
      # 30-day error ratio (the SLI complement) via a long-window recording rule
      - record: gateway:request_error_ratio:rate30d
        expr: |
          sum(rate(kong_http_requests_total{code=~"5.."}[30d]))
          /
          sum(rate(kong_http_requests_total[30d]))
      # Fraction of budget still available: 1 means untouched, 0 means exhausted
      - record: gateway:error_budget_remaining:ratio
        expr: 1 - (gateway:request_error_ratio:rate30d / 0.001)

When gateway:error_budget_remaining:ratio crosses zero you have spent the month’s budget; a healthy value near 1 is your license to keep shipping.

Step 4 — Wire multi-window burn-rate alerts

Burn rate is how fast you are spending the budget relative to the window. A burn rate of 1 exhausts the budget exactly at the 30-day mark; 14.4 exhausts it in about two days. The standard is two alerts — a fast-burn page and a slow-burn ticket — each with a short confirmation window so a lone scrape spike cannot trip it.

# Prometheus alerting rules — two-tier burn rate for a 99.9% SLO (budget = 0.001)
groups:
  - name: gateway-slo-alerts
    rules:
      # PAGE: 14.4x burn over 1h, confirmed by 5m window
      - alert: GatewayFastBurn
        expr: |
          gateway:request_error_ratio:rate1h  > (14.4 * 0.001)
          and
          gateway:request_error_ratio:rate5m  > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "Gateway fast burn: budget exhausts in ~2 days at this rate"

      # TICKET: 3x burn over 6h, confirmed by 30m window
      - alert: GatewaySlowBurn
        expr: |
          gateway:request_error_ratio:rate6h  > (3 * 0.001)
          and
          gateway:request_error_ratio:rate30m > (3 * 0.001)
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "Gateway slow burn: gradual budget leak over hours"

The same pattern applies to the latency SLI: swap the error-ratio recording rules for 1 - latency_SLI and reuse the identical burn-rate thresholds. This keeps latency and availability alerting mechanically consistent. Because these fire from metrics measured at the gateway trust boundary, they capture requests that never reached an upstream — including load shed by circuit breaking and retry budgets, where a correctly opened breaker protects the budget and a retry storm burns it.

Decision matrix: which alert tier for which symptom

Symptom Window pair Burn rate Routing Why
Total upstream outage 1h and 5m 14.4x Page Budget gone in ~2 days; wake someone
Partial 5xx spike on one route 1h and 5m 14.4x Page Still fast enough to threaten the month
Slow latency creep past threshold 6h and 30m 3x Ticket Gradual; fix in business hours
Elevated 429 throttling separate SLI n/a Dashboard Intended protection, track but don’t page

Gotchas and failure signals

Alerting on the SLI value instead of burn rate. A rule that pages when the 5m error ratio exceeds the raw budget fires constantly during normal variance. Always multiply the budget by a burn-rate factor and require a confirmation window.

Latency SLO with a bucket that misses the threshold. If the SLO is P99 under 300 ms but the histogram has no le="300" bucket, your SLI silently measures the wrong threshold. Add a bucket boundary at exactly each SLO threshold when you instrument the gateway.

Counting 4xx as budget burn. Including client errors inflates the error ratio and burns budget for failures you did not cause. Scope the “bad” selector to 5.. plus explicit connection-failure metrics.

A 30-day rate() that is expensive or empty. Long-window rate() over raw counters is heavy and can return no data if the counter has not existed for 30 days. Precompute it as a recording rule and backfill expectations for newly created SLOs.

Muting instead of adjusting. If an alert pages repeatedly for non-incidents, the target is wrong, not the alert. Re-derive the target from trailing data rather than silencing the rule — a muted SLO is no SLO.

Validation

  • Availability and latency SLIs are defined as good-over-total ratios and stored as recording rules.
  • The latency SLI uses a threshold-bucket ratio, with a histogram boundary at exactly the SLO threshold.
  • Targets were checked against trailing 30-day data before being committed, not chosen as round numbers.
  • The SLO window is a rolling 30 days for alerting; calendar views are reporting-only.
  • Error-budget-remaining is a recording rule readable by both dashboards and alerts.
  • Exactly two burn-rate alerts exist (fast-burn page, slow-burn ticket), each with a confirmation window.
  • The “bad events” selector counts 5xx and connection failures but excludes 4xx.
  • Alerts have been fire-tested by injecting synthetic upstream failures.

FAQ

What SLO window should I use for a gateway — rolling or calendar?

Use a rolling 30-day window for the SLO that governs alerting, because it moves continuously and avoids the artificial budget reset that a calendar month creates on the first of the month. Keep a calendar-quarter view for reporting to stakeholders who think in fiscal periods. The burn-rate alerts themselves evaluate over short windows of minutes to hours; the 30-day figure is the denominator that those short windows are measured against.

How do I compute a P99 latency SLI from a Prometheus histogram?

For an SLO stated as a threshold, do not compute the P99 value and compare it. Instead compute the fraction of requests faster than the threshold directly from the bucket counts: divide the rate of requests in buckets at or below your target le boundary by the total request rate. That ratio is the latency SLI, and it is what you compare against the target. Computing histogram_quantile is fine for dashboards, but the threshold-ratio form is the correct SLI for burn-rate math because it is a good-events-over-total-events count.

Which requests count as bad for a gateway availability SLI?

Count server-side failures the gateway is responsible for: 5xx responses, connection failures to upstreams, and requests dropped due to connection-pool exhaustion. Exclude 4xx client errors, since a 400 or 401 reflects a bad request, not gateway unreliability. Rate-limit 429s are a judgement call: if throttling is your intended protective behaviour, exclude them; if they indicate under-provisioning that harms users, track them in a separate SLI so they stay visible.

How many burn-rate alerts should a gateway SLO have?

Two is the practical standard: a fast-burn page and a slow-burn ticket. The fast-burn alert uses a high burn-rate multiplier over a short window to catch outages that would exhaust the budget in hours. The slow-burn alert uses a lower multiplier over a longer window to catch gradual erosion. Each condition is paired with a shorter confirmation window so a single scrape spike cannot fire it. Adding more tiers rarely improves signal and usually adds noise.

Parent: Gateway Metrics & SLOs