Prometheus Recording Rules for Golden Signals

Golden-signal dashboards built from raw histogram queries are slow, and slow dashboards get abandoned during exactly the incident they were built for. Recording rules precompute the expensive parts on a schedule, so a panel that took eight seconds takes fifty milliseconds. This page covers the rules worth writing for a gateway, and the naming discipline that keeps them usable.

Prerequisite concepts

This assumes the metric set and cardinality material in gateway metrics and SLOs, and it produces the queries consumed by defining SLOs for gateway latency and error budgets.

Why a raw quantile query is slow

The cost is in the buckets, and it is paid on every dashboard refresh A p99 over forty routes with twelve histogram buckets and eight gateway instances touches nearly four thousand series each time the panel loads. A recording rule evaluates that once per interval and stores one series per route, which the dashboard then reads directly. series read per panel refresh raw quantile query 40 routes × 12 buckets × 8 instances ≈ 3,840 series, every refresh recording rule 40 series — one per route, precomputed once per interval The saving compounds: an incident dashboard with eight panels over a six-hour window is the difference between a page that loads instantly and one that times out while someone is trying to read it. Alerting rules benefit identically, and an alert that evaluates slowly is an alert that fires late.

The rules worth writing

groups:
  - name: gateway-golden-signals
    interval: 30s
    rules:
      # rate
      - record: route:gateway_requests:rate5m
        expr: sum by (route, code_class) (rate(gateway_requests_total[5m]))

      # errors, as a ratio — the form every SLO query needs
      - record: route:gateway_errors:ratio5m
        expr: |
          sum by (route) (rate(gateway_requests_total{code_class="5xx"}[5m]))
          /
          sum by (route) (rate(gateway_requests_total[5m]))

      # duration — the expensive one
      - record: route:gateway_duration:p99_5m
        expr: histogram_quantile(0.99, sum by (le, route) (rate(gateway_request_duration_bucket[5m])))
      - record: route:gateway_duration:p50_5m
        expr: histogram_quantile(0.50, sum by (le, route) (rate(gateway_request_duration_bucket[5m])))

      # saturation — gateway overhead, separated from upstream time
      - record: route:gateway_overhead:p99_5m
        expr: |
          histogram_quantile(0.99, sum by (le, route) (rate(gateway_request_duration_bucket[5m])))
          -
          histogram_quantile(0.99, sum by (le, route) (rate(gateway_upstream_duration_bucket[5m])))
  # longer windows for burn-rate alerting, built from the same source
  - name: gateway-burn-windows
    interval: 60s
    rules:
      - record: route:gateway_errors:ratio1h
        expr: |
          sum by (route) (rate(gateway_requests_total{code_class="5xx"}[1h]))
          / sum by (route) (rate(gateway_requests_total[1h]))
      - record: route:gateway_errors:ratio6h
        expr: |
          sum by (route) (rate(gateway_requests_total{code_class="5xx"}[6h]))
          / sum by (route) (rate(gateway_requests_total[6h]))

Note that the error ratio is recorded rather than the error count. Ratios compose into burn-rate alerts directly; counts have to be divided again at query time, which reintroduces the cost the rule was meant to remove.

Naming, and why it matters more than it seems

Three components, and each one answers a question The level says what the series is aggregated by, the metric names what is measured, and the operation says what was done to it and over what window. A name following this convention tells a reader whether the series is the one they want without opening the rule definition. route:gateway_duration:p99_5m route gateway_duration p99_5m aggregated by this label what is measured operation and window Without a convention, a metrics store accumulates a dozen near-identical rules nobody trusts, and every dashboard author writes a thirteenth rather than working out which existing one is right. Include the window in the name — two rules that differ only by window and share a name are indistinguishable. A rule group that cannot finish inside its interval serves stale data A group evaluating in eight seconds on a thirty second interval has ample headroom. At twenty-six seconds it is close to the limit and one slow evaluation puts it behind. Past the interval it falls behind permanently and the recorded series silently become stale, which no dashboard indicates. evaluation duration against a 30 s interval 30 s 8 s healthy headroom 26 s close to the limit 38 s permanently behind — series are stale and nothing says so Split large groups rather than lengthening the interval: a longer interval makes every alert built on the group slower to fire.

A final practical note: add the rules incrementally rather than in one large group. A single group containing every rule you might want evaluates as a unit, so one expensive expression delays every other series in it, and the resulting staleness is attributed to whichever dashboard notices first rather than to the rule that caused it.

Decision matrix

Query used for Record it?
A dashboard panel loaded during incidents yes
An alerting rule with a long window yes
An ad-hoc investigation no — the cost is paid once
A quantile over many series yes, always
A simple counter rate over one series no — it is already cheap

Gotchas and failure signals

Quantiles do not average. Recording a p99 per instance and then averaging those series produces a number with no statistical meaning. Aggregate the buckets first, then take the quantile — which is what the rules above do.

A recording rule that takes longer than its interval falls behind silently and produces stale series. Watch the rule evaluation duration, and split large groups.

Changing a rule’s expression without changing its name leaves history that means one thing and new data that means another, in one series. Version the name when the meaning changes.

Recorded series inherit the cardinality of their inputs. A rule aggregating by a high-cardinality label produces a high-cardinality output, and the rule now runs on every evaluation interval rather than only when someone opens a dashboard.

There is a related discipline worth adopting alongside the rules themselves: keep them in the same repository as the dashboards and alerts that consume them, and review changes to all three together. A recording rule is an interface, and the moment it lives somewhere different from its consumers it acquires the usual problems of an interface with no owner — renamed without notice, duplicated because nobody found the original, and eventually distrusted enough that dashboard authors go back to writing raw queries.

Validation

  • Every incident-dashboard panel reads a recorded series, not a raw quantile
  • Error ratios recorded as ratios, not counts
  • Rule evaluation duration monitored against the group interval
  • Naming convention documented and applied to every rule
  • Bucket aggregation happens before the quantile, everywhere
  • Burn-rate windows recorded for each window used by an alert
  • Rules, dashboards and alerting rules live in one repository and are reviewed together, so a rename cannot land without its consumers being updated in the same change
  • Each rule has a comment naming the dashboard or alert it exists for, so an unused rule can be deleted with confidence rather than kept indefinitely

FAQ

Which gateway queries are worth recording?

Anything a dashboard loads during an incident, anything an alert evaluates over a long window, and every quantile over many series. A p99 across forty routes with twelve buckets and eight instances touches nearly four thousand series on each refresh; the recording rule evaluates it once per interval and stores one series per route. A simple counter rate over a handful of series is already cheap and does not need one.

Why record ratios instead of counts?

Because ratios compose directly into burn-rate alerts, while counts have to be divided again at query time — which reintroduces exactly the cost the rule was written to remove. Record the error ratio per route per window, and the alerting expression becomes a comparison against a number rather than another aggregation.

Can I average recorded p99 series?

No. Quantiles do not average: taking a p99 per instance and then averaging those values produces a number with no statistical meaning. Aggregate the histogram buckets across instances first and take the quantile afterwards, which is why the recording rules sum by the bucket boundary label before calling the quantile function.

What goes wrong with recording rules over time?

Two things. A rule group that takes longer to evaluate than its interval falls behind silently and serves stale series, so watch evaluation duration against interval. And a rule whose expression changes without its name changing leaves one series where the history means something different from the new data — version the name whenever the meaning changes.

Parent: Gateway Metrics & SLOs