Outlier Detection vs Active Health Checks

Active health checks ask every upstream host whether it is well. Outlier detection watches real traffic and ejects hosts that are behaving badly. They sound like alternatives and are complements, because they detect different failures on different timescales — and running only one of them leaves a specific, common gap.

Prerequisite concepts

This assumes the resilience primitives in circuit breaking and retry budgets and the failover behaviour in high availability topologies.

What each one sees

A synthetic probe and a real observation answer different questions An active health check calls a dedicated endpoint on a fixed interval, so it detects a host that is down or whose health endpoint reports a dependency failure. Outlier detection counts the outcomes of real requests, so it detects a host that answers the probe correctly while failing the traffic that matters. active health check gateway every 5 s GET /healthz detects: process down, dependency the health endpoint knows about outlier detection gateway real traffic count 5xx per host detects: a host that passes the probe and fails the actual requests The second row is the gap: a host with a corrupt cache, a bad config generation or a failing dependency the health endpoint does not consult will answer /healthz with 200 all day. 
# Envoy 1.32+ — both, on one cluster
clusters:
  - name: orders_cluster
    health_checks:
      - timeout: 1s
        interval: 5s
        unhealthy_threshold: 3          # 15 s to eject a dead host
        healthy_threshold: 2
        http_health_check: { path: "/healthz" }
    outlier_detection:
      consecutive_5xx: 5                # eject after 5 consecutive failures
      interval: 10s                     # evaluation sweep
      base_ejection_time: 30s           # doubles on each successive ejection
      max_ejection_percent: 30          # never eject more than 30% of the pool
      enforcing_consecutive_5xx: 100
      split_external_local_origin_errors: true

max_ejection_percent is the setting that prevents outlier detection from causing the outage it exists to prevent. Without it, a bad deploy that makes every host return 500 results in every host being ejected, and an upstream with no healthy members either fails all requests or — depending on the panic-threshold configuration — reverts to sending traffic to everyone anyway.

The panic threshold, and why it exists

Below the panic threshold, health status is ignored on purpose While most hosts are healthy, traffic goes only to healthy hosts. Once the healthy proportion falls below the panic threshold, the load balancer distributes across all hosts including unhealthy ones, on the reasoning that an upstream reporting almost everything unhealthy is more likely to have a broken health check than a broken fleet. healthy hosts in this upstream 80% healthy healthy only 60% healthy healthy only 40% healthy below the 50% panic threshold — all hosts used again The reasoning: an upstream reporting 60% of hosts unhealthy is more often a broken health check than a broken fleet, and sending traffic to possibly-degraded hosts beats sending it nowhere. Surprising during an incident if you do not know it exists — put it in the runbook. Why aggressive ejection can consume its own cluster Six hosts share the load. Two are ejected for returning errors, so the remaining four each absorb fifty percent more traffic. That extra load pushes two of them past their latency ceiling, they begin failing, and they are ejected in turn — leaving two hosts carrying everything. load per surviving host as ejection proceeds 6 hosts healthy 17% each — comfortable 2 ejected, 4 left 25% each — latency starts climbing 4 ejected, 2 left 50% each — they fail too max_ejection_percent stops the cascade at a chosen point; a retry budget stops retries from adding to the load that causes the next ejection. Both are needed — either alone leaves one half of the loop running.

Decision matrix

Failure Detected by
Process crashed or port closed active health check
Host draining for a deploy active health check
Dependency the health endpoint checks is down active health check
Host returns 500 on real routes, 200 on /healthz outlier detection
One host far slower than its peers outlier detection, on a latency signal
Entire fleet failing after a bad deploy neither — that is a rollback

Gotchas and failure signals

A health endpoint that returns 200 whenever the process is alive makes active checks nearly useless. It should exercise the dependencies the service needs, and it should be cheap enough to call every few seconds.

Ejection is per gateway node. Each node forms its own view, so a host may serve some nodes and not others. That is usually fine and is confusing during an incident if you expect a single global decision.

Aggressive ejection plus retries can cascade. Ejecting hosts concentrates load on survivors, which makes them slower, which ejects them too. max_ejection_percent and a retry budget together bound this.

Local origin errors are not upstream errors. A connection failure caused by the gateway’s own resource exhaustion will eject a perfectly healthy host unless local and external errors are counted separately.

Latency-based ejection, and why it needs care

Ejecting on error rate is uncontroversial: a host returning 500s is not serving. Ejecting on latency is more useful and considerably more dangerous, because latency is relative and a host can be slow for reasons that have nothing to do with its health.

The mechanism compares each host’s mean response time against the fleet-wide mean and ejects those that are a configured number of standard deviations above it. That works well when hosts are homogeneous and traffic is evenly distributed. It works badly when one host handles a different mix of requests — a shard with larger objects, a zone with more distant clients, an instance that happens to be serving the one expensive endpoint — because the model reads a legitimate difference as a fault.

Two settings make it safe. A minimum sample size stops a host being ejected on the basis of three slow requests. And a minimum absolute threshold means a host is never ejected for being slower than its peers if it is still comfortably inside the latency budget: being twice as fast as necessary is not a reason to eject the one host that is merely fast enough.

Validation

  • Both mechanisms configured; neither is a substitute for the other
  • Health endpoint exercises real dependencies and is cheap to call
  • max_ejection_percent set below 100 and understood
  • Panic threshold behaviour documented in the runbook
  • Local and external origin errors counted separately
  • Ejection events exported as a metric with the host label

FAQ

Do I need both?

Yes, because they detect different failures. Active checks find a host that is down, draining, or whose health endpoint knows about a failed dependency. Outlier detection finds a host that answers the probe with 200 while returning 500 on the routes that matter — a corrupt cache, a bad config generation, a dependency the health endpoint does not consult. Neither substitutes for the other.

What does max_ejection_percent protect against?

Outlier detection causing the outage it exists to prevent. A bad deploy that makes every host fail would otherwise eject every host, leaving an upstream cluster with no members. Capping ejection at something like thirty percent keeps a working majority in rotation and makes the underlying problem visible as errors rather than as a total loss of upstream.

Why does traffic sometimes go to hosts marked unhealthy?

The panic threshold. When the healthy proportion of an upstream cluster falls below a configured level, typically fifty percent, the load balancer distributes across all hosts again including unhealthy ones. The reasoning is that an upstream reporting most hosts unhealthy is more often a broken health check than a broken fleet. It is correct and extremely surprising mid-incident, so it belongs in the runbook.

Should health check failures and request failures be counted together?

No — and Envoy lets you separate local origin errors from external ones for exactly this reason. A connection failure caused by the gateway’s own resource exhaustion is not evidence that the upstream host is unwell, and counting it as such ejects healthy hosts precisely when the fleet is already under pressure.

Parent: Circuit Breaking & Retry Budgets