Gateway Metrics & SLOs

A gateway that cannot describe its own health in numbers is a gateway you operate by anecdote. Every request that crosses the trust boundary is an opportunity to record three facts — did it happen, did it fail, how long did it take — and those three facts are the entire foundation of reliability engineering for a routing layer. This page is part of gateway observability and operations, and it focuses on turning raw request telemetry into the RED signals, service level objectives, and error budgets that let a platform team make release and scaling decisions from evidence rather than intuition. Get the metric shape wrong at the start and you inherit two chronic failures: dashboards that lie because they average the tail away, and a Prometheus instance that falls over under label cardinality it was never sized for.

Design invariants every gateway metrics layer must satisfy:

• Latency is recorded as a histogram with explicit buckets, never as a pre-computed average or a single gauge — percentiles are derived at query time.
• Request and error counters are labelled on the matched route name, not the raw URI, so cardinality is bounded by your route table size.
• Every metric that feeds an SLO is emitted at the gateway trust boundary so rejected, rate-limited, and connection-failed requests are counted, not just those that reached an upstream.
• Config-propagation state is observable: gateway_route_config_version is exported per data-plane node and alerted on when it lags the control plane.
• SLOs are expressed as an SLI query plus a target plus a window, all stored in version control alongside the alerting rules that enforce them.
• Alerting is driven by error-budget burn rate, not by raw error-count thresholds, so pages correlate with user-visible harm.

The RED method as a mental model

Before any exposition format or query language, decide what to measure. For a request-driven system such as a gateway, the RED method gives a complete and minimal signal set:

• Rate — requests per second, the throughput the gateway is handling. Exposed as a monotonic counter (request_total) and rated in queries.
• Errors — the count of failed requests, as a fraction of Rate. This is the raw material of an availability SLI.
• Duration — the distribution of request latency, exposed as a histogram so any percentile can be recovered.

RED is the request-side complement to the USE method (Utilization, Saturation, Errors) that you apply to the gateway’s own resources — CPU, worker saturation, connection-pool occupancy. You need both: RED tells you what users experience, USE tells you why. An SLO is always built on RED signals because an SLO is a promise about user experience, and users experience rate, errors, and latency — not CPU.

Which metrics to expose

The temptation is to export everything the gateway can emit. Resist it — every series has a storage and query cost, and a smaller, deliberately chosen set is easier to reason about. A production gateway should expose at minimum:

• request_total{route, status, upstream} — a counter of requests, labelled by matched route name, HTTP status (or status class), and the selected upstream cluster. This single counter yields both Rate (its derivative) and Errors (the fraction with 5xx status).
• request_duration_seconds_bucket{route, le} — a histogram of end-to-end latency with explicit buckets. Keep a separate upstream-latency histogram if you need to distinguish gateway-added time from upstream time.
• upstream_connections_active{cluster} and upstream_connections_pending{cluster} — gauges tracking connection-pool occupancy. These are your leading indicator for pool exhaustion, which produces 503s before a circuit breaker has enough samples to open.
• gateway_route_config_version — a gauge exporting the generation of the routing table each data-plane node currently holds. If a node lags the control plane by more than one generation during a push, that is a config-propagation incident, not a normal state.

Notice what is not here: no per-request-ID label, no raw path, no per-consumer label by default. Those are the cardinality traps covered below.

Kong 3.x — Prometheus plugin exposition

Kong exposes metrics through the bundled prometheus plugin, enabled globally so every service and route is covered. The critical knobs are the toggles that control which label dimensions are emitted, because they are the difference between a healthy series count and a cardinality explosion.

# Kong 3.x — declarative config enabling Prometheus exposition
_format_version: "3.0"

plugins:
  - name: prometheus
    config:
      status_code_metrics: true      # request counts by status code
      latency_metrics: true          # request + upstream latency histograms
      bandwidth_metrics: true        # ingress/egress byte counters
      upstream_health_metrics: true  # target health gauges per upstream
      per_consumer: false            # CRITICAL: leave off unless budgeted

With this enabled Kong serves an OpenMetrics endpoint on the admin/status listener at /metrics, exposing kong_http_requests_total, kong_request_latency_ms_bucket, kong_upstream_target_health, and kong_datastore_reachable. Scrape it with a standard Prometheus job pointed at the status port:

# Prometheus scrape config for a Kong data-plane fleet
scrape_configs:
  - job_name: kong-gateway
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets: ["kong-dp-1:8100", "kong-dp-2:8100"]

The single most consequential line is per_consumer: false. Turning it on multiplies every request series by the number of distinct consumers — on a platform with 50,000 API keys that is catastrophic. Enable it only for a small, named set of high-value consumers via a scoped plugin, never globally.

Envoy 1.32+ — stats sinks and histogram buckets

Envoy computes stats internally and exposes them through a stats sink. For Prometheus you scrape the admin endpoint’s /stats/prometheus path, but the important configuration is the histogram bucket definition, because Envoy’s default buckets are coarse and you will want custom le boundaries aligned to your SLO thresholds.

# Envoy 1.32+ — bootstrap stats config with custom histogram buckets
stats_config:
  stats_tags:
    - tag_name: envoy_cluster_name
      regex: "^cluster\\.((.+?)\\.)"
  histogram_bucket_settings:
    - match:
        prefix: "cluster."
      buckets: [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000]

# Admin listener exposes /stats/prometheus for scraping
admin:
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

# Prometheus job for Envoy, filtering to the metrics you actually use
scrape_configs:
  - job_name: envoy-gateway
    metrics_path: /stats/prometheus
    scrape_interval: 15s
    static_configs:
      - targets: ["envoy-edge-1:9901", "envoy-edge-2:9901"]
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "envoy_cluster_upstream_(rq_xx|rq_time_bucket|cx_active)"
        action: keep

The metric_relabel_configs keep action is doing real work here: Envoy emits thousands of stats by default, and dropping everything except the series you build SLOs and dashboards from keeps your Prometheus footprint small. Custom buckets aligned to 100, 250, 500, 1000 ms make your P99 latency SLI query accurate near the threshold that matters, rather than interpolating across a 1-second-wide default bucket.

Defining SLIs, SLOs, and error budgets

A service level indicator (SLI) is a carefully defined ratio of good events to total events. For a gateway the two canonical SLIs are:

• Availability SLI — the fraction of requests that did not return a server error: 1 - (sum(rate(5xx)) / sum(rate(all))). Client errors (4xx) are generally excluded because a 400 is the client’s fault, not a reliability failure of the gateway.
• Latency SLI — the fraction of requests served faster than a threshold, computed from the histogram: the count in buckets at or below your target le divided by the total count.

A service level objective (SLO) is an SLI plus a target plus a window: “99.9% of requests succeed, measured over a rolling 30 days.” The complement of the target is the error budget — with a 99.9% availability SLO you are permitted a 0.1% failure rate, which over 30 days is roughly 43 minutes of full downtime, or a much longer period of partial degradation. That budget is the currency that connects reliability to release velocity: while it is healthy, teams ship; when it drains, releases pause. Encode SLIs as Prometheus recording rules so the expensive ratio is computed once and reused by both dashboards and alerts:

# Prometheus recording rules — gateway availability SLI over multiple windows
groups:
  - name: gateway-slo-sli
    rules:
      - record: gateway:request_error_ratio:rate5m
        expr: |
          sum(rate(kong_http_requests_total{code=~"5.."}[5m]))
          /
          sum(rate(kong_http_requests_total[5m]))
      - record: gateway:request_error_ratio:rate1h
        expr: |
          sum(rate(kong_http_requests_total{code=~"5.."}[1h]))
          /
          sum(rate(kong_http_requests_total[1h]))

The companion walkthrough on defining SLOs for gateway latency and error budgets works the full example of picking targets, choosing windows, and wiring the burn-rate rules end to end.

Alerting on burn rate, not on error counts

The naive alert — “page when the 5xx rate exceeds 1%” — fails in both directions. It pages for a brief harmless blip and stays silent through a slow leak that quietly exhausts the monthly budget. The fix is burn-rate alerting: alert on how fast the error budget is being consumed relative to the SLO window.

Burn rate is a multiplier. A burn rate of 1 means you are consuming budget exactly fast enough to exhaust it at the end of the window; a burn rate of 14.4 means you would exhaust a 30-day budget in roughly two days. Google’s SRE workbook recommends a multi-window multi-burn-rate scheme: a fast-burn page (high threshold, short window plus a short confirmation window) for severe outages, and a slow-burn ticket (lower threshold, long window) for gradual erosion.

# Prometheus alerting rules — multi-window burn-rate for a 99.9% SLO
groups:
  - name: gateway-slo-burn
    rules:
      # Fast burn: 14.4x over 1h, confirmed by 5m window -> PAGE
      - alert: GatewayErrorBudgetFastBurn
        expr: |
          gateway:request_error_ratio:rate1h > (14.4 * 0.001)
          and
          gateway:request_error_ratio:rate5m > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "Gateway burning error budget 14.4x — outage-level"

      # Slow burn: 3x over 6h, confirmed by 30m -> TICKET
      - alert: GatewayErrorBudgetSlowBurn
        expr: |
          gateway:request_error_ratio:rate6h > (3 * 0.001)
          and
          gateway:request_error_ratio:rate30m > (3 * 0.001)
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "Gateway slow error-budget leak — 3x burn over 6h"

The confirmation window (the shorter of the two conditions in each and) prevents a single scrape spike from firing a page: both the long and short windows must agree. This is the alerting shape that keeps on-call sane. Because the SLI is measured at the gateway, these alerts fire on user-visible harm — including requests rejected before they reached an upstream, which is exactly where circuit breaking and retry budgets interact with your error budget: a breaker that opens correctly protects the budget by shedding load, while a misconfigured retry storm burns it.

Controlling cardinality

Cardinality is the number of distinct time series, and it is the resource that kills Prometheus instances. Each unique combination of a metric name and its label values is one series, consuming memory in the head block and disk in the TSDB. The failure mode is insidious: everything works in staging with ten routes and three consumers, then production with real traffic detonates the series count.

The rules that keep cardinality bounded:

• Label on the matched route name, never the raw path. /api/v1/users/12345 and /api/v1/users/67890 must both map to the label route="users-detail". Raw paths with embedded IDs are unbounded.
• Never label on user, request, or trace IDs. These belong in logs and traces, not in metrics. This is where the boundary between metrics and structured logging matters — high-cardinality context lives in logs.
• Bucket status codes into classes where you can. status="5xx" is cheaper than the full three-digit code and usually sufficient for an availability SLI.
• Keep per_consumer metrics off by default and scope them to a named allowlist when you genuinely need per-tenant reliability numbers.

The safe estimate for series count is the product of label cardinalities: routes × status classes × upstreams × instances. Ten routes, six status classes, ten upstreams, and twenty gateway instances is already 12,000 series for a single metric family — manageable. Swap the route label for a raw path and that first factor jumps from 10 to millions.

Comparative implementation table

Operational gotchas

Averaging latency instead of using percentiles. A dashboard panel showing avg(request_latency) is worse than no panel because it inspires false confidence. The mean is dominated by the fast majority and blind to the slow tail that an SLO exists to protect. Always use histogram_quantile(0.99, ...) over a _bucket series. If you only have a gauge and no histogram, you cannot compute a percentile at all — fix the exposition first.

Cardinality explosion from an innocent-looking label. Adding a label “just to slice by customer” can multiply your series count by five orders of magnitude. Every new label must be justified against its cardinality. Audit series growth after each change with topk(10, count by (__name__)({__name__=~".+"})).

Counter resets on gateway restart. Prometheus counters reset to zero when a process restarts. Always wrap counters in rate() or increase(), which handle resets correctly — never subtract raw counter values across a scrape.

Histogram bucket boundaries that miss the SLO threshold. If your SLO is “P99 under 250 ms” but your nearest histogram buckets are le=100 and le=500, histogram_quantile interpolates linearly across a 400 ms-wide bucket and your P99 estimate can be off by a hundred milliseconds. Define a bucket boundary exactly at every SLO threshold.

Scrape interval longer than the alert window. A 60-second scrape interval feeding a rule that evaluates over a 5-minute window gives you only five data points — enough to be noisy. Keep scrape interval at or below one-quarter of your shortest alert window, typically 15 seconds.

As request volume grows, the metrics layer itself becomes a scaling concern; the interaction between series count, scrape load, and gateway throughput is covered in scaling limits and capacity planning.

Production Configuration Checklist

• Latency is exposed as a histogram with explicit buckets, with a boundary at every SLO threshold — no averaged gauges.
• request_total is labelled on matched route name, not raw URI path.
• per_consumer (Kong) and equivalent high-cardinality labels are disabled globally and scoped to an allowlist where needed.
• Envoy scrapes use metric_relabel_configs to keep only the series that feed dashboards and SLOs.
• SLIs are stored as Prometheus recording rules in version control, not computed ad hoc in dashboards.
• Every SLO is expressed as SLI + target + window and documented alongside its alerting rules.
• Burn-rate alerts use a multi-window multi-burn-rate scheme with a short confirmation window on each condition.
• gateway_route_config_version is exported per node and alerted on when it lags the control plane.
• Counters are always queried through rate() or increase() to survive restarts.
• Scrape interval is at most one-quarter of the shortest alert window.
• Series growth is audited after every metrics-config change with a count by (__name__) query.

FAQ

Why is averaging gateway latency a mistake when defining SLOs?

An average hides the tail. A gateway serving 10,000 fast requests and 100 requests that take five seconds shows a healthy mean while a meaningful fraction of users experience timeouts. SLOs must be defined on percentiles computed from a latency histogram — P95, P99, P99.9 — because those quantiles describe the experience of your slowest users, which is exactly the population an SLO is meant to protect. Always expose a histogram metric with explicit buckets and derive quantiles with histogram_quantile rather than exporting a pre-averaged gauge.

What causes label cardinality explosion in gateway metrics and how do I prevent it?

Cardinality explosion happens when a label takes unbounded distinct values — raw URI paths with IDs, per-consumer identifiers, or full status text. Each unique label-set combination is a separate time series in Prometheus, and the product of label values multiplies fast. Prevent it by labelling on the matched route name rather than the raw path, bucketing status into classes where possible, never labelling on user or request IDs, and disabling per_consumer in Kong’s prometheus plugin unless you have deliberately budgeted for the series count.

How do error budgets connect gateway metrics to release decisions?

An error budget is the allowed unreliability implied by an SLO: a 99.9% availability target over 30 days permits roughly 43 minutes of downtime. You compute budget consumption directly from the gateway’s request and error counters. When the budget is healthy, teams ship freely; when burn-rate alerts fire because the budget is draining too fast, non-critical releases pause and reliability work is prioritised. The gateway’s request_total and error counters are the source of truth for that calculation.

What is multi-window multi-burn-rate alerting and why use it at the gateway?

Multi-window multi-burn-rate alerting pairs a fast-burn condition over a short window with a slow-burn condition over a long window, and requires a short confirmation window to agree before firing. This gives you a page for a severe outage that would exhaust the monthly budget in hours, and a ticket for a slow leak that would exhaust it over days, without the false pages a single-threshold alert produces. At the gateway it is computed from the same error-ratio SLI so paging tracks user-visible impact rather than raw error spikes.

Should I run SLOs off gateway metrics or upstream service metrics?

The gateway is the best place to measure a user-facing SLO because it observes every request at the trust boundary, including requests that never reached an upstream because they were rejected, rate-limited, or failed to connect. Upstream metrics miss connection-pool exhaustion and gateway-originated 5xx responses. Measure availability and latency SLIs at the gateway for the user-facing contract, and keep per-service metrics for drilling into which upstream caused a budget burn.

Related

• Defining SLOs for Gateway Latency and Error Budgets — the step-by-step worked example of picking targets, windows, and burn-rate rules.
• Circuit Breaking & Retry Budgets — how load-shedding primitives protect or burn the error budget you measure here.
• Scaling Limits & Capacity Planning — sizing the gateway and its metrics pipeline as request volume and series count grow.
• Gateway Observability & Operations — the parent overview covering tracing, structured logging, and metrics together.

← Gateway Observability & Operations