Gateway Observability & Operations

An API gateway sits on the critical path of every request that crosses your perimeter, which makes it the single most valuable — and most dangerous — vantage point in a distributed system. When it is observable, it is the first place an operator looks during an incident: one query tells you which route degraded, which upstream is shedding load, and which deploy generation a misbehaving node is running. When it is not, the gateway becomes a black box that terminates connections, rewrites headers, and makes routing decisions with no record of why, turning every incident into an archaeology project. This page covers how to make a gateway both observable and operable in production — the three telemetry pillars of tracing, metrics, and logging, plus the incident-response, config-propagation, and rollout disciplines that turn raw signals into control. It builds directly on the routing and policy mechanics established across API gateway fundamentals and architecture, the middleware chains and request transformation layer, and advanced routing and API versioning.

Design invariants every observable gateway must satisfy:

• Every request emits all three signals — a trace span, RED metrics, and one structured log line — on the success path, not only when something fails.
• A single traceparent correlates the gateway span with upstream spans and with the structured log line; trace ID, span ID, and request ID appear in all three.
• Metric labels are bounded: routes are labeled by matched route name or pattern, never by raw request path or per-user identifier.
• Sampling is decided once at ingress and honored consistently downstream so no span is orphaned and no trace is half-recorded.
• Config propagation is a first-class signal: gateway_route_config_version is exported per data-plane node and alerted on when any node lags the control plane.
• Telemetry collection degrades gracefully — a Collector outage or backend backpressure must never block or drop production traffic on the hot path.
• SLOs are defined against user-visible symptoms (latency, availability) with an error budget that gates rollout velocity, not against internal resource gauges.

Overview: Telemetry Data Flow From Data-Plane to Backends

The diagram below traces how a single request produces three correlated signals at the gateway data-plane, how the OpenTelemetry Collector tier receives and processes them, and where each signal lands. Control-plane configuration and its version metric flow alongside — a separate axis from request telemetry but the one that governs whether the data plane is serving the routes you think it is.

Core Concept 1: The Telemetry Data Model — Control-Plane vs Data-Plane Signals

Before wiring exporters, decide what the gateway is actually a source of. Two signal families originate at a gateway, and conflating them is the root of most observability gaps.

Data-plane signals describe request handling on the hot path: which route matched, which upstream was chosen, the response status, per-phase latency, retries, and circuit-breaker state. These are high-volume, per-request, and are the raw material for traces, RED metrics, and access logs. Control-plane signals describe the configuration and health of the gateway itself: the current config generation each node is running, xDS sync status, certificate expiry, plugin load errors, and worker restarts. These are low-volume but high-leverage — a single stale config generation on one node can silently misroute a slice of traffic that no data-plane metric will flag as an error, because from the node’s perspective it is routing correctly against an old table.

A production-grade gateway must emit both. Envoy exposes data-plane telemetry through two distinct sinks: an access-log subsystem that emits one record per request (to file, gRPC, or stdout) and a stats subsystem that aggregates counters, gauges, and histograms into a sink such as Prometheus or the OTLP metrics exporter. The two are configured independently, and a common mistake is wiring access logs to a collector while leaving stats on the default admin-endpoint scrape only.

# Envoy 1.32+ — bootstrap stats sink + admin scrape for control-plane visibility
stats_sinks:
  - name: envoy.stat_sinks.open_telemetry
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.stat_sinks.open_telemetry.v3.SinkConfig
      grpc_service:
        envoy_grpc:
          cluster_name: otel_collector
      report_counters_as_deltas: false
      emit_tags_as_attributes: true
stats_config:
  # Drop per-connection stats that explode cardinality; keep cluster + http rollups
  stats_matcher:
    inclusion_list:
      patterns:
        - prefix: "cluster.upstream_rq"
        - prefix: "http.ingress_http.downstream_rq"
        - prefix: "server.uptime"
        - prefix: "control_plane."

For Kong Gateway the equivalent split is two plugins. The opentelemetry plugin emits traces over OTLP, while the prometheus plugin exposes data-plane metrics on the admin listener. Kong’s control-plane health — in hybrid mode, where data-plane nodes receive config from a control-plane node rather than reading Postgres directly — surfaces through the /status and /clustering endpoints and the data_plane_config_hash, which is Kong’s analogue of a config generation number.

# Kong 3.x — declarative telemetry config: OTel traces + Prometheus metrics
_format_version: "3.0"

plugins:
  - name: opentelemetry
    config:
      traces_endpoint: http://otel-collector.internal:4318/v1/traces
      resource_attributes:
        service.name: edge-gateway
      # Honor inbound sampling decision rather than re-sampling at Kong
      sampling_rate: 1.0
      header_type: w3c

  - name: prometheus
    config:
      status_code_metrics: true
      latency_metrics: true
      bandwidth_metrics: true
      per_consumer: false   # keep consumer off the label set to bound cardinality

Note the deliberate choice of per_consumer: false: enabling per-consumer metrics multiplies every series by the number of API consumers, which is exactly the cardinality trap discussed below. Consumer identity belongs in traces and logs, where it is a searchable attribute, not in metrics, where it is a label dimension.

Core Concept 2: Trace Context Propagation and Span Attributes on Routing Decisions

A trace is only useful if it is unbroken. The gateway is the seam where an external request becomes an internal one, so it is the single most important propagation point in the system — get it wrong and every downstream span is orphaned into its own trace. Distributed tracing and context propagation covers the full mechanics; the essential contract is the W3C Trace Context standard.

The gateway reads the inbound traceparent header, which encodes four fields: version, a 16-byte trace-id, the caller’s 8-byte span-id as parent-id, and trace-flags whose low bit is the sampled flag. The gateway starts a server span as a child of that parent, then — critically — injects a new traceparent into the request it forwards upstream, replacing parent-id with its own span-id so the upstream service becomes a child of the gateway span, not of the original client. The tracestate header, which carries opaque vendor-specific key-value context, is propagated unchanged except that the gateway may prepend its own entry.

# Inbound from client
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  └── trace-id (32 hex) ────────┘ └─ parent span ─┘ └ flags (sampled=1)

# Forwarded upstream — gateway is now the parent
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-a1b2c3d4e5f60718-01
tracestate:  edge-gw=route:payments-v2;up:payments-blue,congo=t61rcWkgMzE

If no traceparent is present, the gateway mints a new trace-id at ingress — this is where a trace begins for any request entering from outside the mesh. The sampling decision must be made once here: a gateway that re-samples independently of the inbound flag will record spans whose children are unsampled (or vice versa), producing traces that are structurally incomplete. Honor the inbound sampled bit, and apply head-based sampling only when minting a fresh trace.

The routing decision itself is the highest-value data on the gateway span. Attach it as span attributes so a trace explains why a request went where it did:

# Envoy 1.32+ — OTel tracing with custom span tags capturing the routing decision
tracing:
  provider:
    name: envoy.tracers.opentelemetry
    typed_config:
      "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
      grpc_service:
        envoy_grpc:
          cluster_name: otel_collector
      service_name: edge-gateway
  # Record the matched route, selected cluster, and retry count as span attributes
  custom_tags:
    - tag: "http.route"
      request_header: { name: ":path" }
    - tag: "gateway.route_name"
      metadata:
        kind: { route: {} }
        metadata_key:
          key: envoy.filters.http.router
          path: [ { key: route_name } ]
    - tag: "gateway.upstream_cluster"
      metadata:
        kind: { host: {} }
        metadata_key:
          key: envoy.lb
          path: [ { key: upstream } ]

Standardize attribute keys on the OpenTelemetry semantic conventions — http.request.method, http.response.status_code, http.route, server.address, url.scheme — plus gateway-specific keys such as gateway.route_name, gateway.upstream_cluster, and gateway.retry_count. Consistent keys are what let one dashboard query span data across a Kong edge tier and an Envoy mesh without per-vendor special-casing.

Core Concept 3: Metrics, SLOs, and Error-Budget Enforcement

Metrics are where observability becomes operations, because they are what you alert and make rollout decisions on. The disciplined baseline is the RED method — Rate, Errors, Duration — computed per route, because a gateway-wide error rate averages a failing route into a sea of healthy ones and hides the incident.

• Rate: requests per second, labeled by route and method.
• Errors: the fraction returning 5xx (and, depending on your contract, specific 4xx such as 429), labeled by route.
• Duration: a latency histogram per route, from which you read p50/p95/p99 and, importantly, the gateway’s own added latency separate from upstream time.

An SLO turns these into a user-facing promise: for example, “99.9% of requests to payments-v2 succeed, and 99% complete in under 300 ms, measured over a rolling 30 days.” The error budget is the inverse — 0.1% of requests may fail — and it is the single most useful operational number a gateway produces, because it converts reliability into a currency you spend on rollout velocity. When the budget is healthy, ship faster; when it is exhausted, freeze risky changes. Encode the SLO as a recording rule so the burn rate is a first-class series:

# Prometheus recording + alert rules — RED-based SLO with multi-window burn rate
groups:
  - name: gateway-slo-payments-v2
    rules:
      # Error ratio per route over a short and long window
      - record: route:request_errors:ratio_rate5m
        expr: |
          sum(rate(gateway_requests_total{route="payments-v2",status=~"5.."}[5m]))
          / sum(rate(gateway_requests_total{route="payments-v2"}[5m]))
      - record: route:request_errors:ratio_rate1h
        expr: |
          sum(rate(gateway_requests_total{route="payments-v2",status=~"5.."}[1h]))
          / sum(rate(gateway_requests_total{route="payments-v2"}[1h]))
      # Fast-burn: 2% budget in 1h. Fire only if BOTH windows breach (kills flapping).
      - alert: PaymentsV2ErrorBudgetFastBurn
        expr: |
          route:request_errors:ratio_rate5m > (14.4 * 0.001)
          and route:request_errors:ratio_rate1h > (14.4 * 0.001)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "payments-v2 burning error budget 14.4x — fast burn"

The 14.4 multiplier is the standard fast-burn factor: at that rate a 30-day budget is consumed in roughly two days, which is the threshold worth paging on. Pairing a short and long window in the and suppresses alerts that fire on a transient 5-minute blip. This burn-rate discipline is what defining SLOs for gateway latency and error budgets develops in full, including how to choose windows and how to separate gateway-added latency from upstream latency so an alert points at the right team.

Alongside RED, export the operational gauges that predict trouble before it becomes an error: gateway_upstream_connection_pool_active per cluster (a leading indicator of pool exhaustion, discussed under circuit breaking and retry budgets), and gateway_route_config_version per node, the config-propagation canary detailed below.

Deployment Topologies for the Telemetry Collection Tier

Where the OpenTelemetry Collector runs determines the latency, blast radius, and cost of your telemetry pipeline. The choice mirrors the gateway topology trade-offs in high-availability topologies, and for anything beyond a single small fleet the answer is a two-tier hybrid.

The hybrid model earns its complexity: the node agent receives telemetry over loopback so the gateway offloads in microseconds and never blocks on a slow backend, attaches resource attributes (node, pod, availability zone) that only the local context knows, and buffers through transient central-tier outages. The gateway tier is where you spend CPU on the expensive operations — tail-based sampling (which requires seeing all spans of a trace before deciding to keep it, so it cannot live on a per-node agent), cardinality reduction, and batching before fan-out. Shipping gateway logs to the OpenTelemetry Collector walks through the receiver and exporter configuration for the log signal specifically.

# OpenTelemetry Collector (gateway tier) — tail sampling + cardinality reduction
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 300 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
  transform/drop-high-cardinality:
    metric_statements:
      - context: datapoint
        statements:
          # Strip raw path label; route name already carries the useful dimension
          - delete_key(attributes, "url.full")
          - delete_key(attributes, "http.request.header.x_request_id")
  batch:
    send_batch_size: 8192
    timeout: 5s

Tail sampling deliberately keeps every error and every slow trace while sampling the successful, fast majority at 5% — the traces you actually need during an incident are retained at 100%, and the volume you pay to store is dominated by the boring ones you can afford to drop.

Observability of Operations: Config Propagation and Incident Response

The signals above tell you about requests. Operating a gateway also requires signals about the gateway, and the most important is config propagation. A control plane pushes routes and policy to data-plane nodes, but propagation is eventually consistent: during a push, some nodes run the new table and some the old. A node that stops applying updates entirely — a wedged xDS stream, a crashed config-reload worker, a network partition to the control plane — becomes a config-propagation black hole. It serves confidently against a stale routing table, so its data-plane metrics look healthy while it quietly sends traffic to a decommissioned upstream or misses a new route.

The defense is a per-node config generation gauge. Envoy exposes control_plane.connected_state and per-xDS-type version_info; Kong exposes data_plane_config_hash. Normalize these into gateway_route_config_version{node} and alert on the spread across the fleet — if any node lags the control plane’s current generation by more than one for longer than the expected propagation window (typically under one second for Envoy xDS, up to a second or two for Kong hybrid mode), it is a routing incident, not noise.

# Prometheus alert — detect a config-propagation black hole across the fleet
groups:
  - name: gateway-config-propagation
    rules:
      - alert: GatewayConfigDrift
        expr: |
          (max(gateway_route_config_version) - gateway_route_config_version) > 1
        for: 60s
        labels: { severity: page }
        annotations:
          summary: "Node  is >1 config generation behind the fleet"
          description: "Suspected config-propagation black hole; node may be serving stale routes."

For incident response, wire the three signals into one workflow. An SLO burn-rate alert fires (metrics); the operator pivots on the affected route name to the exemplar traces attached to the breaching latency histogram (traces); each trace’s trace-id links to the structured log lines for those exact requests (logs), revealing the upstream, consumer, and retry count. This metrics-to-traces-to-logs pivot only works if all three carry the same identifiers — which is why identifier consistency is a design invariant, not a nice-to-have.

Rollout and Rollback Discipline

Observability exists to make change safe. A gateway change — a new route, a reordered plugin chain, a shifted upstream weight — should be gated by the same error budget the SLO defines. The rollout pattern is progressive: shift a small slice of traffic to the new configuration, watch the RED metrics and burn rate for that slice against the stable baseline, and promote only if the new slice’s error and latency signals stay within budget. If they degrade, roll back automatically before the budget is spent.

NGINX Plus fits the same model through its own primitives: the status_zone directive groups metrics per virtual server or location for RED-style breakdowns, the R32+ native OpenTelemetry module (otel_trace, otel_span_attr) emits spans with W3C propagation, and the shared-memory zone stats are scraped by the Prometheus exporter. The mechanics differ; the invariants — bounded labels, per-route signals, trace-correlated logs, a config-version signal — do not.

# NGINX Plus R32+ — native OTel tracing with W3C propagation + per-location metrics
otel_exporter {
    endpoint otel-collector.internal:4317;
}
otel_service_name edge-gateway;
otel_trace on;
otel_trace_context propagate;          # read + inject W3C traceparent

server {
    listen 443 ssl;
    status_zone edge_frontend;         # RED metrics grouped per zone

    location /v2/payments {
        status_zone payments_v2;
        otel_span_name payments-v2;
        otel_span_attr gateway.route_name payments-v2;
        proxy_pass http://payments_pool;
    }
}

Failure Modes and Resilience Patterns

Cardinality explosion. The most common way to take down an observability stack is a metric label with unbounded values — raw request paths carrying IDs, full URLs, per-consumer identifiers, or unbounded error strings. Each unique combination is a new time series, and a gateway fronting thousands of routes can mint millions of series that OOM the metrics backend. Label by matched route name or route pattern, keep consumer and request IDs out of the metric label set (they belong in traces and logs), and drop stragglers at the Collector with a transform or filter processor. Alert on active series count as a first-class SLI of the telemetry system itself.

Log volume blowout. Full-body or verbose access logging at the perimeter generates terabytes per day and can cost more than the gateway compute it observes. Emit one structured line per request with a bounded field set, sample successful high-volume routes at the log layer (while keeping 100% of errors), and never log request or response bodies at the gateway by default. Structured access logging covers field selection and PII redaction in detail.

Sampling bias. Head-based probabilistic sampling keeps a fixed fraction of all traces, which means rare error traces are dropped at the same rate as common success traces — exactly backwards from what you need in an incident. Tail-based sampling in the Collector gateway tier fixes this by deciding after seeing the whole trace: keep all errors and slow traces, sample the fast-and-successful majority. The cost is that the Collector must buffer spans for decision_wait, which is why tail sampling lives in the scaled central tier, not on the node agent.

Config-propagation black holes. As above, a node that silently stops applying config serves stale routes with healthy-looking data-plane metrics. The gateway_route_config_version fleet-spread alert is the only reliable detector; data-plane error rates will not catch it because the node is not erroring, it is confidently wrong. This is the observability analogue of the routing-table drift problem covered in advanced routing and API versioning.

Telemetry backpressure on the hot path. If the gateway exports telemetry synchronously and a backend or Collector slows down, the export can block request handling and turn an observability outage into a traffic outage. Always export asynchronously with a bounded queue that drops telemetry (never requests) under backpressure, and place a node-local Collector agent as the offload target so the gateway’s export is a loopback write that returns immediately.

Trace gaps at the seam. A gateway that fails to propagate traceparent — or that re-samples inconsistently — orphans every downstream span. The zero-trust security boundary at the edge must be configured to sanitize untrusted inbound trace headers from public clients (which could inject a forged trace-id) while still propagating trust-boundary-internal context — a nuance that propagating W3C trace context through Kong and Envoy addresses head-on.

For the capacity dimension of all of this — how much telemetry a fleet can emit before the pipeline, not the gateway, becomes the bottleneck — see scaling limits and capacity planning.

Implementation Blueprint

Technical Validation Checklist

• Every request emits a trace span, RED metrics, and one structured log line on the success path — not only on failure.
• Inbound traceparent is honored and its sampled flag is respected; a fresh trace-id is minted only when none is present.
• The gateway injects an updated traceparent (its own span-id as parent-id) into upstream requests and passes tracestate through.
• Untrusted inbound trace headers from public clients are sanitized at the edge before internal propagation.
• Span attributes use OpenTelemetry semantic conventions plus gateway.route_name and gateway.upstream_cluster for the routing decision.
• RED metrics are computed per route; gateway-added latency is measured separately from upstream latency.
• Metric labels are bounded — routes labeled by matched name/pattern; consumer and request IDs are excluded from metric labels.
• Active-time-series count is monitored and alerted on as an SLI of the telemetry pipeline itself.
• SLOs are defined per critical route with a documented error budget and multi-window burn-rate alerts.
• gateway_route_config_version (or equivalent hash) is exported per node and alerted on fleet-spread drift.
• Telemetry is exported asynchronously with a bounded queue that drops telemetry — never requests — under backpressure.
• An OpenTelemetry Collector agent runs node-local as the offload target; a scaled gateway tier performs tail sampling and cardinality reduction.
• Tail sampling keeps 100% of error and slow traces while sampling the fast-successful majority.
• Access logs carry trace_id, route, upstream, status, and latency so metrics-to-traces-to-logs pivots resolve to the same requests.
• Config and rollout changes are gated by error-budget burn rate with an automated rollback trigger.

FAQ

What are the three telemetry pillars an API gateway must emit?

Distributed traces, metrics, and structured logs. Traces carry a propagated traceparent so a single request can be correlated across the gateway and every downstream service. Metrics expose the RED signals — request rate, error rate, and duration — per route as the basis for SLOs and error budgets. Structured logs record one machine-parseable JSON line per request with route name, upstream, status, latency, trace ID, and consumer identity. All three must share the same trace and request identifiers so an operator can pivot between them during an incident.

How does W3C Trace Context propagate through an API gateway?

The gateway reads the inbound traceparent header, which carries the trace-id, the caller’s span-id as parent-id, and trace-flags including the sampled bit. It starts its own server span as a child, then injects an updated traceparent — with its own span-id as the new parent-id — into the request forwarded upstream. The tracestate header is passed through unmodified except for the gateway’s own vendor entry. If no traceparent is present, the gateway mints a new trace-id at the edge. Sampling decisions must be made once at ingress and honored consistently so spans are not orphaned.

What causes metric cardinality explosion at the gateway and how do I prevent it?

Cardinality explosion happens when a metric label takes unbounded values — raw request paths with IDs, full URLs, per-user identifiers, or unbounded status strings. Each unique label combination becomes a separate time series, and a gateway fronting thousands of routes can generate millions of series that exhaust the metrics backend. Prevent it by labeling with the matched route name or route pattern rather than the raw path, bucketing status into classes where high resolution is not needed, dropping high-cardinality labels at the OpenTelemetry Collector with a filter or transform processor, and alerting on active series count before the backend falls over.

Why is a config version metric critical for gateway operations?

Control-plane configuration propagates to data-plane nodes eventually, not instantly. During a push some nodes run new routes while others run old ones, and a node that silently stops applying updates becomes a config-propagation black hole that serves stale routing decisions. A gateway_route_config_version gauge exported per node lets you detect drift: if any node lags the control plane by more than one generation for longer than the expected propagation window, it is a routing incident. Alerting on the spread of this metric across the fleet catches partial-push failures before users see 503s from deleted upstreams.

Should the OpenTelemetry Collector run as an agent, a gateway tier, or both?

Run both in most production deployments. A Collector deployed as a per-node agent (DaemonSet or sidecar) receives telemetry over localhost with minimal latency and adds resource attributes such as node and pod identity. It forwards to a horizontally scaled Collector gateway tier that performs tail-based sampling, cardinality reduction, batching, and fan-out to trace, metric, and log backends. This two-tier topology keeps the hot path cheap, centralizes expensive processing where it can be scaled independently, and gives you one place to enforce sampling and retention policy.

Related

• Distributed Tracing & Context Propagation — W3C Trace Context mechanics, span attributes, and sampling across the gateway seam.
• Gateway Metrics & SLOs — RED metrics, error budgets, and multi-window burn-rate alerting for latency and availability.
• Structured Access Logging — JSON log field selection, trace correlation, PII redaction, and shipping to the Collector.
• Circuit Breaking & Retry Budgets — the resilience signals (pool exhaustion, retry amplification) that gateway metrics must surface.
• Security Boundaries & Zero-Trust — sanitizing untrusted inbound trace headers while propagating internal context.

← Home