Distributed Tracing & Context Propagation

A distributed trace is only as good as its weakest hop, and the gateway is the first hop every external request crosses. When the gateway silently drops or rewrites trace context, every downstream span becomes an orphan and the trace that should have told you which upstream added 400 ms instead shows a disconnected forest of single-service fragments. This page covers how a gateway becomes a well-behaved participant in a distributed trace: how it reads and forwards W3C traceparent and tracestate, how it emits a span for its own routing decision, which attributes that span must carry, how head-based and tail-based sampling interact, and how context survives crossing a protocol boundary. It sits under Gateway Observability & Operations, which frames tracing alongside metrics and structured logging as the three pillars of gateway telemetry.

The gateway occupies a privileged position: it is the trust boundary where an untrusted inbound traceparent must either be adopted or replaced, and it is the only component that knows both the client-facing route and the internal upstream that served it. Get propagation right here and the rest of the mesh inherits a coherent trace; get it wrong and no amount of downstream instrumentation can reconstruct the parent relationship.

Design invariants for a gateway that participates in traces:

• The inbound traceparent is read before any header transformation runs — never after a plugin that could strip it.
• The gateway emits exactly one server span per request, parented to the inbound context, and one client span per upstream attempt.
• A single propagation format is authoritative across the fleet — W3C traceparent/tracestate — with B3 accepted only through an explicit compatibility shim, never silently.
• The sampling decision is made once and encoded in the traceparent sampled flag so every downstream honours the same decision.
• Trace context is regenerated, not blindly trusted, at the edge: an external client must not be able to inject a trace-id that collides with internal traces or poison tracestate.
• Every routing decision is a span attribute — matched route, upstream cluster, retry count — not a free-text log line.

The mental model: trace context is a baton, the gateway is the first runner

Distributed tracing works because every service passes a small, fixed-size context to the next service in the call chain. Under the W3C Trace Context standard that context lives in two HTTP headers. traceparent is a single line with four hyphen-separated fields:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^version ^trace-id (16 bytes)         ^parent-id (8B)  ^flags

The version is 00. The trace-id is the 128-bit identifier shared by every span in the trace. The parent-id (also called span-id) is the ID of the immediate caller’s span — the gateway overwrites this with its own span ID before forwarding, which is exactly how the parent/child relationship is built. The flags byte carries the sampled bit: 01 means “recorded, propagate me”; 00 means “not sampled”. tracestate is a companion header holding vendor-specific key/value pairs (for example tracestate: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE) that must be forwarded unchanged even by a gateway that does not understand any of the keys.

The gateway’s job is mechanical but unforgiving. On ingress it parses the inbound traceparent; if present and valid it continues that trace, if absent or malformed it starts a new one. It creates a server span whose parent is the inbound parent-id. It runs the routing decision and records the outcome as span attributes. For each upstream attempt it opens a child client span, writes a fresh traceparent whose trace-id is unchanged but whose parent-id is the client span’s ID, and forwards that to the backend. When the response returns it closes the spans and hands them to an exporter. The single most important rule: the parse-inbound step must happen before any middleware that mutates headers, or the baton is dropped before the first runner even grips it.

Trace context flow through the gateway

The diagram traces a single trace-id from the client, through the gateway’s server and client spans, out to two upstream services, and finally into the collector where the spans are reassembled into one trace.

Primary concept: emitting the routing span across gateways

The value of a gateway span is not that it exists but that it is annotated. A span that only records duration tells you the gateway was slow; a span that records the matched route, the chosen upstream cluster, and the retry count tells you why. Below are runnable tracer configurations for the two gateways most teams run at the edge and in the mesh, plus NGINX Plus where it sits in front.

Kong 3.x enables tracing through the bundled opentelemetry plugin. It reads the inbound traceparent, produces a Kong request span, and exports over OTLP/HTTP. Two settings matter most: header_type fixes the propagation format, and sampling_rate sets the head-based decision.

# Kong 3.x — declarative config: OpenTelemetry tracing on a routed service
_format_version: "3.0"

services:
  - name: orders-api
    url: http://orders.internal:8080
    routes:
      - name: orders-v2
        paths: ["/v2/orders"]
        strip_path: true

plugins:
  # Global tracer: read + emit W3C trace context, export OTLP/HTTP to the collector
  - name: opentelemetry
    config:
      endpoint: "http://otel-collector.internal:4318/v1/traces"
      resource_attributes:
        service.name: "edge-gateway"
      header_type: w3c          # authoritative format; do not leave as "preserve"
      sampling_rate: 1.0        # head-based: sample everything, let the collector tail-sample
      queue:
        max_batch_size: 200
        max_coalescing_delay: 1 # seconds; bound export latency

Kong also needs request tracing switched on at the node level so spans are generated for the proxy path — set tracing_instrumentations = request,router,balancer and tracing_sampling_rate = 1.0 in kong.conf. The router and balancer instrumentations are what surface the matched route and the balancer’s upstream selection as child spans; without them you get a single opaque gateway span with no routing detail.

Envoy 1.32+ configures tracing on the http_connection_manager with the native OpenTelemetry tracer. Envoy is stricter and more powerful here: it distinguishes the sampling decision, the propagation, and the span attributes, and it can add custom tags from request headers or metadata.

# Envoy 1.32+ — HTTP connection manager with the OpenTelemetry tracer
http_connection_manager:
  # Emit an ingress span and honour the inbound sampled flag
  tracing:
    # 100% here = head decision delegated downstream to the collector's tail sampler
    random_sampling:
      value: 100
    provider:
      name: envoy.tracers.opentelemetry
      typed_config:
        "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
        service_name: "mesh-gateway"
        grpc_service:
          envoy_grpc:
            cluster_name: otel_collector
          timeout: 1s
    # Attributes that make the routing decision queryable
    custom_tags:
      - tag: "route.matched"
        request_header:
          name: ":path"
      - tag: "upstream.cluster"
        metadata:
          kind: { host: {} }
          metadata_key:
            key: "envoy.lb"
            path: [ { key: "canary" } ]
  # W3C is Envoy's default propagation; keep it explicit and consistent fleet-wide
  request_headers_to_add:
    - header: { key: "x-gateway-trace", value: "%REQ(TRACEPARENT)%" }
      keep_empty_value: false

Envoy automatically annotates the span with http.status_code, response_flags, the matched route_name when routes are named, and — critically for resilience debugging — the retry count via the x-envoy-attempt-count mechanism when include_attempt_count_in_response is set. Naming every route (name: on each route entry) is what turns route.matched from a raw path into a stable, low-cardinality attribute you can group by.

NGINX Plus (R32+) carries tracing through the ngx_otel_module, loaded dynamically. It is the right choice when NGINX Plus is the TLS-terminating edge in front of a Kong or Envoy tier and you need the very first span to originate at the true edge.

# NGINX Plus R32+ — OpenTelemetry module: propagate W3C, export OTLP/gRPC
load_module modules/ngx_otel_module.so;

http {
  otel_exporter {
    endpoint otel-collector.internal:4317;   # OTLP/gRPC
    interval 2s;
    batch_size 512;
  }
  otel_service_name edge-nginx;
  otel_trace on;
  otel_trace_context propagate;               # read inbound + inject outbound traceparent

  server {
    listen 443 ssl;
    location /v2/ {
      otel_span_name orders_edge;
      otel_span_attr http.route "/v2/orders";
      proxy_set_header traceparent $otel_trace_id; # ensure context reaches the upstream tier
      proxy_pass http://gateway_tier;
    }
  }
}

The otel_trace_context propagate directive is the load-bearing setting: propagate reads the inbound traceparent and forwards it; the alternative inject would ignore any inbound context and start a fresh trace, silently orphaning everything the client already started. This one keyword is the difference between a joined trace and a broken one at the NGINX edge.

Secondary concept: sampling strategy and its biases

Sampling is where tracing economics and tracing usefulness collide. Record every span at full fidelity and you drown the backend and the bill; record too few and the one trace you needed during an incident was thrown away. There are two decision points.

Head-based sampling makes the keep/drop decision at the first hop — the gateway — and encodes it in the traceparent sampled flag. It is cheap and predictable: overhead is bounded because unsampled requests never allocate span buffers, and the decision is consistent because every downstream reads the same flag. Its weakness is that the decision is made before the request outcome is known, so a purely head-sampled system at 5 percent will, on average, discard 95 percent of your errors too.

Tail-based sampling defers the decision to the OpenTelemetry Collector, which buffers all spans of a trace until it completes, then applies policies — keep everything with an error, keep everything slower than the P99, down-sample the boring successes. It captures exactly the traces you want but requires the collector to hold complete traces in memory and to see all spans of a trace, which constrains how you shard collectors.

The productive combination is head-based at a high rate at the gateway (often 100 percent, delegating the real decision downstream) plus a tail sampler in the collector:

# OpenTelemetry Collector — tail_sampling: keep all errors + slow traces, thin the rest
processors:
  tail_sampling:
    decision_wait: 10s          # hold spans this long to assemble the full trace
    num_traces: 100000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Sampling bias is the trap. If the gateway head-samples at 5 percent and the collector also only ever sees that 5 percent, tail-based policies can only rescue errors that survived the head cut — you cannot tail-sample a trace whose spans were never recorded. That is why the head rate at the gateway must be at or above the tail sampler’s ceiling: set the gateway to 100 percent (or AlwaysOn) and let the collector be the only thinning stage, or you introduce a silent bias that hides low-frequency errors precisely when they matter.

Comparative implementation

Because the authentication and token-validation layer usually runs as a header-mutating plugin, its ordering relative to the tracer is a correctness concern, not just a performance one: a tracer that reads traceparent after an aggressive header allow-list has already run will see nothing to continue.

Operational gotchas

Broken trace from header stripping. A request-transformer that removes headers, or an ingress with a strict traceparent-excluding allow-list, deletes the context before the tracer reads it. Signal: every gateway span is a root span with no parent; downstream traces are single-service fragments. Remediation: order the tracer plugin before any transformer, and explicitly add traceparent and tracestate to any header allow-list.

B3 vs W3C mismatch. A legacy service emits Zipkin B3 (X-B3-TraceId, X-B3-SpanId) while the gateway only reads traceparent. The gateway sees no W3C context and starts a fresh trace, severing the client’s B3 trace. Signal: two disconnected traces per request, one B3 and one W3C. Remediation: standardise on W3C and, at any boundary you cannot convert, run a multi-format extractor (Kong header_type: preserve with both, or Envoy’s B3-to-W3C shim) — but adopt exactly one authoritative format everywhere else.

Sampling bias hiding errors. Head sampling at a low rate discards most error traces before the tail sampler can see them. Signal: error dashboards show incidents that have almost no example traces. Remediation: raise the gateway head rate to at or above the collector’s tail ceiling; make the collector the only thinning stage.

Context lost across a protocol boundary. When the gateway translates protocols — REST to gRPC, HTTP/1.1 to HTTP/2, or into a message queue — the traceparent HTTP header has no automatic home in the target protocol. For gRPC the context must be copied into a traceparent metadata entry; for messaging it must be written into a message property. Signal: the trace ends abruptly at the translation hop. Remediation: enable the tracer’s protocol-aware propagator (Envoy handles gRPC metadata natively; for queues, inject context into the message envelope explicitly).

Cardinality blow-up in span attributes. Tagging spans with the raw :path (which contains IDs) instead of the matched route template explodes attribute cardinality and slows the backend. Signal: trace-store ingestion cost climbs and queries slow after a release that added path tags. Remediation: tag with the route name or template (/v2/orders/{id}), never the concrete path.

Clock skew reordering spans. Gateway and upstream clocks drift, so a child span appears to start before its parent. Signal: negative or impossible durations in the waterfall. Remediation: run NTP/chrony on every node and rely on span parent/child links rather than wall-clock ordering for causality.

Production Configuration Checklist

• The tracer reads the inbound traceparent before any header-mutating plugin runs.
• traceparent and tracestate are on every header allow-list at the edge and between tiers.
• One authoritative propagation format (W3C) is set fleet-wide; B3 is accepted only through an explicit shim.
• Every route is named so the matched-route span attribute is low-cardinality.
• Span attributes include matched route, upstream cluster, retry count, and route-config version.
• Head-based sampling at the gateway is at or above the collector’s tail-sampling ceiling.
• Tail-based sampling in the collector keeps all error and high-latency traces.
• Spans export to a local OpenTelemetry Collector, not directly to the backend, with a bounded batch/queue.
• gRPC and messaging hops propagate context via metadata/message properties, not just HTTP headers.
• Span attributes use route templates, never concrete paths with embedded IDs.
• NTP/chrony runs on all gateway and upstream nodes to bound clock skew.
• An end-to-end test asserts a single trace-id is continuous from client through gateway to a downstream service.

FAQ

Should a gateway use head-based or tail-based sampling?

Use head-based sampling at the gateway when you need bounded, predictable overhead and simple config — the decision is made once at ingress and encoded in the traceparent sampled flag so every downstream honours it. Use tail-based sampling in the OpenTelemetry Collector when you must keep every error and slow trace regardless of the head decision. The common production pattern is head-based sampling at a high rate (or 100 percent) at the gateway, then a tail sampler in the collector that keeps all error and high-latency traces and down-samples the successful ones.

Why does my trace break at the gateway?

The most common cause is header stripping: a request-transformer or a restrictive header allow-list removes the incoming traceparent before the tracer reads it, so the gateway starts a brand-new trace with no parent. Other causes are a B3-versus-W3C format mismatch (the client sends b3 but the gateway only reads traceparent), a sampling flag set to 00 that makes downstreams drop the span, and clock skew that makes spans appear out of order. Confirm the gateway both reads the inbound traceparent and forwards a traceparent whose trace-id matches to the upstream. The step-by-step fix is covered in propagating W3C trace context through Kong and Envoy.

What span attributes should a gateway record on a routing decision?

Record the matched route name or ID, the selected upstream cluster or service, the retry count, the HTTP status and route-config version, plus standard semantic-convention attributes like http.request.method, url.path, and server.address. These let you slice trace data by route and upstream when diagnosing a latency regression, and the retry count exposes retry amplification that raw latency numbers hide.

Do I need to run an OpenTelemetry Collector, or can the gateway export directly to a backend?

Both Kong and Envoy can export OTLP directly to a backend, but a Collector between the gateway and the backend is strongly recommended in production. It decouples the gateway from backend availability with a queue and retry buffer, centralises tail-based sampling and attribute redaction, and lets you switch tracing backends without touching gateway config. Point every gateway at a local Collector agent and let the Collector fan out to the backend.

Related

• Propagating W3C Trace Context Through Kong and Envoy — the step-by-step how-to for keeping a trace unbroken across an edge gateway and a sidecar mesh.
• Gateway Observability & Operations — how tracing sits alongside metrics and structured logging in the gateway telemetry stack.
• Authentication Proxying & Token Validation — why header-mutating auth plugins must be ordered after the tracer reads trace context.
• Scaling Limits & Capacity Planning — how sampling rate and span export volume factor into gateway throughput ceilings.
• Protocol Translation Patterns — where trace context must jump protocol boundaries and how propagation survives the translation.

← Gateway Observability & Operations