Propagating W3C Trace Context Through Kong and Envoy

The most common way to lose a distributed trace is to run two correctly-instrumented gateways that simply do not agree on how to hand the context between them. A Kong 3.x gateway terminates TLS at the edge and forwards to services fronted by Envoy 1.32+ sidecars; both emit spans; yet the trace viewer shows a Kong root span and, disconnected from it, a separate Envoy-rooted trace for the same request. The operational stakes are concrete: during an incident you cannot tell whether the 800 ms belonged to Kong, the network, or the upstream, because the waterfall that would show it was never assembled. This how-to walks through configuring both gateways so a single traceparent survives the edge-to-mesh handoff, then how to verify continuity and debug it when it breaks.

Prerequisite concepts

This guide assumes you understand the W3C traceparent and tracestate mechanics and the sampling model — specifically that the trace-id is constant across hops while the parent-id is rewritten by each service. It also assumes familiarity with the broader gateway observability and operations practice that pairs tracing with metrics and structured logs. If your Kong tier also runs a header-mutating auth step, review authentication proxying and token validation first, because plugin ordering around the tracer is what determines whether context survives.

The topology and where it breaks

traceparent handoff from Kong edge to Envoy sidecar A client sends a traceparent to a Kong edge gateway. Kong reads it, emits a span, and forwards a traceparent with the same trace-id but a new parent-id to an Envoy sidecar. The sidecar continues the trace and forwards it to the application. The trace-id stays constant while the parent-id changes at each hop; the break point is any hop that fails to read or forward the header. Client traceparent Kong 3.x edge opentelemetry header_type: w3c Envoy sidecar OTel tracer W3C propagation Application continues trace trace-id constant on every arrow; parent-id rewritten at each box; a break = a hop that drops or ignores traceparent

There are only three places this handoff fails: Kong does not forward a W3C traceparent, the Envoy sidecar does not read or emit spans, or a header transform between them strips the context. The steps below close all three.

Step 1 — Preserve the header at the Kong edge

Enabling the tracer is necessary but not sufficient; Kong must also be told to emit W3C and to keep traceparent through any transform. Set the node-level instrumentation in kong.conf, then the plugin.

# Kong 3.x — kong.conf: generate proxy/router/balancer spans, sample everything
tracing_instrumentations = request,router,balancer
tracing_sampling_rate = 1.0
# Kong 3.x — declarative: emit W3C traceparent to the upstream Envoy tier
_format_version: "3.0"

services:
  - name: checkout
    url: http://checkout.mesh.internal:8080   # fronted by an Envoy sidecar
    routes:
      - name: checkout-v1
        paths: ["/v1/checkout"]

plugins:
  - name: opentelemetry
    config:
      endpoint: "http://otel-collector.internal:4318/v1/traces"
      header_type: w3c        # forward a W3C traceparent (NOT "preserve"/"b3")
      sampling_rate: 1.0
      resource_attributes:
        service.name: "kong-edge"

  # If ANY transformer runs, it must NOT drop trace headers.
  - name: request-transformer
    config:
      # do this — never list traceparent/tracestate under `remove.headers`
      add:
        headers: ["x-edge:kong"]

The header_type: w3c setting is the load-bearing one: with the default preserve, Kong echoes whatever format arrived, so a client that sent nothing gets nothing forwarded, and a client that sent B3 keeps B3 — either way the downstream Envoy expecting W3C sees no traceparent. Setting w3c guarantees Kong always injects a W3C traceparent on the outbound request.

Step 2 — Enable the tracer on the Envoy sidecar

The sidecar must read the traceparent Kong sent and emit its own child span. Envoy’s default propagation is already W3C traceparent, so the work is enabling the tracer and pointing it at the same collector.

# Envoy 1.32+ — sidecar http_connection_manager: OTel tracer, honour inbound sampled flag
http_connection_manager:
  tracing:
    # No random_sampling override => Envoy respects the inbound traceparent sampled bit
    provider:
      name: envoy.tracers.opentelemetry
      typed_config:
        "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
        service_name: "checkout-sidecar"
        grpc_service:
          envoy_grpc:
            cluster_name: otel_collector
          timeout: 1s
  route_config:
    virtual_hosts:
      - name: checkout_local
        domains: ["*"]
        routes:
          - match: { prefix: "/v1/checkout" }
            name: checkout_route      # named route => stable span attribute
            route: { cluster: checkout_app }

Omitting random_sampling here is deliberate: when it is unset, Envoy honours the sampled flag in the traceparent Kong already decided, so the sidecar does not make a second, conflicting sampling decision. If you were to set random_sampling.value: 5 on the sidecar, Envoy could drop a span the edge chose to keep, tearing a hole in the middle of the trace.

Step 3 — Verify traceparent continuity

Send a request with a traceparent you control and follow the trace-id (the middle 32 hex characters) through every hop. It must be byte-for-byte identical everywhere; only the parent-id should change.

# Inject a known trace-id and watch it survive both hops
TP="00-11112222333344445555666677778888-aaaabbbbccccdddd-01"

curl -sv https://edge.example.com/v1/checkout \
  -H "traceparent: $TP" 2>&1 | grep -i traceparent

# On the upstream app, log the received header and confirm the trace-id matches:
#   received traceparent: 00-11112222333344445555666677778888-<NEW parent-id>-01
# trace-id (11112222...8888) is IDENTICAL; parent-id differs. That is a healthy trace.

Then confirm the same trace-id shows a single connected waterfall — client, kong-edge, checkout-sidecar, application — in your trace backend. If all four spans share the trace-id and nest correctly, propagation is unbroken.

Step 4 — Debug a broken trace

Work the hops in order; the first hop where the trace-id changes or a span goes missing is the culprit.

Symptom Likely cause Fix
Kong span is a root with no parent inbound traceparent stripped before tracer, or client sent none move tracer ahead of transforms; add traceparent to allow-list
trace-id changes after Kong header_type is preserve/b3, not w3c set header_type: w3c on the opentelemetry plugin
Gap between Kong span and app span Envoy tracer not enabled on the sidecar add the envoy.tracers.opentelemetry provider to the sidecar
Two traces per request (B3 + W3C) Kong emits W3C, sidecar expects B3 standardise both on W3C traceparent
Sidecar span present but marked not-sampled sidecar random_sampling overrides the inbound sampled flag remove random_sampling so Envoy honours traceparent

The fastest single diagnostic is Step 3’s trace-id grep: it localises the break to one hop in a couple of commands, which beats reading four tracer configs hoping to spot the discrepancy.

Decision summary

Concern Kong 3.x edge Envoy 1.32+ sidecar
Read inbound context automatic when opentelemetry enabled automatic; W3C is the default propagator
Emit outbound traceparent requires header_type: w3c automatic; keep W3C
Sampling decision make it here (sampling_rate/head) inherit it — do not override random_sampling
Proxy span detail needs tracing_instrumentations = request,router,balancer name every route for a stable attribute

Gotchas & failure signals

  • header_type: preserve on Kong silently forwards whatever arrived, so an un-instrumented client produces no downstream context. Signal: Envoy span is a root. Force w3c.
  • A request-transformer listing traceparent under remove.headers deletes the baton mid-chain. Signal: Kong span exists, Envoy span is a new root. Never strip trace headers.
  • Double sampling decisions — a head decision at Kong and a second random_sampling at the sidecar — produce spans that are recorded on one hop and dropped on the next. Signal: waterfalls that are missing their middle span. Let the edge decide, the sidecar inherit.
  • B3 leaking in from a legacy caller while the fleet is W3C severs the client’s original trace at Kong. Signal: a short B3 trace and a separate W3C trace for one request. Standardise on W3C and convert only at the legacy boundary.

Validation

  • tracing_instrumentations = request,router,balancer is set in kong.conf.
  • The Kong opentelemetry plugin has header_type: w3c.
  • No transform lists traceparent or tracestate under remove.headers.
  • The Envoy sidecar http_connection_manager has the OpenTelemetry tracer provider enabled.
  • The sidecar does not override random_sampling, so it honours the inbound sampled flag.
  • A curl with a known traceparent shows an unchanged trace-id at every hop.
  • The trace backend shows one connected waterfall: client, kong-edge, checkout-sidecar, application.

FAQ

How do I verify the traceparent is continuous across Kong and Envoy?

Send a request with a known traceparent and grep the trace-id (the middle 32 hex characters) out of each hop. It must be identical at the client, in Kong’s emitted span, in the traceparent Envoy receives, and in the upstream’s span. The parent-id (the third field) must change at each hop because every service overwrites it with its own span ID, but the trace-id must never change. If the trace-id differs after a hop, that hop started a new trace and the context was dropped there.

Does Kong need extra config beyond the opentelemetry plugin to forward traceparent?

Yes. Enabling the opentelemetry plugin makes Kong export spans, but you must also set header_type to w3c so Kong emits a W3C traceparent to the upstream, and you must set tracing_instrumentations in kong.conf to at least request,router,balancer so proxy spans are generated. If a request-transformer plugin runs before the tracer and strips headers, add traceparent and tracestate back explicitly, because Kong will otherwise forward a request with no context.

Why does the trace break specifically at the Envoy sidecar?

The usual cause is that the Envoy tracer is not enabled on the sidecar’s http_connection_manager, so Envoy proxies the request but neither reads nor emits a span, leaving a gap between Kong’s span and the application’s span. The second cause is a propagation mismatch: Kong emits W3C traceparent but the sidecar is configured to expect B3. Enable the OpenTelemetry tracer on the sidecar and confirm both sides use W3C.

Parent: Distributed Tracing & Context Propagation