Structured Access Logging

An access log is the one telemetry signal a gateway emits for every single request, whether or not anything went wrong. Metrics tell you a route’s error rate climbed; traces show you one request’s path through the mesh; but the access log is the exhaustive, per-request record you reach for when someone asks “which consumer, on which route, got that burst of 503s at 14:02, and what was the upstream doing?” If those lines are free-text strings, answering that question means writing regex under incident pressure. If they are structured JSON with a fixed, indexed field set, it is a single query. This page is part of gateway observability and operations, and it covers what fields every line must carry, how to correlate a log with its trace, how to keep volume and cost bounded through log levels and sampling, how to redact secrets before they are written, and how to ship the result to a store. Runnable configuration is given for Kong 3.x, Envoy 1.32+, and NGINX Plus R32+.

Design invariants every structured logging setup must satisfy:

• Every line is a single valid JSON object with a stable, documented schema — new fields are additive, never renamed, so downstream queries do not break.
• The trace correlation identifier (trace_id extracted from traceparent) is a top-level indexed field on every line, not buried inside a message string.
• Credentials never appear: no Authorization header, no Cookie, no raw bodies, no query-string secrets. Consumer identity is logged as a resolved ID, not the token.
• Latency is logged as two distinct fields — time spent at the upstream versus total time in the gateway — so you can attribute slowness to the backend or to the proxy.
• The success path is sampleable; the error path and slow path are always logged in full.
• Logs are shipped through a collector that batches, redacts as a second layer, and can be re-routed to a different store without touching gateway config.

Overview: The Log Pipeline

Structured logging is a pipeline, not a file. The gateway emits JSON on the hot path; a collector receives, redacts, batches, and routes; a store indexes it for query. Each stage is independently scalable and independently a failure point. The diagram traces one line from the gateway’s log phase through to the query surface, and shows where trace correlation and redaction happen.

Required Fields: The Log Schema Contract

A structured log line is only as useful as the fields it commits to. Treat the schema as a contract: name every field once, keep names stable forever, and make additions purely additive. The following field set is the operational minimum for a gateway access log. Anything less and you will be parsing message strings during an incident.

The two-latency rule is worth dwelling on. A single latency number cannot tell you whether a slow request was the backend’s fault or the gateway’s. Logging upstream_latency_ms and total_latency_ms separately means the gap between them is queueing, plugin execution, and connection-pool wait — a leading signal of gateway saturation that no upstream metric will show you. This is the same distinction that gateway metrics and SLOs formalize into latency histograms, and the log gives you the per-request granularity the histogram aggregates away.

Primary Concept: Emitting JSON on the Hot Path

Each gateway writes its access log in a different mechanism, but the target is identical: one JSON object per line, containing the schema above, redacted, on stdout or over the network. Below are production-ready configurations for three gateways.

Kong 3.x uses either the file-log plugin (write JSON to a path or /dev/stdout) or the http-log plugin (POST batches to an HTTP endpoint). Kong’s default log serializer already emits structured JSON; the key is enriching it with custom fields — the trace ID, the resolved consumer, and rate-limit state — via custom_fields_by_lua, which runs a small Lua expression in the log phase.

# Kong 3.x declarative — structured access log with custom correlation fields
_format_version: "3.0"

plugins:
  # http-log ships batched JSON to a collector's HTTP receiver
  - name: http-log
    config:
      http_endpoint: http://otel-collector.internal:4318/v1/logs-raw
      method: POST
      content_type: application/json
      queue_size: 100          # batch up to 100 lines per flush
      flush_timeout: 2         # or flush every 2s, whichever first
      # custom_fields_by_lua injects derived fields into every log object.
      # Values are Lua expressions evaluated in the log phase.
      custom_fields_by_lua:
        trace_id: "return (kong.request.get_header('traceparent') or '-'):sub(4,35)"
        consumer: "return (kong.client.get_consumer() or {}).username or 'anonymous'"
        ratelimit_remaining: "return kong.ctx.shared.ratelimit_remaining or -1"
        # Redaction: guarantee the auth header is never serialized
        authorization: "return nil"
        cookie: "return nil"

The traceparent header has the shape 00-<32 hex trace-id>-<16 hex span-id>-<flags>; sub(4,35) extracts exactly the trace-id segment so trace_id matches what the tracing backend indexes. Setting authorization and cookie to nil in the serializer is belt-and-braces: Kong does not log request headers by default, but pinning them to nil documents the intent and survives a future default change.

NGINX Plus R32+ builds JSON with a log_format directive using its escape=json mode, which correctly escapes control characters so a hostile header value cannot break the line. Variables map directly to the schema; $upstream_response_time and $request_time give the two-latency split.

# NGINX Plus R32+ — JSON access log with the required schema
log_format json_access escape=json
  '{'
    '"timestamp":"$time_iso8601",'
    '"route":"$route_name",'                 # set via map on location
    '"upstream":"$upstream_addr",'
    '"status":$status,'
    '"upstream_latency_ms":"$upstream_response_time",'
    '"total_latency_ms":"$request_time",'
    '"trace_id":"$trace_id",'                # extracted below
    '"consumer":"$jwt_claim_sub",'           # from auth_jwt module
    '"ratelimit_remaining":"$limit_req_status",'
    '"method":"$request_method",'
    '"path":"$uri",'                         # $uri excludes the query string
    '"client_ip":"$remote_addr"'
  '}';

# Extract the 32-char trace-id out of the incoming traceparent header
map $http_traceparent $trace_id {
  "~^00-(?<tid>[0-9a-f]{32})-" $tid;
  default "-";
}

server {
  listen 443 ssl;
  # $uri (not $request_uri) is logged so api_key=... query secrets never land in the log
  access_log /var/log/nginx/access.json json_access;
}

Logging $uri rather than $request_uri is a deliberate redaction choice: $request_uri includes the raw query string, which routinely carries api_key, token, or signature parameters. If you need query parameters, allowlist and log them individually rather than dumping the whole string.

Envoy 1.32+ configures a JSON access logger directly on the http_connection_manager via json_format. Envoy exposes rich command operators — %REQ(...)%, %RESP(...)%, %UPSTREAM_HOST%, %DURATION%, %RESPONSE_DURATION% — that map cleanly onto the schema, and %REQ(TRACEPARENT)% combined with a substring gives the trace ID.

# Envoy 1.32+ — JSON access log on the HTTP connection manager
access_log:
  - name: envoy.access_loggers.stdout
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
      log_format:
        json_format:
          timestamp: "%START_TIME%"
          route: "%ROUTE_NAME%"
          upstream: "%UPSTREAM_HOST%"
          status: "%RESPONSE_CODE%"
          # RESPONSE_DURATION = time to upstream first byte; DURATION = total
          upstream_latency_ms: "%RESPONSE_DURATION%"
          total_latency_ms: "%DURATION%"
          # TRACEPARENT is 55 chars: 00-<32>-<16>-<2>; chars 4-35 are the trace id
          trace_id: "%REQ(TRACEPARENT):4:32%"
          consumer: "%DYNAMIC_METADATA(envoy.filters.http.jwt_authn:sub)%"
          ratelimit_remaining: "%RESP(X-RATELIMIT-REMAINING)%"
          method: "%REQ(:METHOD)%"
          path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
          client_ip: "%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%"

The %REQ(TRACEPARENT):4:32% operator slices 32 characters starting at offset 4 — exactly the trace-id segment — so Envoy’s log lines join to the same traces as Kong’s. Note that Envoy never logs request headers unless you name them, so the Authorization header is safe by omission; the redaction risk in Envoy is in bodies (%REQUEST_BODY% / %RESPONSE_BODY% operators) and in dynamic metadata that might carry a raw claim — log the resolved sub, not the whole token.

Secondary Concept: Correlation, Log Levels, and Sampling

Trace correlation is the single feature that turns access logs from a forensic archive into a navigation surface. Because all three configurations above extract the trace ID from the same traceparent header into a field named trace_id, a log line and its distributed trace share a join key. In practice this means: your log UI renders trace_id as a clickable link to the trace, and your tracing UI renders a “view logs” button that queries trace_id = <id>. The mechanics of getting a consistent traceparent onto every request — generation at the edge, propagation to upstreams, and sampling-flag handling — are covered in distributed tracing and context propagation; structured logging is the consumer of that header, and the two signals are only as good as the discipline that keeps the ID identical in both.

Log levels on an access log are not the same as application log levels. There is no DEBUG access log line; instead, the “level” is really a decision about which requests to record in full. The workable taxonomy is: always record every non-2xx response, always record every request slower than a threshold (say the SLO latency budget), and treat the remaining fast 2xx traffic as sampleable. This keeps the signal — errors and slow requests, the things you actually investigate — at 100% fidelity while letting the high-volume success path be thinned.

Sampling the success path is where volume is controlled without losing incident-critical data. The anti-pattern is blind uniform sampling: dropping one line in ten uniformly means a one-in-ten chance the single 500 that explains the outage is gone. Instead, sample conditionally. Envoy expresses this directly with filter blocks on the access logger:

# Envoy 1.32+ — log 100% of errors and slow requests, sample the 2xx path at 10%
access_log:
  # Logger 1: everything that is NOT a fast success — kept in full
  - name: envoy.access_loggers.stdout
    filter:
      or_filter:
        filters:
          - status_code_filter:
              comparison: { op: GE, value: { default_value: 400 } }
          - duration_filter:
              comparison: { op: GE, value: { default_value: 500 } }   # >=500ms
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
      # ... json_format as above ...
  # Logger 2: the fast 2xx remainder, sampled at 10%
  - name: envoy.access_loggers.stdout
    filter:
      and_filter:
        filters:
          - status_code_filter:
              comparison: { op: LE, value: { default_value: 399 } }
          - runtime_filter:
              runtime_key: access_log.sample_success
              percent_sampled: { numerator: 10, denominator: HUNDRED }
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
      # ... json_format as above ...

Keep the log sampling decision aligned with trace sampling: if a request is sampled into the trace store but its log line is dropped, the “view logs” pivot returns nothing. Where possible, drive both from the same trace-flags bit so a sampled-in request always has both signals.

PII Redaction and Comparative Behavior

Redaction is the difference between an access log and a data breach. The three highest-risk sinks are the Authorization header (bearer tokens), the Cookie header (session identifiers), and request or response bodies (everything). The rule is to redact at the source, before the line is serialized, because once a token is written to disk or shipped to a third-party store, it must be treated as compromised and rotated. Redaction in the collector is a valuable second layer — it catches a field someone added upstream without thinking — but it is not a substitute for never emitting the secret in the first place.

The safe defaults across all three gateways: do not log request headers by name unless explicitly needed; log the resolved consumer ID, not the credential; strip query strings or allowlist parameters; and never enable body logging outside a time-boxed debugging session. When a consumer identity is genuinely sensitive, log a stable HMAC of it rather than the raw value, so you can still group by consumer without storing the identifier. These practices sit alongside the trust-boundary controls in security boundaries and zero-trust, where the same principle — sanitize at the edge, never trust that a downstream layer will clean up — governs header handling generally.

The comparative table summarizes how each gateway approaches structured logging, so you can match the mechanism to your operational model.

Operational Gotchas

Log volume blowout. Structured JSON is 3–5× larger per line than combined log format, and the moment someone enables body logging “just to debug”, volume can jump by orders of magnitude and saturate the collector or the store’s ingest quota. Bound the schema to a fixed field set, never log bodies by default, sample the success path, and compress at the collector. Watch the collector’s ingest-rate metric as closely as you watch the gateway’s request rate — a body-logging change deployed to production is a self-inflicted denial-of-service on your logging pipeline.

Unredacted tokens in logs. The classic failure is a log_format or serializer that dumps all request headers, shipping every bearer token to a third-party log SaaS. Because logs are often retained for months and replicated across regions, a single such line is a durable credential leak. Enforce a redaction review on any change to log configuration, and run a collector-side processor that drops or masks fields matching authorization, cookie, set-cookie, and secret-shaped query parameters — so a mistake upstream fails safe.

Free-text logs that cannot be queried. A line like GET /v2/pay 200 upstream=10.0.1.4 in 42ms for alice reads fine to a human and is useless to a machine: every field extraction is a bespoke regex, and one format change breaks every saved query. If you inherit free-text logs, the migration is to define the JSON schema first, then map each fragment to a field. The cost of not doing this is paid every incident, in minutes lost to grep.

Clock skew corrupting ordering. If gateway nodes drift, timestamp ordering across nodes becomes unreliable and correlation windows break. Log in UTC with millisecond precision, run NTP on every node, and prefer the collector’s receive time only as a fallback, never as the primary event time.

Dropping logs on backpressure. When the collector or store is slow, the gateway must not block the request path to write a log. Kong’s http-log and Envoy’s OTLP logger buffer and can drop under pressure — make that drop explicit and monitored (emit a metric for dropped lines) rather than discovering silent gaps during a post-mortem.

Production Configuration Checklist

• Every access log line is a single valid JSON object conforming to a documented, version-controlled schema.
• route, upstream, status, upstream_latency_ms, total_latency_ms, trace_id, consumer, and ratelimit_remaining are present on every line.
• trace_id is extracted from traceparent (offset 4, length 32) and named to match what the tracing backend indexes.
• Upstream latency and total gateway latency are logged as two separate fields.
• Authorization, Cookie, and Set-Cookie are never serialized; consumer identity is a resolved ID or HMAC, never the raw token.
• Query-string secrets are stripped — $uri/X-ENVOY-ORIGINAL-PATH logged, not the raw request URI — or parameters are allowlisted.
• Body logging is disabled by default and only enabled in a time-boxed debugging session.
• 100% of non-2xx responses and all requests above the latency threshold are logged; only the fast 2xx path is sampled.
• Log sampling is aligned with trace sampling so sampled-in requests carry both a log line and a trace.
• A collector-side redaction processor masks secret-shaped fields as a defence-in-depth second layer.
• Timestamps are UTC with millisecond precision and NTP runs on every node.
• Dropped-log-line counts are exported as a metric and alerted on.
• Retention is set per class: error and audit logs retained longer than sampled success logs.

FAQ

What fields must a gateway access log line contain to be useful in an incident?

At minimum: matched route or service name, upstream target, HTTP status, upstream_latency_ms, total_latency_ms, the trace_id, authenticated consumer identity, and rate-limit remaining. These let you pivot from a status-code spike to the exact route, upstream, and consumer responsible, and jump straight to the correlated trace. Free-text logs that concatenate these values into a message string are not queryable at scale and force regex parsing during an incident.

How do I correlate a gateway access log line with its distributed trace?

Emit the 32-character trace ID extracted from the W3C traceparent header as a top-level field in every log line, using the same field name your tracing backend indexes (typically trace_id). The gateway must parse the traceparent it received or generated and log the trace-id segment, not the whole header. Once trace_id is a first-class indexed field, one click pivots to the full trace, and distributed tracing and context propagation covers keeping that ID consistent end to end.

Should I sample gateway access logs?

Log 100% of errors and slow requests, and sample only the high-volume 2xx success path. A common pattern keeps every non-2xx response and every request above a latency threshold, then samples the remaining successes at 1–10%. Never apply blind uniform sampling that can drop the one 500 that explains an incident. Keep the sampling decision consistent with your trace sampling so a sampled-in request has both a log line and a trace.

How do I stop access logs from leaking tokens and PII?

Redact at the source before the line is written, not downstream. Never log the Authorization header, Cookie header, or raw request bodies. Log a stable hash or the consumer ID resolved by the auth layer instead of the credential. Strip or mask query-string secrets such as api_key, and add a redaction processor in the collector as a defence-in-depth second layer so a new field added upstream cannot silently ship a secret to your log store.

Why did my log volume and storage bill explode after enabling structured logging?

Structured JSON is larger per line than a terse combined-log-format line, and logging request/response bodies or every header multiplies volume by orders of magnitude. Bound the schema to a fixed field set, never log bodies by default, sample the success path, and batch and compress at the collector. Set retention by class: keep error and audit logs longer than sampled success logs, which can often be dropped within days.

Related

• Shipping Gateway Logs to the OpenTelemetry Collector — route the JSON emitted here through OTLP and filelog receivers, with redaction and batching processors.
• Distributed Tracing & Context Propagation — how the traceparent your logs correlate on is generated and propagated through the gateway.
• Gateway Metrics & SLOs — the aggregate latency histograms that the per-request latency fields here feed into.
• Security Boundaries & Zero-Trust — edge sanitization and trust-boundary discipline that underpins log redaction.

← Gateway Observability & Operations