Caching & Response Optimization

Gateway-layer caching is one of the highest-leverage performance controls available to platform teams, yet it is also one of the most common sources of security incidents and hard-to-diagnose latency regressions. Getting it right means understanding exactly where the cache interceptor sits within the broader middleware chain, how cache keys must be constructed to prevent cross-tenant data leakage, and how invalidation models must be designed so that event-driven purges — not just TTL expiry — keep data fresh. This page covers the architectural baseline, runnable configuration for Kong 3.x, NGINX 1.25+, and Envoy 1.32+, and the operational gotchas most likely to cause production incidents.

Architectural Baseline

Before configuring caching, engineers need a firm mental model of two constraints that all subsequent decisions flow from.

Security boundary first. The cache interceptor must execute after authentication proxying and token validation. If the cache check precedes authentication, an unauthenticated request can receive a previously cached authenticated response. This is not hypothetical — it is a class of vulnerability that has reached production in real systems when cache middleware was bolted on without auditing pipeline order.

Cache key correctness determines hit ratio and isolation. A cache key is a deterministic fingerprint of everything that makes two requests semantically equivalent. Get it wrong in either direction — too narrow (missing a Vary dimension like tenant ID) and you serve one tenant’s data to another; too broad (including X-Request-ID) and every request is a miss.

The diagram below shows the canonical request flow through a gateway with caching enabled, including the HIT and MISS branches.

Cache Key Normalization and Storage Tiers

Storage Tier Architecture

Production gateway caching uses three distinct storage tiers. Tier isolation prevents cross-tenant leakage and controls infrastructure cost.

Each tier should be configured with its own TTL envelope. L1 entries are short-lived (seconds to low minutes) to limit the stale window; L2 entries are medium-lived (minutes to hours); L3 is only consulted on misses and does not itself serve cached responses.

Cache Key Normalization

Deterministic key construction is the foundation of a correct cache. A normalization pipeline must:

1. Lowercase and percent-decode the path.
2. Sort query parameters lexicographically and remove parameters not relevant to the response (tracking IDs, analytics tokens).
3. Include only the Vary headers that genuinely differentiate responses: Accept-Encoding, Accept-Language, and any custom tenant or API version headers.
4. Explicitly exclude Authorization, Cookie, X-Request-ID, X-Correlation-ID, and X-Forwarded-For — these create per-user key fragments that destroy hit ratios and risk leaking session-scoped content across tenants.
5. Apply SHA-256 and truncate to 32 hex characters to produce a fixed-length key safe for all backing stores.

Kong 3.x — Proxy Cache Plugin

# Kong 3.x — declarative (deck) config, proxy-cache plugin
_format_version: "3.0"

services:
  - name: products-api
    url: http://products-upstream:8080
    routes:
      - name: products-route
        paths: ["/v1/products"]
        methods: ["GET", "HEAD"]
    plugins:
      - name: proxy-cache
        config:
          response_code: [200, 301, 404]
          request_method: ["GET", "HEAD"]
          content_type: ["application/json", "application/json; charset=utf-8"]
          cache_ttl: 300               # seconds; L2 envelope
          storage_ttl: 600             # backing-store TTL; must be >= cache_ttl
          strategy: memory             # or 'redis' for L2
          cache_control: true          # honour upstream Cache-Control directives
          ignore_uri_case: true
          vary_headers: ["Accept-Encoding", "X-Tenant-ID"]

cache_control: true tells Kong to respect Cache-Control: no-store and Cache-Control: private from the origin. Without this flag Kong will cache responses the origin has explicitly marked as uncacheable — a common misconfiguration on shared clusters.

For Redis-backed L2 caching, add the Redis connection stanza under config.redis:

          strategy: redis
          redis:
            host: redis.internal
            port: 6379
            timeout: 2000
            database: 0

NGINX 1.25+ — proxy_cache Directive

# nginx.conf excerpt — NGINX 1.25+
http {
    proxy_cache_path /var/cache/nginx
        levels=1:2
        keys_zone=api_cache:64m
        max_size=4g
        inactive=10m
        use_temp_path=off;

    server {
        listen 443 ssl http2;

        location /v1/products {
            proxy_pass         http://products_upstream;
            proxy_cache        api_cache;
            proxy_cache_key    "$scheme$host$uri$is_args$args";
            proxy_cache_valid  200 301  5m;
            proxy_cache_valid  404      1m;
            proxy_cache_use_stale  updating error timeout http_500 http_502 http_503;
            proxy_cache_lock       on;        # prevent thundering herd on MISS
            proxy_cache_lock_timeout 5s;

            # Serve stale for 60 s while background revalidation runs
            add_header Cache-Control "public, max-age=300, stale-while-revalidate=60";

            # Strip auth/session headers before key generation happens upstream
            proxy_hide_header Set-Cookie;
            proxy_hide_header Authorization;

            add_header X-Cache-Status $upstream_cache_status;
        }
    }
}

proxy_cache_lock on is the NGINX equivalent of a request coalescing mechanism: when many requests arrive simultaneously for an expired key, only one is forwarded to the origin. The rest wait (up to proxy_cache_lock_timeout) and then receive the freshly populated cached response. Without this, a burst of 200 requests after a single TTL expiry translates to 200 simultaneous origin calls.

Envoy 1.32+ — CacheFilter

# Envoy 1.32+ — HttpConnectionManager with CacheFilter
- name: envoy.filters.http.cache
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.cache.v3.CacheConfig
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.http.cache.simple_http_cache.v3.SimpleHttpCacheConfig
    allowed_vary_headers:
      - safeRegex:
          regex: "accept-encoding|x-tenant-id"
    cache_key_query_parameters:
      - name: "category"
        treat_missing_as_empty: true
      - name: "page"
        treat_missing_as_empty: true
    # Exclude tracking/session query params from key
    ignore_query_parameters:
      - "utm_source"
      - "utm_campaign"
      - "_ga"

Envoy’s CacheFilter is HTTP/2-aware and integrates directly with x-envoy-upstream-service-time spans, making cache status visible in the distributed trace without additional configuration.

Invalidation Models and stale-while-revalidate

stale-while-revalidate and stale-if-error (RFC 5861)

With plain max-age, a cache miss at expiry forces a synchronous round-trip to the origin — every client waiting on that entry experiences the full upstream latency spike simultaneously. stale-while-revalidate eliminates this by serving the expired entry immediately while issuing a background revalidation request. Clients see zero added latency; the cache updates silently once the origin responds.

The timeline below shows how the three cache states interact over a request stream.

The Cache-Control directive that produces this behaviour:

Cache-Control: public, max-age=60, stale-while-revalidate=30

This says: serve from cache for 60 seconds without checking the origin; for the next 30 seconds after that, continue serving the stale entry but revalidate in the background; after 90 seconds total, the entry is too stale to serve without blocking revalidation.

Complement this with stale-if-error for resilience during incident recovery:

Cache-Control: public, max-age=60, stale-while-revalidate=30, stale-if-error=600

stale-if-error=600 instructs the cache to serve the stale entry for up to 10 minutes if the origin returns a 5xx or is unreachable — a critical safety net that prevents a degraded upstream from cascading into a user-facing outage.

Event-Driven Purge and Tag-Based Invalidation

TTL-only invalidation is a safety net, not a strategy. When an upstream service mutates data, TTL allows the stale entry to persist until expiry — which can be minutes. For APIs serving product inventory, pricing, or user-facing state, this is unacceptable.

A production invalidation model has three layers:

1. Webhook triggers. The origin service publishes a mutation event to a message broker (Kafka, Redis Streams, or a webhook). The gateway subscribes and executes an immediate key-delete or PURGE request.
2. Tag-based purge. Related resources are grouped under logical tags — product:42, user:profile:99, tenant:acme. A single purge call clears all tagged keys atomically without scanning the full keyspace. Kong’s proxy-cache-advanced plugin supports tag purging via the Admin API; Varnish’s xkey module provides equivalent functionality for NGINX-fronted stacks.
3. TTL as a floor. If an event fails to propagate (network partition, broker downtime), the TTL guarantees eventual expiry. Size TTLs to the maximum acceptable staleness window for each endpoint class.

Kong Admin API purge by tag:

# Kong 3.x — purge all cache entries tagged 'product:42'
curl -X DELETE http://kong-admin:8001/proxy-cache/caches \
  -H "Content-Type: application/json" \
  -d '{"tags": ["product:42"]}'

Conditional GET Revalidation

For responses with ETag or Last-Modified, the cache should issue conditional requests on revalidation rather than re-fetching the full body:

• Client or cache sends If-None-Match: "etag-value" or If-Modified-Since: <date>.
• If the origin confirms the resource is unchanged, it returns 304 Not Modified (no body).
• The cache resets the TTL and returns the stored body to the client.

This cuts origin bandwidth significantly for large JSON payloads that rarely change.

Comparative Implementation Table

Middleware Pipeline Position

Rate limiting and throttling and cache logic interact in a way that has billing and quota implications. The decision depends on your accounting model:

• Cache interceptor after rate limiter: Cached responses consume quota. Clients cannot bypass rate limits by saturating the cache.
• Cache interceptor before rate limiter: Cached responses are free of quota cost. High-traffic read endpoints are served cheaply, but a client could issue unlimited reads for cached resources.

Neither is universally correct. Document the model in your gateway configuration and enforce it consistently across all routes.

The canonical pipeline order for a production gateway:

Request Parser & Normalizer
  → Auth Validator          (token/JWT/mTLS verification)
  → Rate Limiter            (quota enforcement)
  → Cache Interceptor       (read: return HIT; write: store response on MISS)
  → Request Transformer     (header injection, body rewrite)
  → Origin Proxy            (upstream routing, circuit breaker, retry)
  → Response Header Sanitizer (strip internal headers, inject X-Cache-Status)

For CORS policy enforcement, preflight OPTIONS requests should be cached aggressively at the edge — they are inherently stateless. Configure a separate, longer TTL for OPTIONS responses to avoid the round-trip cost on every browser preflight:

location /v1/ {
    if ($request_method = OPTIONS) {
        add_header Cache-Control "public, max-age=86400";
        add_header Access-Control-Max-Age 86400;
        return 204;
    }
}

Observability

Cache behavior must be fully observable. Without structured telemetry, a hit ratio regression or a stampede event is invisible until end users report latency.

Span attributes. Inject cache.status with values HIT, MISS, STALE, BYPASS, REVALIDATED, and ERROR into every distributed trace span. Propagate traceparent (W3C Trace Context) to origin services so that a MISS can be correlated end-to-end. Kong’s OpenTelemetry plugin and Envoy’s built-in tracing both support custom attribute injection.

Metrics. Expose and alert on:

• Cache hit ratio (alert on sustained drop below threshold — e.g., below 80% for endpoints expected to be cache-heavy).
• Stale-serve latency (should be near zero; a spike indicates background revalidation is queuing).
• Purge latency (event-driven purge should complete in under 200 ms; longer indicates broker lag).
• Origin offload percentage (the fraction of requests served without reaching the upstream).

Header audit. Log Cache-Control, ETag, Vary, and X-Cache-Status headers on every response. Discrepancies between what the origin sends and what the cache applies are the leading indicator of misconfiguration.

X-Cache-Status header. Always inject this into responses headed to clients. It is the single fastest debugging tool available to API consumers and support engineers:

X-Cache-Status: HIT          # served from cache
X-Cache-Status: MISS         # fetched from origin, now cached
X-Cache-Status: BYPASS       # cache skipped (no-cache directive, auth route, etc.)
X-Cache-Status: STALE        # served stale while revalidating
X-Cache-Status: REVALIDATED  # origin confirmed fresh; TTL reset

Operational Gotchas

Caching responses that contain Set-Cookie. A cached response that includes a Set-Cookie header will deliver a previous user’s session cookie to a subsequent user. The cache layer must strip or block Set-Cookie before storing any response. Kong’s proxy-cache does not strip cookies automatically — configure proxy_hide_header Set-Cookie in NGINX or a Response Transformer plugin in Kong.

Vary: * from the origin. If an upstream service responds with Vary: *, it signals that no two requests are equivalent. A cache that honors this correctly will never store the response. If the header appears erroneously (a misconfigured framework default), fix it at the origin or strip it at the gateway using a response transformer.

TTL misalignment between L1 and L2. If an L1 (edge) TTL is longer than the L2 (Redis) TTL, a cache node may serve an entry from memory after it has been purged from Redis. Keep L1 TTLs shorter than L2 TTLs and purge both tiers atomically.

Thundering herd on cold start. When a cache node restarts or a new region comes online, all entries are cold simultaneously. Traffic that was previously absorbed by cache hits floods the origin. Mitigate with: (1) request coalescing (proxy_cache_lock on in NGINX; Kong’s batch-coalescing behavior), (2) a cache warm-up job that pre-populates common keys from a snapshot, and (3) stale-if-error to keep the previous region’s content valid during the transition. For guidance on how this fits into broader resilience planning, see high-availability topologies.

Caching 4xx responses. Caching a 404 or 403 can be intentional (reduce origin load for invalid paths) or catastrophic (a momentary permissions error becomes permanent for TTL duration). Gate 4xx caching behind an explicit allowlist and use very short TTLs (seconds, not minutes).

Cache bypass on Cache-Control: no-cache vs no-store. no-cache means “revalidate before serving from cache” — the gateway should issue a conditional GET. no-store means “do not cache at all.” These are frequently conflated in gateway configuration. Test both directive values explicitly during validation.

Production Configuration Checklist

• Cache interceptor runs after authentication in the middleware chain.
• Cache key excludes Authorization, Cookie, X-Request-ID, and X-Forwarded-For.
• Vary header dimensions are explicitly enumerated and match the key construction logic.
• Set-Cookie headers are stripped from all stored responses.
• stale-while-revalidate is configured on all high-traffic GET endpoints.
• stale-if-error is set on all endpoints where origin degradation must not cause user-facing errors.
• Request coalescing (cache lock) is enabled to prevent thundering herd on MISS.
• Event-driven purge is wired to all state-mutating upstream services.
• Tag-based purge groups are defined for related resource families.
• X-Cache-Status header is injected on all cached and bypass responses.
• Cache hit ratio, stale-serve rate, and purge latency are instrumented and alerted.
• Cache-Control: no-store responses are verified never to be stored.
• L1 TTL is shorter than L2 TTL to prevent stale memory entries after purge.
• Cold-start warm-up procedure is documented and tested.
• 4xx caching is either disabled or uses TTLs of 30 seconds or less.

FAQ

Where in the middleware pipeline should cache logic run?

The cache interceptor must run after authentication to prevent serving cached responses to unauthenticated requests. Place it after rate limiting if cached responses should count against quota; before rate limiting if you want cached responses served without consuming quota.

What headers must be excluded from cache key computation?

Exclude Authorization, Cookie, X-Request-ID, and X-Forwarded-For from cache keys. Including these creates per-user key fragments that destroy hit ratios and risk leaking session-scoped content across tenants.

How does stale-while-revalidate differ from plain max-age caching?

With plain max-age, a cache miss at expiry forces a synchronous origin request, adding full round-trip latency. stale-while-revalidate (RFC 5861) serves the expired entry immediately while revalidation runs in the background, eliminating that latency spike for high-traffic endpoints.

Up: Middleware Chains & Request Transformation

Related:

• Edge Caching: Varnish vs Gateway-Native Cache — when a dedicated Varnish tier beats the gateway’s built-in response cache
• Authentication Proxying & Token Validation — must run before the cache layer to prevent unauthenticated cache hits
• CORS & Cross-Origin Security — preflight OPTIONS caching strategy and header coordination
• High Availability Topologies — how distributed cache tiers fit into multi-region gateway deployments
• Scaling Limits & Capacity Planning — cache-layer sizing, Redis cluster capacity, and origin offload targets
• Kong vs Tyk vs Envoy for Microservices — gateway capability comparison relevant to cache feature support