Rotating JWT Signing Keys Without Downtime

Signing-key rotation is the operation most teams postpone until a compromise forces it — and then discover their gateways reject a wave of otherwise-valid tokens the moment the new key goes live. The failure is almost never cryptographic. It is a timing problem: the issuer starts signing with a new key before every validating gateway has learned that key, or an operator retires the old key while tokens signed by it are still in flight. This runbook rotates RS256 or ES256 signing keys behind a Kong or Envoy authentication layer with zero rejected requests, using overlapping key validity, a multi-kid JWKS endpoint, and a staged rollout that is trivially reversible.

Prerequisite concepts

This procedure assumes the gateway already validates tokens as part of its authentication proxying and token validation layer, and that you understand where signature verification sits in the broader middleware chain — auth runs early, before rate limiting and transformation, so a validation failure here rejects the request before any downstream work happens. If you are configuring Kong’s JWT verification for the first time, start with implementing JWT validation in Kong plugins; this runbook builds on a working validator and focuses only on rotating the key material underneath it.

The core mechanism that makes zero-downtime rotation possible is the kid (key ID) header inside every JWT. When the issuer signs a token, it stamps the header with the kid of the signing key. The gateway reads that kid, looks up the matching public key in a JSON Web Key Set (JWKS), and verifies the signature. Because the JWKS can hold many keys at once, the old and new keys coexist during the transition — which is what turns a hard cutover into a safe overlap.

The overlap model

Three clocks govern a safe rotation, and the overlap window must be longer than all of them combined:

The overlap window must satisfy: overlap ≥ max_token_lifetime + max_jwks_cache_ttl + safety_margin. If access tokens live 15 minutes and the longest gateway cache TTL is 10 minutes, the retiring key must stay published for at least 25 minutes; round up to an hour in practice. Every phase below exists to keep that inequality true.

Step 1 — Publish the new key before signing with it

Generate the new key pair (RS256 shown; ES256 is identical except for alg and crv) and add its public half to the JWKS while the old key still signs every token. The JWKS now advertises two keys with distinct kids:

{
  "keys": [
    {
      "kty": "RSA",
      "kid": "2025-q4",
      "use": "sig",
      "alg": "RS256",
      "n": "0vx7ag...old-modulus...",
      "e": "AQAB"
    },
    {
      "kty": "RSA",
      "kid": "2026-q1",
      "use": "sig",
      "alg": "RS256",
      "n": "sXchD1...new-modulus...",
      "e": "AQAB"
    }
  ]
}

Do not sign a single production token with 2026-q1 yet. The purpose of this phase is only to let every gateway learn the new key so that when you flip the signer, the key is already trusted everywhere.

Step 2 — Let every gateway refresh, then verify

Both gateways fetch the JWKS on a timer and cache it. You must wait out that cache before switching the signer.

Kong 3.x — the openid-connect plugin discovers the JWKS from the issuer and refreshes it. Pin the refresh behaviour explicitly so it is not left to defaults:

# Kong 3.x — openid-connect with bounded JWKS refresh
plugins:
  - name: openid-connect
    config:
      issuer: https://idp.internal/.well-known/openid-configuration
      # re-fetch the discovery + JWKS document on this interval (seconds)
      rediscovery_lifetime: 300
      # verify signature against the discovered JWKS
      verify_signature: true
      # on an unknown kid, force an immediate JWKS re-fetch before rejecting
      rediscovery_lifetime_on_error: 10
      scopes_required: ["api.read"]

For the leaner jwt-signer plugin, the equivalent knob is <key>_jwks_uri paired with a rotation interval:

# Kong 3.x — jwt-signer verifying against a remote JWKS
plugins:
  - name: jwt-signer
    config:
      verify_access_token_signature: true
      access_token_jwks_uri: https://idp.internal/jwks.json
      # cache lifetime for the fetched JWKS, in seconds
      access_token_keyset: "rotating-keyset"
      access_token_upstream_leeway: 30   # clock-skew tolerance (seconds)

Envoy 1.32+ — the jwt_authn filter fetches remote_jwks and caches it for cache_duration. Set that duration deliberately; it is the term that dominates your overlap math:

# Envoy 1.32+ — jwt_authn with a remote, timed-refresh JWKS
http_filters:
  - name: envoy.filters.http.jwt_authn
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
      providers:
        idp_provider:
          issuer: https://idp.internal/
          audiences: ["api.internal"]
          # allow tokens signed by any kid currently in the JWKS
          remote_jwks:
            http_uri:
              uri: https://idp.internal/jwks.json
              cluster: idp_jwks_cluster
              timeout: 5s
            # how long Envoy caches the fetched key set before re-fetching
            cache_duration: 600s
          # tolerate small issuer/gateway clock differences
          clock_skew_seconds: 60
      rules:
        - match: { prefix: "/api/" }
          requires: { provider_name: idp_provider }

Confirm the refresh actually happened before proceeding. Mint a test token signed with the new kid and send it through each gateway; a 200 proves that gateway has the new key. Do not rely on wall-clock time alone across a fleet — a node that was slow to refresh, or whose fetch to the identity provider failed, still holds only the old key and will reject new-key tokens.

Step 3 — Switch the signer

With both keys trusted everywhere, point the issuer at 2026-q1 and start stamping new tokens with the new kid. Nothing on the gateways changes. Old tokens (still carrying kid=2025-q4) continue to verify against the old key that remains in the JWKS; new tokens verify against the new key. This is the only moment traffic changes, and because both keys are live, there is no window where a valid token lacks a matching key.

Step 4 — Drain, then retire the old key

Wait out the drain: hold until every token signed by the old key has passed its exp. Only then remove 2025-q4 from the JWKS — as a separate, final change. Because the old key was never withdrawn while it could still sign or verify live tokens, rollback at any earlier point is trivial: repoint the signer to the old kid, which is still published, and no gateway edit is needed.

Decision matrix

Gotchas and failure signals

Stale JWKS cache. The signature-failure spike right after you flip the signer almost always means a gateway is still serving new-kid tokens against a cached key set that predates Step 1. Signal: 401 with a “no matching key”/“kid not found” reason on a subset of nodes. Fix: never advance to Step 3 until a new-kid test token returns 200 on every node; set rediscovery_lifetime / cache_duration low enough that the overlap window comfortably exceeds it.

Missing kid in the token header. If the issuer omits kid, the gateway must try every key in the JWKS or guess, and during a two-key overlap it may pick the wrong one. Signal: intermittent 401 that correlates with which key the gateway tried first. Fix: require a kid header on every signed token — it is what makes multi-key JWKS lookups deterministic.

Clock skew. A new key that “doesn’t work” can actually be a token whose nbf/iat sits slightly in the gateway’s future because the issuer’s clock runs ahead. Signal: 401 with a not-yet-valid reason clustered on freshly issued tokens. Fix: set clock_skew_seconds / access_token_upstream_leeway to 30–60 s and keep NTP healthy on both issuer and gateways.

Retiring the old key too early. Deleting 2025-q4 before the drain completes rejects every still-valid old token at once. Signal: a step-change in 401s the instant you edit the JWKS, not gradual. Fix: gate Step 4 behind “max token lifetime has elapsed since the signer switch,” and treat old-key removal as its own change with its own verification.

Validation

• New public key is published to the JWKS with a unique kid before any token is signed with it.
• A test token signed with the new kid returns 200 on every gateway node before the signer is switched.
• Overlap window ≥ max token lifetime + longest JWKS cache TTL (cache_duration / rediscovery_lifetime) + margin.
• Every issued token carries an explicit kid header.
• clock_skew_seconds / access_token_upstream_leeway is set (30–60 s) and NTP is healthy on issuer and gateways.
• Old key is removed from the JWKS only after the full drain, as a separate change.
• Rollback rehearsed: repointing the signer to the old kid restores service with no gateway edit.
• 401 rate by reason (kid not found, token expired, not yet valid) is dashboarded across the rotation window.

FAQ

How long must old and new JWT signing keys overlap during rotation?

The overlap window must be at least the maximum token lifetime plus the longest JWKS cache TTL across every validating gateway, plus a safety margin. If access tokens live 15 minutes and gateways cache the JWKS for 10 minutes, keep the retiring key published for at least 25 minutes, and in practice round up to an hour. Retiring the old key before every in-flight token has expired and every cache has refreshed causes valid tokens to fail signature verification.

Why do some tokens fail validation immediately after I publish a new signing key?

The most common cause is a stale JWKS cache. Gateways such as Kong and Envoy fetch the JWKS once and cache it for a configured TTL, so a token signed with a brand-new kid arrives before the gateway has refreshed its key set and fails because the kid is unknown. Always publish the new public key to the JWKS endpoint and let every gateway refresh before you begin signing new tokens with it. The second most common cause is signing tokens without a kid header, which forces the gateway to guess which key to use.

Do I need to restart Kong or Envoy to pick up a rotated JWKS?

No. Both gateways refresh remote JWKS on a timer without a restart. Envoy’s jwt_authn filter re-fetches remote_jwks based on cache_duration, and Kong’s openid-connect plugin refreshes the discovered JWKS on its own interval or on encountering an unknown kid. Restarting only helps if you have misconfigured the refresh interval or the identity provider is unreachable, and a restart during rotation risks dropping in-flight connections, so rely on timed refresh instead.

How do I roll back a JWT key rotation that went wrong?

Because a correct rotation never removes the old key until the new one is fully trusted, rollback is simply switching the signer back to the previous kid while both keys are still published in the JWKS. Point the token issuer at the old signing key, confirm the old kid is still in the JWKS, and no gateway change is required because the old public key was never withdrawn. This is why the runbook publishes before signing and removes the old key only in a final, separate step.

Parent: Authentication Proxying & Token Validation

Related

• Implementing JWT Validation in Kong Plugins — the base validator this runbook rotates keys underneath.
• Authentication Proxying & Token Validation — where signature verification sits in the token-handling flow.
• Middleware Chains & Request Transformation — how auth ordering interacts with the rest of the policy chain.