Load Testing an API Gateway with k6

A gateway load test that reports a big requests-per-second number and nothing else is almost worthless. The number you actually need is the knee — the load level at which latency stops scaling linearly and turns sharply upward because some resource, usually the upstream connection pool or worker capacity, has saturated. Finding that knee, and proving it sits comfortably above your peak traffic with headroom to spare, is the whole job. This runbook walks through designing a k6 test that models real routes and authentication, ramps load in controlled steps, enforces P99 and error-rate thresholds as pass/fail gates, and reads the results to locate the knee — with the Kong 3.x and Envoy 1.32+ tuning knobs that move it.

The scenario and what is at stake

You are about to put a gateway in front of production traffic, or you have changed its configuration, and you need to know how much load it absorbs before latency degrades. The stakes are asymmetric: a test that is too optimistic — because it accidentally measured a warm cache, or hammered the load generator instead of the gateway — gives you false confidence that shatters during the first real traffic spike. A rigorous test tells you the safe operating ceiling, which resource you will hit first, and which knob to turn to move it.

Prerequisite concepts

This runbook operationalizes the theory in scaling limits and capacity planning — connection-pool sizing, throughput ceilings, and the route-table growth curve — and assumes the request-lifecycle model from API gateway fundamentals and architecture, because you cannot interpret a latency spike without knowing which stage of the request path produced it. Retry behavior interacts with load testing in a way that will surprise you if you ignore it: an aggressive retry policy turns a momentary saturation into a self-amplifying storm, so read circuit breaking and retry budgets before you enable retries on the path under test, or your knee will appear far earlier than the gateway’s real ceiling.

Test design: model the real request path

A load test is only as honest as its model. If your production traffic is 70% authenticated GETs across 40 routes with a mix of cache hits and misses, a test that fires one unauthenticated route at a static endpoint measures nothing you care about. Model three things: the route distribution (weight requests the way production does), the auth path (send real tokens through the real validation, because JWT verification is CPU work the gateway actually does), and the payload sizes that match production.

The diagram shows the closed loop you are building and where each measurement is taken.

The k6 script

The script below models a weighted route mix, sends a bearer token through real JWT validation, and encodes the SLO as thresholds so the run fails automatically when the SLO is breached. It uses a ramping-vus executor to step load up and find the knee rather than firing a flat rate.

// k6 — gateway capacity test: ramp VUs, weighted routes, SLO thresholds
import http from 'k6/http';
import { check } from 'k6';
import { Rate } from 'k6/metrics';

const errorRate = new Rate('gateway_errors');

export const options = {
  scenarios: {
    ramp: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '2m', target: 200 },   // warm up, fill keepalive pool
        { duration: '3m', target: 800 },    // approach expected peak
        { duration: '3m', target: 1600 },   // push past peak to find the knee
        { duration: '2m', target: 0 },      // ramp down
      ],
      gracefulRampDown: '30s',
    },
  },
  // Thresholds ARE the pass/fail gate — set from your SLO, not from a prior run.
  thresholds: {
    http_req_duration: ['p(95)<150', 'p(99)<400'], // ms
    gateway_errors: ['rate<0.005'],                 // < 0.5% errors
    checks: ['rate>0.999'],
  },
  // Reuse connections so we test the gateway, not TLS handshake cost.
  noConnectionReuse: false,
  discardResponseBodies: true,
};

const BASE = 'https://gateway.internal';
const TOKEN = __ENV.API_TOKEN; // real token: exercise JWT validation on the hot path

// Weighted route mix mirroring production traffic shape.
const routes = [
  { path: '/api/v2/catalog',        weight: 55 },
  { path: '/api/v2/pricing',        weight: 25 },
  { path: '/api/v2/orders/search',  weight: 15 },
  { path: '/api/v2/account',        weight: 5 },
];

function pickRoute() {
  const r = Math.random() * 100;
  let acc = 0;
  for (const route of routes) {
    acc += route.weight;
    if (r <= acc) return route.path;
  }
  return routes[0].path;
}

export default function () {
  const res = http.get(`${BASE}${pickRoute()}`, {
    headers: { Authorization: `Bearer ${TOKEN}` },
    tags: { name: 'gateway_request' }, // group metrics by logical name, not raw URL
  });
  const ok = check(res, {
    'status is 2xx': (r) => r.status >= 200 && r.status < 300,
  });
  errorRate.add(!ok);
}

Two details in that script are load-bearing. First, noConnectionReuse: false keeps keepalive on, so you measure the gateway’s steady-state routing cost rather than a TCP-plus-TLS handshake on every request — testing with reuse off measures your handshake path, not your gateway. Second, the tags: { name: ... } on the request groups metrics by a stable logical name; without it, k6 fragments its percentiles across every distinct URL and your P99 becomes meaningless.

Gateway tuning knobs

When the test reveals a knee below your target, these are the knobs that move it. On Kong 3.x, the upstream keepalive pool and worker connections govern how many concurrent upstream requests a node sustains:

# Kong 3.x — the knobs that move the connection-pool knee (kong.conf / env)
KONG_NGINX_WORKER_PROCESSES=auto        # one worker per core
KONG_NGINX_HTTP_KEEPALIVE_REQUESTS=10000 # requests per upstream keepalive conn
KONG_UPSTREAM_KEEPALIVE_POOL_SIZE=512    # idle upstream conns kept warm per worker
KONG_UPSTREAM_KEEPALIVE_MAX_REQUESTS=10000
KONG_NGINX_EVENTS_WORKER_CONNECTIONS=16384

On Envoy 1.32+, the equivalent ceiling is the upstream cluster’s circuit-breaker thresholds, which cap concurrent connections and pending requests to the upstream — set them too low and Envoy queues or rejects long before the host is saturated:

# Envoy 1.32+ — cluster circuit-breaker thresholds cap the upstream pool
clusters:
  - name: origin
    circuit_breakers:
      thresholds:
        - priority: DEFAULT
          max_connections: 2048
          max_pending_requests: 1024
          max_requests: 4096
          max_retries: 32
    upstream_connection_options:
      tcp_keepalive: { keepalive_probes: 3, keepalive_time: 30 }

If P99 climbs while max_pending_requests is being hit, requests are queuing for a connection — raise max_connections (and the origin’s own accept capacity) to move the knee outward. If CPU saturates first, you have hit the worker/event-loop ceiling and the answer is horizontal scale, not pool tuning.

Test profiles

Different questions need different load shapes. Pick the profile that matches the question you are asking.

The ramp profile finds the knee; the soak profile catches the slow connection or file-descriptor leak that a short test never sees; the breakpoint profile with ramping-arrival-rate (open model — arrivals do not wait for prior responses) reveals the true ceiling because it does not let a slow gateway self-throttle the offered load, which a closed VU model silently does.

Reading the results

Plot P99 latency against achieved throughput across the ramp. Three regions appear: a flat region where latency is stable as RPS rises, the knee where P99 curves sharply upward while RPS stops increasing, and the saturated region past the knee where latency explodes and errors appear. Your safe operating ceiling is comfortably below the knee — typically 60–70% of the knee’s throughput — to leave headroom for traffic bursts and node failures. Cross-reference the moment P99 curves with the gateway metric that flatlined at its maximum: if max_pending_requests or the Kong pending-connection count pinned at its ceiling exactly at the knee, the connection pool is your limit; if worker CPU hit 100% first, compute is the limit.

Gotchas and failure signals

Testing the client, not the gateway. If the origin is a real service, you may be measuring the origin’s saturation, not the gateway’s. Front the gateway with a fixed-latency stub so the origin can never bottleneck, or baseline the origin alone and subtract it.

Keepalive skew. Testing with connection reuse disabled measures TCP/TLS handshake cost and drastically understates the gateway’s real throughput; testing with an unrealistically high reuse rate overstates it. Match the reuse behavior to production client behavior.

The load generator is the bottleneck. A single k6 host runs out of CPU, ephemeral ports, or file descriptors long before a well-tuned gateway does. If results plateau while the gateway’s CPU is idle, check the generator’s own htop and ulimit -n before concluding anything about the gateway. Distribute k6 across multiple hosts for high VU counts.

Cold-cache and cold-pool bias. The first seconds of a test hit cold caches and empty keepalive pools, inflating latency. Include a warm-up stage and read percentiles only from the steady-state window.

Retries masking the knee. If retries are enabled on the tested path, a saturating gateway retries failing requests and amplifies its own load, moving the apparent knee earlier and making the failure look worse than the raw ceiling. Understand circuit breaking and retry budgets before interpreting a test with retries on.

Percentile fragmentation. Without a stable metric name tag, k6 computes percentiles per distinct URL and your aggregate P99 is noise. Always tag requests by logical route.

Validation

• Origin is a fixed-latency stub, or the origin baseline is measured and subtracted, so the test isolates the gateway.
• Route mix, auth path, and payload sizes mirror production traffic shape — not a single trivial endpoint.
• Thresholds encode the committed SLO (P99, P95, error rate, check rate) and fail the run on breach.
• Connection reuse (keepalive) matches production client behavior; handshake cost is not accidentally dominating.
• The load generator has verified spare CPU, ports, and file descriptors at peak VUs (distributed if needed).
• A warm-up stage precedes measurement; percentiles are read from the steady-state window only.
• The knee is located by plotting P99 vs throughput, and the saturating gateway metric (pending requests / CPU) is identified.
• Retries on the tested path are understood and either disabled or accounted for in interpretation.
• The chosen safe operating ceiling sits well below the knee with documented headroom for bursts and node loss.

FAQ

How do I make sure I am load testing the gateway and not my origin?

Point the gateway at a mock or fixed-latency stub upstream so the origin can never be the bottleneck, then watch the gateway’s own resource and connection metrics. If the origin serves an instant static response and the P99 still climbs as you add load, the added latency is the gateway’s. If you must test against a real origin, run a baseline directly against the origin first and subtract it, so you are measuring the gateway’s marginal contribution rather than the origin’s saturation.

What thresholds should a gateway load test enforce?

Encode your SLO as k6 thresholds so the test fails automatically when it is breached. Typical thresholds are a P99 request duration ceiling, a P95 for the common case, an error rate under a fraction of a percent, and a check-failure rate near zero. Set them from the SLO you actually commit to, not from whatever the current run produces, so the test becomes a regression gate rather than a rubber stamp.

What is the connection-pool knee and how do I find it?

The knee is the load level where latency stops rising linearly and turns sharply upward because a resource has saturated — most often the gateway’s upstream connection pool or its worker capacity. You find it with a ramping test that increases virtual users in steps while you plot P99 against throughput. The point where P99 curves away from flat while requests-per-second stops increasing is the knee; the pool metric that flatlines at its maximum at that same moment tells you which resource saturated.

Why do my k6 results get worse when I add more virtual users but CPU is idle?

That pattern points to a connection or queue limit rather than compute saturation. Either the gateway’s upstream connection pool is exhausted and requests are queuing for a free connection, or keepalive is misconfigured so every request pays a new TCP and TLS handshake, or the k6 load generator itself is the bottleneck. Check the gateway’s pending-request and active-connection metrics, confirm keepalive is on for both client and upstream sides, and verify the load generator has spare CPU and file descriptors before blaming the gateway.

Related

• Scaling Limits & Capacity Planning — connection-pool sizing and throughput-ceiling theory that this test measures empirically.
• Circuit Breaking & Retry Budgets — why retries on the tested path move the apparent knee and how to bound them.
• API Gateway Fundamentals & Architecture — the request-lifecycle model you need to interpret which stage produced a latency spike.

← Scaling Limits & Capacity Planning