Yong Lai

A passing /health check doesn't mean your vLLM engine can decode

2026-11-08MLOps: deploy, observe, operate · code & data

The claim

vLLM’s /health endpoint tells you the API server is alive, not that the engine can still produce tokens. When I froze the engine process, /health kept returning 200 and Kubernetes kept sending traffic to the pod. A readiness probe that runs a real 4-token completion removed the pod from the Service after 37 seconds. I also expected it to catch a broken compiler toolchain. It didn’t need to: with the prebuilt kernels a missing compiler is harmless, and without them vLLM’s own startup crashes the pod before it ever reports ready.

Setup

One GeForce RTX 3080 (10 GB, sm_86, driver 595.91.07) on a single-node k3s box. vllm/vllm-openai:v0.30.0 serving Qwen2.5-1.5B-Instruct in bf16, FlashInfer 0.6.18.post1. One replica, Recreate strategy. Startup and liveness probes are httpGet /health. Only the readiness probe changes between runs:

A client outside the cluster sends one such completion per second through the Service with a 5 s timeout. Each scenario below is a single run, except the healthy baseline.

Evidence

Engine frozen (SIGSTOP on the EngineCore process):

health-only probe decode probe
Pod leaves endpoints never (108 s observed) after 37 s
Client requests during freeze 18 of 18 timed out at 5 s 7 timed out at 5 s, then instant connection-refused
Restarts 0 0
After thaw 200s within 1 s pod READY and 200s within about 1 s

During the freeze I also called the pod directly: /health returned 200 in 18 ms while a sampled completion timed out at 5 s.

Healthy pod: time to READY was 45 to 50 s in both modes, and the decode probe never failed. The first user request took 40 to 43 ms behind the health probe and 29 ms behind the decode probe, because the probe’s own request had already warmed the path (one decode run only).

Broken toolchain, which I expected the probe to catch:

Why it happens

vLLM runs the API server and the engine as separate processes. /health is answered by the API server, so a stopped engine doesn’t affect it. Liveness uses the same endpoint, so it never fired either: 108 s of freeze, zero restarts.

The decode probe works because it asks the question users actually care about: can this pod return a token right now? The detection time is just failureThreshold × periodSeconds plus the timeout, so you can trade speed against false positives.

Startup is a different story. vLLM already runs a sampler pass while initialising, so a missing compiler fails the pod loudly before it ever reports ready. That is good behaviour, and it means the decode probe adds nothing for that class of failure.

What this does not show

Reproduce or verify it

Everything is in vllm-k8s/readiness/ of the repo linked above: the probe (decode_probe.py), the freeze script, the deployment with env switches (PROBE_MODE, BREAK_NVCC, REMOVE_JIT_CACHE), and a README with the exact commands. To check the core claim yourself, freeze the engine and compare the two calls:

kubectl -n vllm exec deploy/vllm-qwen -- python3 /experiment/freeze_engine.py freeze
# /health still answers 200; a sampled /v1/completions times out
kubectl -n vllm exec deploy/vllm-qwen -- python3 /experiment/freeze_engine.py thaw