Running AI systems in production: what works, what breaks, and why
I'm Yong, an engineer working on inference serving, MLOps and AI platforms. Every post makes one claim, shows the evidence behind it, and says what that evidence doesn't cover.
- A passing /health check doesn't mean your vLLM engine can decode
I froze the engine of a single-GPU vLLM pod on k3s. The default health check stayed green and the pod kept receiving traffic until I thawed it 108 seconds later. A dummy-decode readiness probe took it out of rotation after 37.
- What a 6-minute ML outage costs: 372 failed requests and 5.7% of the month's error budget
I scaled a model server to zero to test my monitoring. The alert fired after 2 minutes, 372 requests failed, and the latency graph went blank instead of red.
- FlashInfer's version guard checked the compiler, not the headers
Why vLLM crashed at sampler warmup on my machine, how a FlashInfer guard picked a CUB API using the wrong version, and why the fix was deleting nine lines instead of repairing the guard.
- Three wrong answers, three checks: Bedrock's grounding tiers against a claim-level verifier
Bedrock's default grounding check passed all three wrong answers from my insurance agent. The Standard tier caught one, a claim-level checker caught all three, and each had a cost. Injected errors made the two Bedrock grounding checks look better than they were.
- Fusion, not CUDA graphs: where the batch-1 decode speedup actually comes from
On a small model at batch 1, kernel fusion delivered 95% of torch.compile's 2.87x decode speedup and CUDA graphs added the rest. Plus a benchmark prompt that nearly fooled me.