One Bad Request Killed Your Whole vLLM Server: AsyncEngineDeadError Explained

Your local LLM server was serving fifty concurrent users happily. Then one request came in, and now every request is failing with the same error, including ones that worked a second ago. The GPU is still allocated, the process is still up, the port still answers, but every generation returns an exception. Restarting the process fixes it instantly, which somehow makes it more alarming, not less.

What happened is that vLLM does not run your requests in isolation. It runs them together, in one shared background loop, because that is what makes it fast. And when something in that loop threw an exception nobody caught, the loop did not fail one request. It died. Every request after that is talking to a corpse.

This is not a config you got wrong. It is a real architectural sharp edge in how high-throughput inference servers trade isolation for speed. You cannot fully remove it, but you can stop it from turning into an outage.

The exact error

vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already.

You will usually see it preceded, once, by the real cause a few lines up in the logs, and then repeated on every subsequent request. The most common upstream trigger is:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate ...
# followed, for every request after, by:
AsyncEngineDeadError: Background loop has errored already.

The first line tells you what killed the loop. Every AsyncEngineDeadError after it is just the engine reporting that it is already dead. If you only grep for the AsyncEngineDeadError, you will miss the actual cause sitting right above the first occurrence.

Why one request kills everything: the shared background loop

To serve many requests on one GPU efficiently, vLLM does not spin up a worker per request. It runs a single asynchronous engine loop that continuously gathers all pending requests, forms them into batches (continuous batching, the core of its throughput), runs a step on the GPU, streams tokens back, and repeats. Every user's request is a passenger on the same loop.

That design is the reason vLLM is fast, and it is also the reason for this failure. If an exception is raised inside that loop and is not handled, the loop task terminates. There is no supervisor that restarts it in place, and there cannot easily be one, because the GPU state at the moment of the crash (a half-allocated KV cache, an OOM condition, a wedged CUDA context) is not something you can cleanly resume from. So the engine marks itself errored, and from then on every request short-circuits to AsyncEngineDeadError.

The triggers vary. A single request asking for far more tokens than the GPU has memory for can OOM the whole batch. A malformed or edge-case input can hit a code path that raises. A resource limit gets crossed under a burst of load. In every case the blast radius is the same: not one failed request, but the whole engine. This is the price of shared batching. True per-request isolation would mean per-request memory arenas and no shared batch, which would throw away most of the throughput that is the entire point of vLLM.

How to contain it, step by step

Step 1: Detect the dead engine and restart automatically

Since recovery means a process restart, the job of your platform is to notice the engine is dead and restart it fast. vLLM's OpenAI-compatible server exposes a /health endpoint that reflects engine state. Wire it to a Kubernetes liveness probe:

livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 60   # model load is slow; do not probe too early
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 5

Set initialDelaySeconds generously, because loading a large model into GPU memory takes time and an aggressive probe will kill a pod that is merely still starting. Be careful not to point liveness and readiness at logic that shares a failure the way I describe in the shared /health endpoint that kills healthy pods; here a simple engine-health check is what you want.

Step 2: Keep bad requests away from the engine

The cheapest crash is the one that never reaches the loop. Put a validation layer in front that rejects the request shapes known to OOM or wedge the engine, and cap the model context at start:

python -m vllm.entrypoints.openai.api_server   --model your-model   --max-model-len 8192   --gpu-memory-utilization 0.90

Then validate at your gateway: reject prompts above a token budget you have actually tested, clamp max_tokens to a sane ceiling, and reject empty or malformed inputs (an empty prompt has been a real crash trigger). Structured, bounded inputs are far less likely to hit a pathological path; my note on constrained decoding and structured outputs is a related habit of not trusting free-form generation to stay in bounds.

Step 3: Leave GPU memory headroom

Setting --gpu-memory-utilization to 0.95 or higher to squeeze in a bigger KV cache is exactly how you turn a large batch into an OOM crash. Leave headroom so a spike in concurrent long-context requests has somewhere to go. It is better to serve slightly fewer tokens per second than to crash the whole engine at peak.

Step 4: Run more than one engine

This is the mitigation that actually changes the outcome. A single vLLM instance is a single point of failure by construction. Run two or more independent replicas behind a load balancer:

spec:
  replicas: 3
  strategy:
    rollingUpdate:
      maxUnavailable: 1

Now when one engine dies, the load balancer routes around it while the liveness probe restarts it, and you lose a third of capacity for a minute instead of the entire service. Replicas do not prevent the crash; they demote it from an outage to a blip.

Verification

Confirm the health signal flips when the engine dies and that your restart actually fires. Check the endpoint directly:

# healthy engine returns 200
curl -s -o /dev/null -w "%{http_code}
" localhost:8000/health

# after a crash, watch the orchestrator restart the pod
kubectl get pods -l app=vllm -w
kubectl logs deploy/vllm --previous | grep -iE "OutOfMemory|AsyncEngineDeadError"

The --previous logs are where the real cause lives, above the first AsyncEngineDeadError. A rising RESTARTS count that correlates with OOM lines is your proof the probe is catching dead engines. If restarts are frequent, the fix is upstream: smaller context, more headroom, or better input validation, not a faster probe.

What people get wrong

Treating it as a config bug to be tuned away. There is no single flag that makes the shared loop crash-proof. Chasing a magic setting wastes time that belongs on the real mitigations: input limits, headroom, health-based restart, replicas.

Wrapping the client call in a retry and calling it done. Retrying against a dead engine just hits AsyncEngineDeadError again immediately, because the engine does not self-heal. Retries only help if they route to a different healthy replica, which means you need multiple replicas and a load balancer first. A retry loop against one dead instance is a busy-wait.

Cranking --gpu-memory-utilization to the ceiling for throughput. This is the single most common way people cause the OOM that kills the loop. The extra cache capacity is not worth converting a slow moment into a crash.

Assuming your version behaves like the old one. vLLM's engine internals have changed substantially over time and newer releases isolate and report engine failures better than early ones did. If you are pinned to an old version because of this exact error, upgrading is a legitimate part of the fix, tested in staging first.

When it is still broken

• Restarts loop. If the pod crashes again within seconds of every restart, the trigger is deterministic, usually a model that does not fit at your --max-model-len and memory utilization. Lower the context length or the utilization, or move to a bigger GPU.
• Crashes only under load. That points at concurrent long-context requests exhausting the KV cache. Reduce max concurrency at the gateway, lower --max-model-len, or add replicas so the batch per engine is smaller.
• No OOM in the logs. If the cause above the first error is not memory, capture the full traceback and match it against the vLLM issue tracker for your version. A specific malformed input or an unsupported sampling parameter can trip a different code path.
• You are shipping this to real users. Before it goes wider, decide what the client experiences when an engine is briefly dead: a clear 503 with retry-after and a fallback, not a hung request. Thinking through failure behaviour before launch is the whole point of evaluating an LLM feature before it touches a customer, and it applies to infrastructure failures just as much as to bad generations.