tune how a model is served, per model, from the model page (v0.4.84)
Decode on one GPU is bound by how fast the weights can be read: measured at
30 tok/s against a ~28-30 tok/s bandwidth ceiling, it is already at the wall and
no setting moves it. Prefill is not -- it is compute-bound, and on a long prompt
it is nearly all of the request. So the levers that matter are the ones that
avoid work rather than hurry it, and each of them now has a field of its own.
Prefill batch (--max-num-batched-tokens). vLLM's default is 2048, so a
60k-token prompt is prefilled in ~30 sequential passes of small GEMMs that
leave the tensor cores idle. The cost of raising it is a higher activation
peak, taken out of the KV cache, so the field says to check the boot report
afterwards rather than implying it is free.
Prefix caching (--enable-prefix-caching), as a tri-state. vLLM's own default
is per model -- on for an ordinary decoder, off for a hybrid-attention one,
where it is supported but still opt-in upstream -- so "leave it alone" has to
be expressible and has to be the default. For those models this switch is the
only way to get it, and where the same document is asked about repeatedly it
turns a whole prefill into almost nothing.
Speculative decoding (--speculative-config). "ngram" is spelled out as a word
because it needs no draft model, no second checkpoint and no VRAM: it drafts
by finding the current text earlier in the prompt, which pays off precisely on
answers that quote their source. Anything else is passed through as the JSON
vLLM documents, so a real draft model needs no code change.
Quantization and dtype, which were accepted by the save route but had no field
anywhere. The hint says to leave quantization blank: it DECLARES how the
weights on disk are packed and converts nothing, vLLM reads that from the
checkpoint and picks its fastest kernel, and a repo whose name says AWQ can
still be compressed-tensors -- so naming a format can only lose speed or stop
the engine booting.
Thinking, per model. A thinking model generates its thought first and charges
it against max_tokens, so on extraction work it is most of the latency and
none of the answer; on a tight budget it spends the whole allowance before the
answer starts and returns an empty one. Until now only a request could turn it
off, which an install does not control -- its clients do. Three states, since
unset must never be read as "do not think" for a model whose answers depend on
it, and an explicit request value still wins either way.
The vLLM card is also shown for a model served by a vLLM POD, which is where
these settings matter most, and that fixes a silent data loss: the card was
hidden for `backend: runpod`, so the collector sent `vllm: {}` and the save route
read an empty block as "this model has no vLLM settings" and deleted the stored
one. Merely opening such a model and saving it dropped its block -- kv-cache
dtype and the cache floors with it -- with no error, and nothing changed until
the next pod booted in a different precision. An engine block is now sent only
when its card was on screen to be edited; otherwise the key is left out, which
the route ignores. The regression test fails on the previous code.
Co-Authored-By: 
Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
Showing
Please
register
or
sign in
to comment