Decode on one GPU is bound by how fast the weights can be read: measured at
30 tok/s against a ~28-30 tok/s bandwidth ceiling, it is already at the wall and
no setting moves it. Prefill is not -- it is compute-bound, and on a long prompt
it is nearly all of the request. So the levers that matter are the ones that
avoid work rather than hurry it, and each of them now has a field of its own.

  Prefill batch (--max-num-batched-tokens). vLLM's default is 2048, so a
  60k-token prompt is prefilled in ~30 sequential passes of small GEMMs that
  leave the tensor cores idle. The cost of raising it is a higher activation
  peak, taken out of the KV cache, so the field says to check the boot report
  afterwards rather than implying it is free.

  Prefix caching (--enable-prefix-caching), as a tri-state. vLLM's own default
  is per model -- on for an ordinary decoder, off for a hybrid-attention one,
  where it is supported but still opt-in upstream -- so "leave it alone" has to
  be expressible and has to be the default. For those models this switch is the
  only way to get it, and where the same document is asked about repeatedly it
  turns a whole prefill into almost nothing.

  Speculative decoding (--speculative-config). "ngram" is spelled out as a word
  because it needs no draft model, no second checkpoint and no VRAM: it drafts
  by finding the current text earlier in the prompt, which pays off precisely on
  answers that quote their source. Anything else is passed through as the JSON
  vLLM documents, so a real draft model needs no code change.

  Quantization and dtype, which were accepted by the save route but had no field
  anywhere. The hint says to leave quantization blank: it DECLARES how the
  weights on disk are packed and converts nothing, vLLM reads that from the
  checkpoint and picks its fastest kernel, and a repo whose name says AWQ can
  still be compressed-tensors -- so naming a format can only lose speed or stop
  the engine booting.

  Thinking, per model. A thinking model generates its thought first and charges
  it against max_tokens, so on extraction work it is most of the latency and
  none of the answer; on a tight budget it spends the whole allowance before the
  answer starts and returns an empty one. Until now only a request could turn it
  off, which an install does not control -- its clients do. Three states, since
  unset must never be read as "do not think" for a model whose answers depend on
  it, and an explicit request value still wins either way.

The vLLM card is also shown for a model served by a vLLM POD, which is where
these settings matter most, and that fixes a silent data loss: the card was
hidden for `backend: runpod`, so the collector sent `vllm: {}` and the save route
read an empty block as "this model has no vLLM settings" and deleted the stored
one. Merely opening such a model and saving it dropped its block -- kv-cache
dtype and the cache floors with it -- with no error, and nothing changed until
the next pod booted in a different precision. An engine block is now sent only
when its card was on screen to be edited; otherwise the key is left out, which
the route ignores. The regression test fails on the previous code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66