tune how a model is served, per model, from the model page (v0.4.84)

Decode on one GPU is bound by how fast the weights can be read: measured at
30 tok/s against a ~28-30 tok/s bandwidth ceiling, it is already at the wall and
no setting moves it. Prefill is not -- it is compute-bound, and on a long prompt
it is nearly all of the request. So the levers that matter are the ones that
avoid work rather than hurry it, and each of them now has a field of its own.

  Prefill batch (--max-num-batched-tokens). vLLM's default is 2048, so a
  60k-token prompt is prefilled in ~30 sequential passes of small GEMMs that
  leave the tensor cores idle. The cost of raising it is a higher activation
  peak, taken out of the KV cache, so the field says to check the boot report
  afterwards rather than implying it is free.

  Prefix caching (--enable-prefix-caching), as a tri-state. vLLM's own default
  is per model -- on for an ordinary decoder, off for a hybrid-attention one,
  where it is supported but still opt-in upstream -- so "leave it alone" has to
  be expressible and has to be the default. For those models this switch is the
  only way to get it, and where the same document is asked about repeatedly it
  turns a whole prefill into almost nothing.

  Speculative decoding (--speculative-config). "ngram" is spelled out as a word
  because it needs no draft model, no second checkpoint and no VRAM: it drafts
  by finding the current text earlier in the prompt, which pays off precisely on
  answers that quote their source. Anything else is passed through as the JSON
  vLLM documents, so a real draft model needs no code change.

  Quantization and dtype, which were accepted by the save route but had no field
  anywhere. The hint says to leave quantization blank: it DECLARES how the
  weights on disk are packed and converts nothing, vLLM reads that from the
  checkpoint and picks its fastest kernel, and a repo whose name says AWQ can
  still be compressed-tensors -- so naming a format can only lose speed or stop
  the engine booting.

  Thinking, per model. A thinking model generates its thought first and charges
  it against max_tokens, so on extraction work it is most of the latency and
  none of the answer; on a tight budget it spends the whole allowance before the
  answer starts and returns an empty one. Until now only a request could turn it
  off, which an install does not control -- its clients do. Three states, since
  unset must never be read as "do not think" for a model whose answers depend on
  it, and an explicit request value still wins either way.

The vLLM card is also shown for a model served by a vLLM POD, which is where
these settings matter most, and that fixes a silent data loss: the card was
hidden for `backend: runpod`, so the collector sent `vllm: {}` and the save route
read an empty block as "this model has no vLLM settings" and deleted the stored
one. Merely opening such a model and saving it dropped its block -- kv-cache
dtype and the cache floors with it -- with no error, and nothing changed until
the next pod booted in a different precision. An engine block is now sent only
when its card was on screen to be edited; otherwise the key is left out, which
the route ignores. The regression test fails on the previous code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.83"
+__version__ = "0.4.84"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -2940,6 +2940,11 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                 "auto_compact_tail_tokens", "auto_compact_tolerance_pct",
                 "auto_compact_min_output", "auto_compact_retry_on_overflow",
                 "suppress_reasoning",
+                # Whether this model thinks when the request does not say.
+                # "auto" leaves the model's own behaviour; "off" is for a model
+                # served for extraction, where the thought is charged against
+                # max_tokens, emitted before the answer, and read by nobody.
+                "thinking",
                 # Speech-to-text (wav2vec2 / vosk / NeMo-Canary) options.
                 "languages", "supports_translation", "keep_resident",
                 # "local" pins the model here even when its whole capability is
@@ -3009,7 +3014,9 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                                      "distributed_executor_backend", "ray_address",
                                      "ray_port", "nodes", "nodes_ready_timeout_s",
                                      "gpu", "gpu_memory_utilization", "max_num_seqs",
-                                     "dtype", "quantization", "extra_args", "extra_env")),
+                                     "dtype", "quantization", "extra_args", "extra_env",
+                                     "max_num_batched_tokens", "prefix_caching",
+                                     "speculative", "speculative_tokens")),
                           ("kt", ("nnodes", "dist_init_addr", "tp_size", "nodes",
                                    "kt_weight_path", "extra_args", "extra_env"))):
         if _blk not in data:

codai/api/runpod_worker.py

@@ -1121,6 +1121,11 @@ _ENGINE_FORWARDED_FIELDS = {
     "kt": ("ctx", "extra_args", "extra_env", "model_id"),
     "vllm": ("ctx", "gpu_memory_utilization", "tensor_parallel_size", "max_num_seqs",
              "dtype", "quantization", "extra_args", "extra_env",
+             # The speed levers describe how to RUN the model, so they belong on
+             # the pod that runs it -- a prefill budget tuned for an A40 is
+             # useless if it stays on the orchestrator.
+             "max_num_batched_tokens", "prefix_caching",
+             "speculative", "speculative_tokens",
              # Only ever forwarded when TRUE: the loop below skips False, so
              # adding it here cannot change an install that has not asked for it.
              # A pod needs it to recover a byte-level vocabulary's characters from
@@ -1792,6 +1797,10 @@ def seed_model_env(entry: dict, served: str = "", source: str = "",
             # context and memory shape describe the MODEL, and a pod that does
             # not get them serves it differently from this install.
             "vllm",
+            # Whether the model thinks by default. A pod leased to a client
+            # answers its requests directly, so a model configured not to think
+            # must not start thinking just because it moved.
+            "thinking",
             "flash_attention", "model_template", "acceleration", "component_quantization",
             "languages", "supports_translation", "parser", "max_instances",
             # LoRA settings travel too: a coderai pod applies them itself, and

codai/api/text.py

@@ -1654,6 +1654,15 @@ async def chat_completions(request: ChatCompletionRequest, http_request: Request
     # thinking model thinks by default, and only an explicit false asks it to
     # stop. The two were indistinguishable while this field defaulted to False.
     enable_thinking_req = getattr(request, 'enable_thinking', None)
+    # Still nothing from the caller? Then the MODEL's own default decides. A
+    # thinking model charges its thought against max_tokens and emits it first,
+    # so on extraction and retrieval work the thought is most of the latency and
+    # none of the answer -- long enough, on a long prompt, to spend the whole
+    # budget before the answer starts. An install that serves such a model for
+    # that kind of work needs to be able to say so once, here, instead of
+    # relying on every client to pass enable_thinking=false.
+    if enable_thinking_req is None:
+        enable_thinking_req = _model_thinking_default(getattr(request, 'model', None) or "")
     enable_thinking_api = bool(enable_thinking_req)
     
     # Parse force_reasoning: can be list (from CLI) or string (legacy)
@@ -2618,7 +2627,7 @@ async def chat_completions(request: ChatCompletionRequest, http_request: Request
                     frequency_penalty=request.frequency_penalty,
                     logprobs=request.logprobs,
                     top_logprobs=request.top_logprobs,
-                    passthrough=_chat_passthrough(request),
+                    passthrough=_chat_passthrough(request, enable_thinking_req),
                 ):
                     yield chunk
             finally:
@@ -2649,7 +2658,7 @@ async def chat_completions(request: ChatCompletionRequest, http_request: Request
                     frequency_penalty=request.frequency_penalty,
                     logprobs=request.logprobs,
                     top_logprobs=request.top_logprobs,
-                    passthrough=_chat_passthrough(request),
+                    passthrough=_chat_passthrough(request, enable_thinking_req),
                 )
 
             try:
@@ -2709,7 +2718,38 @@ _CHAT_CONSUMED_FIELDS = frozenset({
 })
 
 
-def _chat_passthrough(request) -> Optional[Dict]:
+#: Per-model thinking defaults, as the model page spells them.
+_THINKING_ON = ("on", "true", "1", "yes", "always")
+_THINKING_OFF = ("off", "false", "0", "no", "never")
+
+
+def _model_thinking_default(model_name: str):
+    """Whether this model should think when the request does not say, or None.
+
+    None means "leave the model's own default", which for a thinking model means
+    it thinks -- deliberately distinct from False, which asks it not to. Only a
+    configured value speaks here; an unset one must not turn thinking off for a
+    model whose answers depend on it.
+    """
+    if not model_name:
+        return None
+    try:
+        from codai.models.manager import multi_model_manager as _mmm
+        cfg = _mmm._config_for_model(model_name) or {}
+    except Exception:
+        return None
+    raw = cfg.get("thinking")
+    if isinstance(raw, bool):
+        return raw
+    val = str(raw or "").strip().lower()
+    if val in _THINKING_ON:
+        return True
+    if val in _THINKING_OFF:
+        return False
+    return None
+
+
+def _chat_passthrough(request, want_think=None) -> Optional[Dict]:
     """The request fields coderai does not consume, for the backend to forward.
 
     Only reaches a backend whose signature accepts it (see
@@ -2726,7 +2766,11 @@ def _chat_passthrough(request) -> Optional[Dict]:
         # model's own default, which for a thinking model means it thinks.
         # Without this, a request could not turn thinking off on a served model
         # at all: coderai consumed the field and no backend forwarded it.
-        want_think = getattr(request, "enable_thinking", None)
+        # The caller's own value, else the one the model was configured with —
+        # already resolved by the caller, so the request path and this one cannot
+        # disagree about whether the model is meant to think.
+        if want_think is None:
+            want_think = getattr(request, "enable_thinking", None)
         if want_think is not None and "chat_template_kwargs" not in out:
             out["chat_template_kwargs"] = {"enable_thinking": bool(want_think)}
         return out or None

codai/api/vllm_worker.py

@@ -293,6 +293,42 @@ def _reasoning_args(cfg, model_config: dict, model_path: str) -> list:
     return ["--reasoning-parser", parser]
 
 
+#: Draft length for "ngram" when none was configured. Five is vLLM's own choice
+#: for the lookup window; drafting much further ahead wastes the verify pass on
+#: tokens the model was never going to accept.
+_NGRAM_DRAFT_TOKENS = 5
+
+
+def _speculative_args(cfg) -> list:
+    """``--speculative-config`` for the configured mode, or nothing.
+
+    "ngram" is spelled out here rather than left to the operator as JSON because
+    it is the mode that needs no draft model, no second checkpoint and no VRAM,
+    and so is the only one that can be turned on from a dropdown. Any other value
+    is taken as the JSON vLLM itself documents, which keeps a real draft model
+    reachable without waiting for a code change.
+    """
+    spec = (getattr(cfg, "speculative", "") or "").strip()
+    if not spec:
+        return []
+    if spec.lower() == "ngram":
+        n = int(getattr(cfg, "speculative_tokens", 0) or 0) or _NGRAM_DRAFT_TOKENS
+        body = {"method": "ngram", "num_speculative_tokens": n}
+    else:
+        try:
+            body = json.loads(spec)
+        except Exception as exc:
+            # A typo here would otherwise stop the engine booting with an
+            # argparse error about JSON, naming neither the model nor the field.
+            print(f"[vllm] speculative config ignored, not valid JSON: {exc}", flush=True)
+            return []
+        if not isinstance(body, dict):
+            print(f"[vllm] speculative config ignored, not an object: {spec!r}", flush=True)
+            return []
+    print(f"[vllm] speculative decoding: {body}", flush=True)
+    return ["--speculative-config", json.dumps(body)]
+
+
 def _launch_cmd(py, cfg, host: str, port: int, model_path: str,
                 served_name: Optional[str] = None, model_config: dict = None,
                 gpu_memory_utilization: Optional[float] = None,
@@ -320,7 +356,10 @@ def _launch_cmd(py, cfg, host: str, port: int, model_path: str,
     # The profiling peak that decides whether the engine can start is driven by how many
     # tokens may be in one batch, NOT by max_model_len (chunked prefill is on), so this
     # is the lever that shrinks the footprint without shrinking the usable context.
-    mnbt = int(max_num_batched_tokens or 0)
+    # A caller's figure (a side job sizing itself down) wins; otherwise the
+    # configured one, which is how an install raises the budget to speed up long
+    # prefills instead of only ever lowering it to fit.
+    mnbt = int(max_num_batched_tokens or 0) or int(getattr(cfg, "max_num_batched_tokens", 0) or 0)
     if mnbt > 0:
         cmd += ["--max-num-batched-tokens", str(mnbt)]
     tp = int(getattr(cfg, "tensor_parallel_size", 0) or 0)
@@ -343,6 +382,16 @@ def _launch_cmd(py, cfg, host: str, port: int, model_path: str,
     quant = (getattr(cfg, "quantization", "") or "").strip()
     if quant:
         cmd += ["--quantization", quant]
+    # Prefix caching. Only ever passed when the operator asked for a value:
+    # vLLM's own default is per MODEL (on for a plain decoder, off for a hybrid
+    # one, where it is supported but kept opt-in upstream), and passing a flag
+    # unasked would override that judgement for every model at once.
+    pc = (getattr(cfg, "prefix_caching", "") or "auto").strip().lower()
+    if pc in ("on", "true", "1", "yes"):
+        cmd += ["--enable-prefix-caching"]
+    elif pc in ("off", "false", "0", "no"):
+        cmd += ["--no-enable-prefix-caching"]
+    cmd += _speculative_args(cfg)
     # LoRA adapters configured on the MODEL (not on the vLLM backend): vLLM takes
     # them at launch as name=source pairs, where source is a path on this host or
     # a HuggingFace repo id. A QLoRA adapter is an ordinary LoRA here — the

codai/backends/overrides.py

@@ -25,6 +25,13 @@ VLLM_FIELDS = ("ctx", "gpu", "gpu_memory_utilization",
                "pipeline_parallel_size", "distributed_executor_backend", "ray_address",
                "ray_port", "nodes", "nodes_ready_timeout_s", "max_num_seqs", "dtype",
                "quantization", "extra_args", "extra_env",
+               # Speed levers, per model because they trade against each other
+               # differently for each one: a bigger prefill batch buys
+               # time-to-first-token out of the KV cache, prefix caching pays
+               # only where prompts repeat, and ngram drafting pays only where
+               # the answer quotes the prompt.
+               "max_num_batched_tokens", "prefix_caching",
+               "speculative", "speculative_tokens",
                # Which parser reads this model's tool-call syntax. Per model,
                # because the syntax belongs to the model family and one install
                # serves several.

codai/config.py

@@ -813,7 +813,39 @@ class VllmConfig:
     tensor_parallel_size: int = 1        # --tensor-parallel-size
     max_num_seqs: int = 0                # --max-num-seqs (0 = vLLM default)
     dtype: str = ""                      # --dtype (blank = auto; e.g. bfloat16/float16)
-    quantization: str = ""               # --quantization (blank = none; e.g. awq/gptq/fp8)
+    # --quantization. Blank means AUTO-DETECT, which is almost always what you
+    # want: vLLM reads the checkpoint's own quantization_config and picks the
+    # fastest kernel it has for it (a compressed-tensors W4A16 checkpoint gets
+    # Marlin). This field DECLARES how the weights on disk are packed -- it does
+    # not convert anything -- so naming the wrong format moves the engine onto a
+    # slower kernel or stops it booting. A checkpoint whose name says AWQ can
+    # still be compressed-tensors; believe the config, not the repo name.
+    quantization: str = ""
+    # How many tokens may be prefilled in ONE engine step (--max-num-batched-tokens).
+    # vLLM's own default is 2048, so a 60k-token prompt is prefilled as ~30
+    # sequential chunks of small, tensor-core-starved GEMMs. Raising it fills the
+    # cards better and cuts time-to-first-token on long prompts; the cost is a
+    # higher activation peak at profiling, which is taken out of the KV cache --
+    # so raise it watching `Available KV cache memory` and the concurrency the
+    # boot report prints. 0 = leave vLLM's default.
+    max_num_batched_tokens: int = 0
+    # Reuse the KV of a prompt prefix already computed (--enable-prefix-caching).
+    # "auto" leaves vLLM's per-model default, which is ON for ordinary decoder
+    # models but OFF for hybrid-attention ones (supported there, but opt-in while
+    # the feature matures upstream). "on" forces it, which is the one setting that
+    # can take a repeated-document workload -- the same file asked about again and
+    # again -- from a full prefill per question to almost none. "off" forces it off.
+    prefix_caching: str = "auto"         # auto | on | off
+    # Speculative decoding (--speculative-config). Decode on one GPU is bound by
+    # weight bandwidth, not arithmetic, so drafting several tokens and checking
+    # them in one pass is the only way to beat that ceiling. "ngram" needs no
+    # draft model and no VRAM: it drafts by finding the current suffix earlier in
+    # the prompt, which is why it pays off exactly on answers that quote their
+    # source (extraction, citation, summarising a document) and does nothing for
+    # free-form prose. Anything else is passed through as raw JSON, so a draft
+    # model can be configured without a code change.
+    speculative: str = ""                # "" | "ngram" | raw JSON for --speculative-config
+    speculative_tokens: int = 0          # draft length for "ngram" (0 = built-in default)
     # Tool calling. vLLM parses a model's tool-call syntax with a named parser
     # and REFUSES a request carrying `tools` unless it was launched with one:
     #     "auto" tool choice requires --enable-auto-tool-choice and

tests/test_model_thinking_default.py

+"""A model can be configured not to think, without every client saying so.
+
+A thinking model generates its thought FIRST and charges it against max_tokens.
+On extraction and retrieval work that thought is most of the latency and none of
+the answer, and on a tight budget it is worse than slow: the allowance runs out
+inside the thought and the answer comes back empty, with the request reporting
+that it merely stopped. Measured on a real one -- a 2,100-token budget, ~6,000
+characters of thought, no visible answer at all.
+
+Until now the only way to turn it off was `enable_thinking: false` on every
+request, which an install does not control: its clients do. So the model itself
+carries a default.
+
+Three states, not two. "auto" has to mean "leave the model's own behaviour" and
+has to be what an unconfigured model gets, because for a reasoning model the
+thought is the answer's quality -- an unset field must never be read as "off".
+An explicit request value always wins over the configured default, in both
+directions: that is what makes the setting a default rather than a policy.
+"""
+import pytest
+
+
+class _Req:
+    """Only what the code under test reads off a request."""
+
+    def __init__(self, model="m", enable_thinking=None, extra=None):
+        self.model = model
+        self.enable_thinking = enable_thinking
+        self.model_extra = extra or {}
+
+
+@pytest.fixture
+def configured(monkeypatch):
+    """Point the per-model lookup at a config this test chooses."""
+    def _set(cfg):
+        from codai.models import manager as mgr
+        monkeypatch.setattr(mgr.multi_model_manager, "_config_for_model",
+                            lambda name: dict(cfg), raising=False)
+    return _set
+
+
+def _default(name="m"):
+    from codai.api.text import _model_thinking_default
+    return _model_thinking_default(name)
+
+
+# -- resolving the configured default ------------------------------------
+
+def test_an_unconfigured_model_keeps_its_own_behaviour(configured):
+    configured({})
+    assert _default() is None, "unset must not be read as 'do not think'"
+
+
+def test_auto_keeps_the_models_own_behaviour(configured):
+    configured({"thinking": "auto"})
+    assert _default() is None
+
+
+@pytest.mark.parametrize("value", ["off", "false", "no", "never", "0", False])
+def test_off_asks_the_model_not_to_think(configured, value):
+    configured({"thinking": value})
+    assert _default() is False
+
+
+@pytest.mark.parametrize("value", ["on", "true", "yes", "always", "1", True])
+def test_on_asks_the_model_to_think(configured, value):
+    configured({"thinking": value})
+    assert _default() is True
+
+
+def test_an_unknown_value_is_not_guessed(configured):
+    configured({"thinking": "sometimes"})
+    assert _default() is None
+
+
+def test_no_model_name_consults_nothing(configured):
+    configured({"thinking": "off"})
+    assert _default("") is None
+
+
+def test_a_broken_config_lookup_leaves_the_model_alone(monkeypatch):
+    from codai.models import manager as mgr
+
+    def _boom(name):
+        raise RuntimeError("models.json is being rewritten")
+
+    monkeypatch.setattr(mgr.multi_model_manager, "_config_for_model", _boom,
+                        raising=False)
+    assert _default() is None
+
+
+# -- what reaches the engine ---------------------------------------------
+
+def _kwargs(request, want_think):
+    from codai.api.text import _chat_passthrough
+    return (_chat_passthrough(request, want_think) or {}).get("chat_template_kwargs")
+
+
+def test_the_default_travels_as_a_template_argument():
+    """Only the chat template can decide whether the model thinks."""
+    assert _kwargs(_Req(), False) == {"enable_thinking": False}
+
+
+def test_thinking_on_travels_too():
+    assert _kwargs(_Req(), True) == {"enable_thinking": True}
+
+
+def test_nothing_configured_sends_nothing():
+    assert _kwargs(_Req(), None) is None
+
+
+def test_an_explicit_request_value_wins_over_the_default():
+    """A client asking to think must get it, whatever the model is set to."""
+    assert _kwargs(_Req(enable_thinking=True), True) == {"enable_thinking": True}
+    assert _kwargs(_Req(enable_thinking=False), False) == {"enable_thinking": False}
+
+
+def test_a_callers_own_template_kwargs_are_not_overwritten():
+    req = _Req(extra={"chat_template_kwargs": {"enable_thinking": True}})
+    assert _kwargs(req, False) == {"enable_thinking": True}
+
+
+def test_enable_thinking_is_not_forwarded_as_a_bare_field():
+    """vLLM takes it as a template argument; as a top-level field it is a 400."""
+    from codai.api.text import _chat_passthrough
+    out = _chat_passthrough(_Req(enable_thinking=False), False) or {}
+    assert "enable_thinking" not in out