prefer the drafter that does not switch async scheduling off

vLLM enables async scheduling -- CPU scheduling overlapped with GPU execution --
by default, and lists plain "ngram" among the methods it cannot run with, so
asking for ngram silently turns it off: the draft is paid for with a throughput
loss somewhere else, and the log says so in a warning nobody reads twice.
"ngram_gpu" drafts by the same prompt-lookup, needs no model and no VRAM either,
and keeps async scheduling. Both spellings work; the field now suggests the one
that does not take something away.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66

codai/admin/templates/models.html

@@ -696,8 +696,8 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
               <option value="off">off</option>
             </select>
             <span class="form-hint" style="font-size:11px">Reuses the KV of a prompt prefix that was computed before, which turns a whole prefill into almost nothing when the same document is asked about again and again. vLLM's own default is on for ordinary models but <b>off for hybrid-attention ones</b>, where it is supported yet still opt-in upstream — so for those this switch is the only way to get it. Worth re-running a retrieval check after turning it on.</span></div>
-          <div class="form-row" style="margin:0"><label class="form-label">Speculative decoding</label><input type="text" id="cfg-vllm-spec" class="form-input" placeholder="off — or ngram, or JSON">
-            <span class="form-hint" style="font-size:11px"><code>ngram</code> needs no draft model and no VRAM: it guesses the next tokens by finding the current text earlier in the prompt. That pays off precisely when the answer quotes its source — extraction, citation, summarising a document — and does nothing for free-form prose. Anything else is passed to vLLM as <code>--speculative-config</code> JSON, so a real draft model can be configured here too.</span></div>
+          <div class="form-row" style="margin:0"><label class="form-label">Speculative decoding</label><input type="text" id="cfg-vllm-spec" class="form-input" placeholder="off — or ngram_gpu, or JSON">
+            <span class="form-hint" style="font-size:11px"><code>ngram_gpu</code> needs no draft model and no VRAM: it guesses the next tokens by finding the current text earlier in the prompt. That pays off precisely when the answer quotes its source — extraction, citation, summarising a document — and does nothing for free-form prose. Prefer it over plain <code>ngram</code>, which vLLM lists as incompatible with async scheduling and so turns that off, paying for the draft with a throughput loss elsewhere. Anything else is passed to vLLM as <code>--speculative-config</code> JSON, so a real draft model can be configured here too.</span></div>
           <div class="form-row" style="margin:0"><label class="form-label">Draft tokens <span class="muted">(ngram)</span></label><input type="number" id="cfg-vllm-spec-tokens" class="form-input" min="1" max="32" placeholder="5"></div>
         </div>
       </div>

codai/api/vllm_worker.py

@@ -298,6 +298,13 @@ def _reasoning_args(cfg, model_config: dict, model_path: str) -> list:
 #: tokens the model was never going to accept.
 _NGRAM_DRAFT_TOKENS = 5
 
+#: Both drafters that need no model, no second checkpoint and no VRAM. Prefer
+#: "ngram_gpu": vLLM enables async scheduling (CPU scheduling overlapped with
+#: GPU execution) by default, and plain "ngram" is on its incompatible list, so
+#: choosing it turns async scheduling OFF -- paying for the draft with a
+#: throughput loss elsewhere. "ngram_gpu" drafts the same way and keeps it.
+_NGRAM_METHODS = ("ngram", "ngram_gpu")
+
 
 def _speculative_args(cfg) -> list:
     """``--speculative-config`` for the configured mode, or nothing.
@@ -311,9 +318,9 @@ def _speculative_args(cfg) -> list:
     spec = (getattr(cfg, "speculative", "") or "").strip()
     if not spec:
         return []
-    if spec.lower() == "ngram":
+    if spec.lower() in _NGRAM_METHODS:
         n = int(getattr(cfg, "speculative_tokens", 0) or 0) or _NGRAM_DRAFT_TOKENS
-        body = {"method": "ngram", "num_speculative_tokens": n}
+        body = {"method": spec.lower(), "num_speculative_tokens": n}
     else:
         try:
             body = json.loads(spec)

tests/test_vllm_speed_levers_per_model.py

@@ -102,10 +102,17 @@ def test_prefix_caching_off_is_forced_explicitly():
     assert "--enable-prefix-caching" not in cmd
 
 
-def test_ngram_needs_no_draft_model_and_no_json_from_the_operator():
-    cmd = _cmd(speculative="ngram")
+@pytest.mark.parametrize("method", ["ngram", "ngram_gpu"])
+def test_ngram_needs_no_draft_model_and_no_json_from_the_operator(method):
+    """Both spellings are drafters that need nothing else configured.
+
+    ngram_gpu is the one to prefer: vLLM turns async scheduling on by default
+    and names plain "ngram" among the things it is incompatible with, so asking
+    for that one pays for the draft with a throughput loss elsewhere.
+    """
+    cmd = _cmd(speculative=method)
     body = json.loads(_flag_value(cmd, "--speculative-config"))
-    assert body["method"] == "ngram"
+    assert body["method"] == method
     assert body["num_speculative_tokens"] >= 1