vLLM was never launched able to accept tool calls (v0.3.92)

Reported from production:

    500 {"detail": "Generation error: the pod answered 500: Generation error:
         400 Bad Request … \"auto\" tool choice requires
         --enable-auto-tool-choice and --tool-call-parser to be set"}

Not a regression: `tools` has never worked on a vLLM-backed model here. The
proxy forwards them correctly — the remote decides what it supports — and
nothing ever launched vLLM able to take them. A request-time 400 from a server
that started cleanly, for a capability the catalogue advertises, naming two
flags the caller has no way to set.

A tool call is TEXT the model emits in its own trained format, so the parser
belongs to the model FAMILY rather than to the deployment, which is what makes
a default possible at all. `tool_call_parser` defaults to "auto" and picks from
the model name (qwen->hermes, llama-3->llama3_json, mistral, deepseek_v3,
glm45, qwen3_coder…), longest match first so qwen3-coder is not swallowed by
qwen. An explicit value wins, per install or per model in its `vllm` block —
and that block now travels to a pod, so a pod serves a model the way this
install does.

"off" is honoured rather than overridden: a family guessed wrongly does not
fail, it parses tool calls into nonsense, and silently producing wrong tool
ARGUMENTS is worse than refusing the request.

Both flags go together or neither: vLLM refuses to START with
--enable-auto-tool-choice and no parser, and an engine that will not boot is
worse than one that cannot do tool calls. An unrecognised family logs which
setting to use, because otherwise the symptom lands on the client as a 400.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.91"
+__version__ = "0.3.92"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/runpod_worker.py

@@ -1665,6 +1665,10 @@ def seed_model_env(entry: dict, served: str = "", source: str = "",
             # to load a 154 GB GGUF with transformers. Other backend pins are
             # stripped below: vulkan/opencl say something about THIS machine.
             "backend", "ds4", "colibri", "k3",
+            # …and the vLLM block, for the same reason: tool-call syntax,
+            # context and memory shape describe the MODEL, and a pod that does
+            # not get them serves it differently from this install.
+            "vllm",
             "flash_attention", "model_template", "acceleration", "component_quantization",
             "languages", "supports_translation", "parser", "max_instances",
             # LoRA settings travel too: a coderai pod applies them itself, and

codai/api/vllm_worker.py

@@ -144,6 +144,64 @@ def service_key_for(cfg, model_path=None, served_name=None):
     return key
 
 
+#: Model family -> the vLLM parser that reads its tool-call syntax. A tool call
+#: is TEXT the model emits in its own trained format, so the parser belongs to
+#: the family, not to the deployment. Longest match wins, so `qwen3-coder`
+#: beats `qwen`.
+_TOOL_PARSERS = (
+    ("qwen3-coder", "qwen3_coder"), ("qwen3_coder", "qwen3_coder"),
+    ("deepseek-v3", "deepseek_v3"), ("deepseek_v3", "deepseek_v3"),
+    ("llama-3", "llama3_json"), ("llama3", "llama3_json"),
+    ("mistral", "mistral"), ("ministral", "mistral"),
+    ("granite", "granite"), ("internlm", "internlm"), ("jamba", "jamba"),
+    ("glm-4.5", "glm45"), ("glm-4.6", "glm45"), ("glm45", "glm45"),
+    # Qwen 2.5/3 and the many Hermes-style finetunes emit <tool_call> JSON.
+    ("qwen", "hermes"), ("hermes", "hermes"),
+)
+
+
+def _tool_parser_for(cfg, model_config: dict, model_path: str) -> str:
+    """Which vLLM tool-call parser to launch with, or '' for none.
+
+    An explicit setting always wins — including "off", because a model whose
+    family is guessed wrongly parses tool calls into nonsense, and silently
+    producing wrong tool arguments is worse than refusing the request.
+    """
+    want = str(getattr(cfg, "tool_call_parser", "") or "").strip()
+    if want and want.lower() not in ("auto", "detect"):
+        return "" if want.lower() in ("off", "none", "false", "0") else want
+    hay = " ".join(str(x or "").lower() for x in (
+        model_path, (model_config or {}).get("path"),
+        (model_config or {}).get("id"), (model_config or {}).get("alias")))
+    best = ""
+    for needle, parser in _TOOL_PARSERS:
+        if needle in hay and len(needle) > len(best):
+            best, found = needle, parser
+    return dict(_TOOL_PARSERS).get(best, "") if best else ""
+
+
+def _tool_args(cfg, model_config: dict, model_path: str) -> list:
+    """`--enable-auto-tool-choice --tool-call-parser X`, when we know X.
+
+    Both or neither: vLLM refuses to START with --enable-auto-tool-choice and
+    no parser, and a pod that will not boot is worse than one that cannot do
+    tool calls. When the family is unrecognised this says so, because the
+    symptom otherwise lands on the CLIENT as a 400 at request time.
+    """
+    parser = _tool_parser_for(cfg, model_config, model_path)
+    if not parser:
+        want = str(getattr(cfg, "tool_call_parser", "") or "").strip().lower()
+        if want not in ("off", "none", "false", "0"):
+            print(f"[vllm] no tool-call parser known for {model_path!r} — "
+                  f"requests carrying `tools` will be refused by vLLM. Set "
+                  f"vllm.tool_call_parser (or the model's vllm block) to one "
+                  f"of: {', '.join(sorted(set(p for _, p in _TOOL_PARSERS)))}",
+                  flush=True)
+        return []
+    print(f"[vllm] tool calls parsed as {parser!r}", flush=True)
+    return ["--enable-auto-tool-choice", "--tool-call-parser", parser]
+
+
 def _launch_cmd(py, cfg, host: str, port: int, model_path: str,
                 served_name: Optional[str] = None, model_config: dict = None,
                 gpu_memory_utilization: Optional[float] = None,
@@ -199,6 +257,9 @@ def _launch_cmd(py, cfg, host: str, port: int, model_path: str,
     # a HuggingFace repo id. A QLoRA adapter is an ordinary LoRA here — the
     # quantisation is how the base model loads, which --quantization covers.
     cmd += _lora_args(model_config)
+    # Tool calling. vLLM answers 400 to any request carrying `tools` unless it
+    # was launched for it, and the message names flags the caller cannot set.
+    cmd += _tool_args(cfg, model_config, model_path)
 
     extra = (getattr(cfg, "extra_args", "") or "").strip()
     if extra:

codai/backends/overrides.py

@@ -21,6 +21,10 @@ VLLM_FIELDS = ("ctx", "gpu", "gpu_memory_utilization", "tensor_parallel_size",
                "pipeline_parallel_size", "distributed_executor_backend", "ray_address",
                "ray_port", "nodes", "nodes_ready_timeout_s", "max_num_seqs", "dtype",
                "quantization", "extra_args", "extra_env",
+               # Which parser reads this model's tool-call syntax. Per model,
+               # because the syntax belongs to the model family and one install
+               # serves several.
+               "tool_call_parser",
                # Per model, so the one checkpoint with a byte-level vocabulary can
                # ask for token ids without changing what every other vLLM model
                # reports (see VllmConfig.return_token_ids).

codai/config.py

@@ -775,6 +775,15 @@ class VllmConfig:
     max_num_seqs: int = 0                # --max-num-seqs (0 = vLLM default)
     dtype: str = ""                      # --dtype (blank = auto; e.g. bfloat16/float16)
     quantization: str = ""               # --quantization (blank = none; e.g. awq/gptq/fp8)
+    # Tool calling. vLLM parses a model's tool-call syntax with a named parser
+    # and REFUSES a request carrying `tools` unless it was launched with one:
+    #     "auto" tool choice requires --enable-auto-tool-choice and
+    #      --tool-call-parser to be set
+    # -- a 400 at request time, from a server that started cleanly, for a
+    # feature the catalogue says the model has. The syntax belongs to the model
+    # family, so "auto" picks it from the model name and an explicit value
+    # (here, or per model in its `vllm` block) wins. "off" disables it.
+    tool_call_parser: str = "auto"
     extra_args: str = ""                 # extra flags for the api_server
     extra_env: str = ""                  # free-form KEY=VALUE env for the subprocess
     auto_build: bool = False             # create the isolated venv + pip install vllm if missing

docs/vllm.md

@@ -73,3 +73,54 @@ large corpus, batch classification/extraction, high-QPS chat.
   not a VLM — see the OCR docs.)
 - Selection stays pin/alias-only to avoid collisions with ds4/colibri/k3/kt, exactly
   like ktransformers.
+
+## Tool calling
+
+vLLM parses a model's tool calls with a named parser, and **refuses any request
+carrying `tools` unless it was launched with one**:
+
+```
+400 "auto" tool choice requires --enable-auto-tool-choice and
+    --tool-call-parser to be set
+```
+
+That is a request-time error from a server that started cleanly, for a
+capability the catalogue says the model has, naming flags the caller cannot
+set. So coderai sets them.
+
+A tool call is TEXT the model emits in its own trained format, so the parser
+belongs to the model FAMILY rather than to the deployment. `tool_call_parser`
+defaults to `auto`, which picks from the model name:
+
+| family | parser |
+|---|---|
+| Qwen 2.5 / 3, Hermes finetunes | `hermes` |
+| Qwen3-Coder | `qwen3_coder` |
+| Llama 3.x | `llama3_json` |
+| Mistral / Ministral | `mistral` |
+| DeepSeek-V3 | `deepseek_v3` |
+| GLM-4.5 / 4.6 | `glm45` |
+| Granite, InternLM, Jamba | `granite`, `internlm`, `jamba` |
+
+Longest match wins, so `qwen3-coder` is not swallowed by `qwen`.
+
+Set it explicitly — globally, or per model in its `vllm` block — for anything
+not on that list:
+
+```json
+{"id": "my-model", "backend": "vllm",
+ "vllm": {"tool_call_parser": "llama3_json"}}
+```
+
+`off` disables tool calling for that model. Use it rather than letting a wrong
+guess stand: a mismatched parser does not fail, it parses tool calls into
+nonsense, and silently producing wrong tool ARGUMENTS is worse than refusing
+the request.
+
+Both flags are passed together or not at all — vLLM refuses to **start** with
+`--enable-auto-tool-choice` and no parser, and an engine that will not boot is
+worse than one that cannot do tool calls. When the family is unrecognised the
+log says what to set, because otherwise the symptom lands on the client.
+
+The `vllm` block travels to a RunPod pod with the model, so a pod serves it the
+same way this install does.