-
Stefy Lanza (nextime / spora ) authored
Reported from production: 500 {"detail": "Generation error: the pod answered 500: Generation error: 400 Bad Request … \"auto\" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set"} Not a regression: `tools` has never worked on a vLLM-backed model here. The proxy forwards them correctly — the remote decides what it supports — and nothing ever launched vLLM able to take them. A request-time 400 from a server that started cleanly, for a capability the catalogue advertises, naming two flags the caller has no way to set. A tool call is TEXT the model emits in its own trained format, so the parser belongs to the model FAMILY rather than to the deployment, which is what makes a default possible at all. `tool_call_parser` defaults to "auto" and picks from the model name (qwen->hermes, llama-3->llama3_json, mistral, deepseek_v3, glm45, qwen3_coder…), longest match first so qwen3-coder is not swallowed by qwen. An explicit value wins, per install or per model in its `vllm` block — and that block now travels to a pod, so a pod serves a model the way this install does. "off" is honoured rather than overridden: a family guessed wrongly does not fail, it parses tool calls into nonsense, and silently producing wrong tool ARGUMENTS is worse than refusing the request. Both flags go together or neither: vLLM refuses to START with --enable-auto-tool-choice and no parser, and an engine that will not boot is worse than one that cannot do tool calls. An unrecognised family logs which setting to use, because otherwise the symptom lands on the client as a 400. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
7601bdf9