• Stefy Lanza (nextime / spora )'s avatar
    vLLM was never launched able to accept tool calls (v0.3.92) · 7601bdf9
    Stefy Lanza (nextime / spora ) authored
    Reported from production:
    
        500 {"detail": "Generation error: the pod answered 500: Generation error:
             400 Bad Request … \"auto\" tool choice requires
             --enable-auto-tool-choice and --tool-call-parser to be set"}
    
    Not a regression: `tools` has never worked on a vLLM-backed model here. The
    proxy forwards them correctly — the remote decides what it supports — and
    nothing ever launched vLLM able to take them. A request-time 400 from a server
    that started cleanly, for a capability the catalogue advertises, naming two
    flags the caller has no way to set.
    
    A tool call is TEXT the model emits in its own trained format, so the parser
    belongs to the model FAMILY rather than to the deployment, which is what makes
    a default possible at all. `tool_call_parser` defaults to "auto" and picks from
    the model name (qwen->hermes, llama-3->llama3_json, mistral, deepseek_v3,
    glm45, qwen3_coder…), longest match first so qwen3-coder is not swallowed by
    qwen. An explicit value wins, per install or per model in its `vllm` block —
    and that block now travels to a pod, so a pod serves a model the way this
    install does.
    
    "off" is honoured rather than overridden: a family guessed wrongly does not
    fail, it parses tool calls into nonsense, and silently producing wrong tool
    ARGUMENTS is worse than refusing the request.
    
    Both flags go together or neither: vLLM refuses to START with
    --enable-auto-tool-choice and no parser, and an engine that will not boot is
    worse than one that cannot do tool calls. An unrecognised family logs which
    setting to use, because otherwise the symptom lands on the client as a 400.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    7601bdf9
config.py 143 KB