• Stefy Lanza (nextime / spora )'s avatar
    let vLLM split thinking from the answer: the tag is a special token (v0.4.65) · ccea5f43
    Stefy Lanza (nextime / spora ) authored
    v0.4.64 aimed one layer too high. Measured against the pod directly:
    
      content           = 'We need answer in Italian... 17 x 23 = ... **391**.'
      reasoning_content = ''
      has </think>      = False          finish = stop
    
    The thought and the answer come back as ONE string with no separator of any
    kind. </think> is a SPECIAL TOKEN and vLLM decodes with skip_special_tokens,
    so the boundary is deleted before the text exists. No parsing downstream can
    split on a boundary that was removed -- the earlier reading, "extraction fails
    on the bare </think>", was simply wrong.
    
    coderai never passed --reasoning-parser to vLLM, for any model, ever. With it
    vLLM splits the two at the TOKEN level, before the tag is discarded, and
    reports reasoning_content as its own field; the v0.4.64 side channel and delta
    re-tagging then carry it to the client. Chosen from the template like the tool
    parser, and only for a model whose template says it thinks -- a reasoning
    parser on a model that does not think waits for a tag that is never coming. An
    explicit vllm.reasoning_parser wins, "off" included, per model or globally.
    
    AND A REGRESSION FROM v0.4.64, fixed here: with the gate switched on and no
    close tag ever arriving, a streamed reply was held back in full as "thinking"
    and the client was sent an empty message -- worse than the leak it replaced
    (reasoning=229 chars, content=0, measured on diogene-alpha and qwen38 alike).
    An unclosed thought now flushes as CONTENT. A thought truly cut off mid
    sentence then reads as the answer, which is untidy; an empty answer is a lost
    reply. With the server separating the channel this path stops being reached.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    ccea5f43
config.py 160 KB