• Stefy Lanza (nextime / spora )'s avatar
    deliver a model's thinking as thinking, while it is thinking (v0.4.64) · 67aa1f5f
    Stefy Lanza (nextime / spora ) authored
    From ONE pod, ONE set of weights, behind two aliases:
    
      qwen38        -> clean answer
      diogene-alpha -> "The user is asking to list hearings from 2026-10-21 to..."
    
    The entire chain of thought went to the client as the answer. Four breaks:
    
    1. THE GATE ASKED THE NAME. _reasoning_active was a regex (qwen3|qwq|
       deepseek-r1|...) over the request's model name: `qwen3` matches "qwen38" and
       not "diogene-alpha", so one alias of one model separated its thought and the
       other did not. It is read from the template now -- whose generation prompt
       ends with '<think>\n', so the model starts INSIDE the thought and emits a
       bare </think>, the form that leaks when nothing looks for it. The regex
       stays as a last resort for a template that cannot be read.
    
    2. THE BUFFERED FORMATTER TOOK NO `reasoning`. Where the thought WAS separated
       it was then dropped: no leak, but no thinking either. litellm's Message has
       no such field, so it is attached after the dump rather than lost -- under
       both spellings, since Qwen's own template reads `reasoning_content`.
    
    3. THE PROXY HOP CARRIED NEITHER FORM. Buffered, the remote sends
       reasoning_content beside an already-cleaned content, which the string
       contract cannot hold: get_last_reasoning(), like get_last_tool_calls().
       Streamed, it sends reasoning DELTAS, and dropping them meant a client saw
       nothing at all until the answer began -- so the <think>/</think> tags go
       back on as they arrive and the caller's existing reasoning gate splits them
       live. An all-thinking stream leaves the tag open on purpose: the gate
       flushes an unclosed thought as reasoning, and there is no content for a
       close tag to precede.
    
    4. THINKING COULD NOT BE TURNED OFF. enable_thinking defaulted to False, so
       "unset" and "explicitly off" were the same value, and the field sat in
       _CHAT_CONSUMED_FIELDS while no backend forwarded it. It defaults to None
       now: unset sends nothing and the model keeps its own default (a thinking
       model thinks), an explicit value travels as chat_template_kwargs. That
       needed the POD backend to gain passthrough at all -- the local vLLM backend
       has forwarded unknown fields for releases, a pod got none of them, so
       chat_template_kwargs could never reach a pod-served model.
    
    ChatMessage silently dropped `reasoning_content`, so a previous turn's thinking
    never reached the template that re-renders it inside <think>. Accepted and
    forwarded now.
    
    The chat page hardcoded stream:false, so thinking could never arrive live
    there, and it read only message.content. It has `stream` and `thinking`
    toggles (both on), renders the thought in a block that fills AS IT ARRIVES and
    collapses when the answer starts, and keeps it in the history it replays.
    Streaming is the request's choice, not forced: cleared, it takes the buffered
    path exactly as before.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    67aa1f5f