• Stefy Lanza (nextime / spora )'s avatar
    deliver a model's thinking as thinking, while it is thinking (v0.4.64) · 67aa1f5f
    Stefy Lanza (nextime / spora ) authored
    From ONE pod, ONE set of weights, behind two aliases:
    
      qwen38        -> clean answer
      diogene-alpha -> "The user is asking to list hearings from 2026-10-21 to..."
    
    The entire chain of thought went to the client as the answer. Four breaks:
    
    1. THE GATE ASKED THE NAME. _reasoning_active was a regex (qwen3|qwq|
       deepseek-r1|...) over the request's model name: `qwen3` matches "qwen38" and
       not "diogene-alpha", so one alias of one model separated its thought and the
       other did not. It is read from the template now -- whose generation prompt
       ends with '<think>\n', so the model starts INSIDE the thought and emits a
       bare </think>, the form that leaks when nothing looks for it. The regex
       stays as a last resort for a template that cannot be read.
    
    2. THE BUFFERED FORMATTER TOOK NO `reasoning`. Where the thought WAS separated
       it was then dropped: no leak, but no thinking either. litellm's Message has
       no such field, so it is attached after the dump rather than lost -- under
       both spellings, since Qwen's own template reads `reasoning_content`.
    
    3. THE PROXY HOP CARRIED NEITHER FORM. Buffered, the remote sends
       reasoning_content beside an already-cleaned content, which the string
       contract cannot hold: get_last_reasoning(), like get_last_tool_calls().
       Streamed, it sends reasoning DELTAS, and dropping them meant a client saw
       nothing at all until the answer began -- so the <think>/</think> tags go
       back on as they arrive and the caller's existing reasoning gate splits them
       live. An all-thinking stream leaves the tag open on purpose: the gate
       flushes an unclosed thought as reasoning, and there is no content for a
       close tag to precede.
    
    4. THINKING COULD NOT BE TURNED OFF. enable_thinking defaulted to False, so
       "unset" and "explicitly off" were the same value, and the field sat in
       _CHAT_CONSUMED_FIELDS while no backend forwarded it. It defaults to None
       now: unset sends nothing and the model keeps its own default (a thinking
       model thinks), an explicit value travels as chat_template_kwargs. That
       needed the POD backend to gain passthrough at all -- the local vLLM backend
       has forwarded unknown fields for releases, a pod got none of them, so
       chat_template_kwargs could never reach a pod-served model.
    
    ChatMessage silently dropped `reasoning_content`, so a previous turn's thinking
    never reached the template that re-renders it inside <think>. Accepted and
    forwarded now.
    
    The chat page hardcoded stream:false, so thinking could never arrive live
    there, and it read only message.content. It has `stream` and `thinking`
    toggles (both on), renders the thought in a block that fills AS IT ARRIVES and
    collapses when the answer starts, and keeps it in the history it replays.
    Streaming is the request's choice, not forced: cleared, it takes the buffered
    path exactly as before.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    67aa1f5f
Name
Last commit
Last update
..
audiogenrequest.py Loading commit data...
embedrequest.py Loading commit data...
imagerequest.py Loading commit data...
textrequest.py Loading commit data...
transcriptionrequest.py Loading commit data...
videorequest.py Loading commit data...