• Stefy Lanza (nextime / spora )'s avatar
    deliver a model's thinking as thinking, while it is thinking (v0.4.64) · 67aa1f5f
    Stefy Lanza (nextime / spora ) authored
    From ONE pod, ONE set of weights, behind two aliases:
    
      qwen38        -> clean answer
      diogene-alpha -> "The user is asking to list hearings from 2026-10-21 to..."
    
    The entire chain of thought went to the client as the answer. Four breaks:
    
    1. THE GATE ASKED THE NAME. _reasoning_active was a regex (qwen3|qwq|
       deepseek-r1|...) over the request's model name: `qwen3` matches "qwen38" and
       not "diogene-alpha", so one alias of one model separated its thought and the
       other did not. It is read from the template now -- whose generation prompt
       ends with '<think>\n', so the model starts INSIDE the thought and emits a
       bare </think>, the form that leaks when nothing looks for it. The regex
       stays as a last resort for a template that cannot be read.
    
    2. THE BUFFERED FORMATTER TOOK NO `reasoning`. Where the thought WAS separated
       it was then dropped: no leak, but no thinking either. litellm's Message has
       no such field, so it is attached after the dump rather than lost -- under
       both spellings, since Qwen's own template reads `reasoning_content`.
    
    3. THE PROXY HOP CARRIED NEITHER FORM. Buffered, the remote sends
       reasoning_content beside an already-cleaned content, which the string
       contract cannot hold: get_last_reasoning(), like get_last_tool_calls().
       Streamed, it sends reasoning DELTAS, and dropping them meant a client saw
       nothing at all until the answer began -- so the <think>/</think> tags go
       back on as they arrive and the caller's existing reasoning gate splits them
       live. An all-thinking stream leaves the tag open on purpose: the gate
       flushes an unclosed thought as reasoning, and there is no content for a
       close tag to precede.
    
    4. THINKING COULD NOT BE TURNED OFF. enable_thinking defaulted to False, so
       "unset" and "explicitly off" were the same value, and the field sat in
       _CHAT_CONSUMED_FIELDS while no backend forwarded it. It defaults to None
       now: unset sends nothing and the model keeps its own default (a thinking
       model thinks), an explicit value travels as chat_template_kwargs. That
       needed the POD backend to gain passthrough at all -- the local vLLM backend
       has forwarded unknown fields for releases, a pod got none of them, so
       chat_template_kwargs could never reach a pod-served model.
    
    ChatMessage silently dropped `reasoning_content`, so a previous turn's thinking
    never reached the template that re-renders it inside <think>. Accepted and
    forwarded now.
    
    The chat page hardcoded stream:false, so thinking could never arrive live
    there, and it read only message.content. It has `stream` and `thinking`
    toggles (both on), renders the thought in a block that fills AS IT ARRIVES and
    collapses when the answer starts, and keeps it in the history it replays.
    Streaming is the request's choice, not forced: cleared, it takes the buffered
    path exactly as before.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    67aa1f5f
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...