• Stefy Lanza (nextime / spora )'s avatar
    say why generation stopped, instead of always "stop" (v0.4.83) · a2e3a0f6
    Stefy Lanza (nextime / spora ) authored
    Measured against the production pod: a reasoning model asked for 64 tokens
    spent all 64 inside its thought, returned NO content, and the response said
    finish_reason "stop".
    
        max_tokens=64    finish=stop  content=0 chars ''             reasoning=262
        max_tokens=1500  finish=stop  content=13 chars 'XMU-5635-FB' reasoning=448
    
    Two layers were each sure the other knew better. The proxy hardcoded
    finish_reason = "stop" and never read choices[0].finish_reason from the remote;
    OpenAIFormatter then hardcoded it again, so even a caller that worked the
    reason out had it overwritten on the way to the client. vLLM had been saying
    "length" all along.
    
    It matters because "length" is the one reason a client can act on. An answer
    cut off at the limit is half a sentence, and told it stopped normally the client
    has nothing to retry on and no sign anything is missing. On a reasoning model
    the thought is generated FIRST and spends the same budget, so a tight limit
    returns an EMPTY answer that claims to have finished of its own accord -- which
    is how this surfaced, as a retrieval test scoring zero for reasons that had
    nothing to do with what it was measuring.
    
    The remote's word is now carried through the per-call sink beside reasoning,
    tool_calls and logprobs (get_last_finish_reason, buffered and streamed), the
    formatter accepts the caller's reason and keeps "stop" only as the default, and
    a tool call still wins -- that describes the answer's shape, and a truncation
    after the call was formed must not relabel it. Fixed at all five response
    exits, not just the one it was found at: streamed chat (both terminal-chunk
    paths), buffered chat, and streamed and buffered /v1/completions.
    
    A local backend is never told a reason, so the budget decides: generation that
    used every token it was allowed was cut off, not finished. Drawn only when a
    limit was actually set and reached -- never from a token count alone, or every
    long completion would be mislabelled.
    
    Tests verified against the reverted files: assert None == 'length' for the
    backends, TypeError: unexpected keyword 'finish_reason' for the formatter.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    a2e3a0f6
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...