• Stefy Lanza (nextime / spora )'s avatar
    say why generation stopped, instead of always "stop" (v0.4.83) · a2e3a0f6
    Stefy Lanza (nextime / spora ) authored
    Measured against the production pod: a reasoning model asked for 64 tokens
    spent all 64 inside its thought, returned NO content, and the response said
    finish_reason "stop".
    
        max_tokens=64    finish=stop  content=0 chars ''             reasoning=262
        max_tokens=1500  finish=stop  content=13 chars 'XMU-5635-FB' reasoning=448
    
    Two layers were each sure the other knew better. The proxy hardcoded
    finish_reason = "stop" and never read choices[0].finish_reason from the remote;
    OpenAIFormatter then hardcoded it again, so even a caller that worked the
    reason out had it overwritten on the way to the client. vLLM had been saying
    "length" all along.
    
    It matters because "length" is the one reason a client can act on. An answer
    cut off at the limit is half a sentence, and told it stopped normally the client
    has nothing to retry on and no sign anything is missing. On a reasoning model
    the thought is generated FIRST and spends the same budget, so a tight limit
    returns an EMPTY answer that claims to have finished of its own accord -- which
    is how this surfaced, as a retrieval test scoring zero for reasons that had
    nothing to do with what it was measuring.
    
    The remote's word is now carried through the per-call sink beside reasoning,
    tool_calls and logprobs (get_last_finish_reason, buffered and streamed), the
    formatter accepts the caller's reason and keeps "stop" only as the default, and
    a tool call still wins -- that describes the answer's shape, and a truncation
    after the call was formed must not relabel it. Fixed at all five response
    exits, not just the one it was found at: streamed chat (both terminal-chunk
    paths), buffered chat, and streamed and buffered /v1/completions.
    
    A local backend is never told a reason, so the budget decides: generation that
    used every token it was allowed was cut off, not finished. Drawn only when a
    limit was actually set and reached -- never from a token count alone, or every
    long completion would be mislabelled.
    
    Tests verified against the reverted files: assert None == 'length' for the
    backends, TypeError: unexpected keyword 'finish_reason' for the formatter.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    a2e3a0f6
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
compose Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
util Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
ocr_fleet_workers.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...