-
Stefy Lanza (nextime / spora ) authored
Measured against the production pod: a reasoning model asked for 64 tokens spent all 64 inside its thought, returned NO content, and the response said finish_reason "stop". max_tokens=64 finish=stop content=0 chars '' reasoning=262 max_tokens=1500 finish=stop content=13 chars 'XMU-5635-FB' reasoning=448 Two layers were each sure the other knew better. The proxy hardcoded finish_reason = "stop" and never read choices[0].finish_reason from the remote; OpenAIFormatter then hardcoded it again, so even a caller that worked the reason out had it overwritten on the way to the client. vLLM had been saying "length" all along. It matters because "length" is the one reason a client can act on. An answer cut off at the limit is half a sentence, and told it stopped normally the client has nothing to retry on and no sign anything is missing. On a reasoning model the thought is generated FIRST and spends the same budget, so a tight limit returns an EMPTY answer that claims to have finished of its own accord -- which is how this surfaced, as a retrieval test scoring zero for reasons that had nothing to do with what it was measuring. The remote's word is now carried through the per-call sink beside reasoning, tool_calls and logprobs (get_last_finish_reason, buffered and streamed), the formatter accepts the caller's reason and keeps "stop" only as the default, and a tool call still wins -- that describes the answer's shape, and a truncation after the call was formed must not relabel it. Fixed at all five response exits, not just the one it was found at: streamed chat (both terminal-chunk paths), buffered chat, and streamed and buffered /v1/completions. A local backend is never told a reason, so the budget decides: generation that used every token it was allowed was cut off, not finished. Drawn only when a limit was actually set and reached -- never from a token count alone, or every long completion would be mislabelled. Tests verified against the reverted files: assert None == 'length' for the backends, TypeError: unexpected keyword 'finish_reason' for the formatter. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
a2e3a0f6
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| admin | ||
| api | ||
| backends | ||
| broker | ||
| cluster | ||
| compose | ||
| frontproxy | ||
| models | ||
| ocr | ||
| openai | ||
| pydantic | ||
| queue | ||
| tasks | ||
| util | ||
| __init__.py | ||
| cli.py | ||
| config.py | ||
| main.py | ||
| ocr_fleet_workers.py | ||
| platform_paths.py | ||
| system_app.py |