• Stefy Lanza (nextime / spora )'s avatar
    a stream that fails is not a stream that finished (v0.4.73) · 8acbf8ff
    Stefy Lanza (nextime / spora ) authored
    Both of these came out of probing the live install rather than the test
    fixtures, and neither was visible from the code.
    
    **A streamed overflow arrived as a successful empty answer.** A remote serving
    an OpenAI API answers 200 the moment the stream opens, so anything that fails
    afterwards can only be said inside the stream -- coderai does exactly that, and
    deliberately, so a client treats it as an error rather than as assistant text.
    But that event carries no `choices`, and the proxy's SSE reader looked only at
    `choices[0].delta`: the event parsed to "no text", was dropped, and the stream
    ended. The client got HTTP 200, `finish_reason: "stop"`, the PREVIOUS request's
    usage and no content. This was not specific to overflow -- every mid-stream
    failure from a pod arrived as silence. Both proxies now raise RemoteStreamError,
    on "present and not None", because `{"error": {}}` is a failure while some
    servers put a literal `"error": null` in every ordinary chunk.
    
    **The refusal parser was written against the wrong wording.** The hand-written
    fixture used the older OpenAI phrasing ("you requested N tokens"); vLLM 0.27
    says "you requested 300 output tokens and your prompt contains at least 32469
    input tokens, for a total of at least 32769 tokens", so nothing matched and the
    message that was supposed to lead with the numbers led with "the request is
    longer than the context window" instead. The refusal a production pod actually
    sent is now a test fixture, wrappers and all, alongside the parameter clause
    ("parameter=input_tokens, value=N") that a build stating only the limit still
    appends.
    
    Verified on the install: `usage.context_size` now reports 32768 where it
    reported 2048 before, thinking still arrives as its own channel (182 chars of
    reasoning at 0.3s, content at 2.7s), and a 181,546-character conversation is
    refused with both token counts inside the first 90 characters.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    8acbf8ff
test_stream_error_is_not_silence.py 7.05 KB