-
Stefy Lanza (nextime / spora ) authored
Both of these came out of probing the live install rather than the test fixtures, and neither was visible from the code. **A streamed overflow arrived as a successful empty answer.** A remote serving an OpenAI API answers 200 the moment the stream opens, so anything that fails afterwards can only be said inside the stream -- coderai does exactly that, and deliberately, so a client treats it as an error rather than as assistant text. But that event carries no `choices`, and the proxy's SSE reader looked only at `choices[0].delta`: the event parsed to "no text", was dropped, and the stream ended. The client got HTTP 200, `finish_reason: "stop"`, the PREVIOUS request's usage and no content. This was not specific to overflow -- every mid-stream failure from a pod arrived as silence. Both proxies now raise RemoteStreamError, on "present and not None", because `{"error": {}}` is a failure while some servers put a literal `"error": null` in every ordinary chunk. **The refusal parser was written against the wrong wording.** The hand-written fixture used the older OpenAI phrasing ("you requested N tokens"); vLLM 0.27 says "you requested 300 output tokens and your prompt contains at least 32469 input tokens, for a total of at least 32769 tokens", so nothing matched and the message that was supposed to lead with the numbers led with "the request is longer than the context window" instead. The refusal a production pod actually sent is now a test fixture, wrappers and all, alongside the parameter clause ("parameter=input_tokens, value=N") that a build stating only the limit still appends. Verified on the install: `usage.context_size` now reports 32768 where it reported 2048 before, thinking still arrives as its own channel (182 chars of reasoning at 0.3s, content at 2.7s), and a 181,546-character conversation is refused with both token counts inside the first 90 characters. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
8acbf8ff