-
Stefy Lanza (nextime / spora ) authored
The Digesta dev's 400 on page resumption, and a bug behind it that was worse than the 400. The vLLM backend built its outgoing request from an ALLOW-LIST of fields it recognised. Resumption asks for two beyond an ordinary request -- continue_final_message, so the model carries on the exact string, and add_generation_prompt: false, so no new turn is opened after the prefilled text -- and neither was ever forwarded, by this or any previous version. What reached vLLM was a conversation ending in an assistant turn with add_generation_prompt left at its default TRUE. surya-ocr-2's chat template then appends a fresh <|im_start|>assistant after the text so far, so the model was asked to START A NEW ANSWER about the same image. It did. And the failure hid behind the guard built to catch it: the new answer repeated the beginning of the page, the no-progress check saw text it already had, logged "came back repeating itself" and discarded the segment. 253 of 262 resumptions on the live install -- read for a day as a model that degenerates on dense pages, when it was a model being asked the wrong question, answering correctly, and having the answer thrown away. It also explains the dev's table exactly: at max_tokens=8000 resumption is barely exercised and OCR holds 93%, while at 4000 and 2000 nearly every page resumes, each resumption is a full generation that produces a restart and is then discarded, and the batch times out. Fixed by forwarding BY EXCLUSION, which is the only form that stays correct: coderai names the fields it CONSUMES (_CHAT_CONSUMED_FIELDS) and everything else goes to the server, which is the thing entitled to decide what it supports. An allow-list means every field a server gains after the list was written is born broken -- the same reasoning as forwarding page diagnostics by exclusion. It reaches only a backend whose signature accepts it, so a backend that cannot carry an unknown field is unaffected rather than broken, and model/messages/stream are never overridable this way. Also: a 4xx explains itself, if nobody discards it. raise_for_status() reports "400 Client Error: Bad Request for url: ..." -- the symptom and nothing else -- while vLLM puts the reason in the BODY. A whole diagnosis cycle was spent guessing at a 400 that had already been explained and thrown away. The error now carries the server's own words. Made tolerant of a response object that reports no status_code, rather than turning an odd shape into an AttributeError inside the request path. Two findings for the operator, not code: the pods serve max-model-len 32768 while the model entry asks for ctx 18432, so per-model vllm settings still are not reaching the pod; and this fix must be in the POD image, because the proxy that dropped the fields runs there. Tests: 16 new (flags reach the payload, an unknown field is forwarded, no passthrough changes nothing, structure not overridable, exclusion at the API layer, signature-probed delivery, the error carries the server's words). Full suite: 25 failures, all pre-existing and identical to the stashed baseline. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
2a45785d