• Stefy Lanza (nextime / spora )'s avatar
    proxy: forward the fields coderai does not understand (v0.3.51) · 2a45785d
    Stefy Lanza (nextime / spora ) authored
    The Digesta dev's 400 on page resumption, and a bug behind it that was worse
    than the 400.
    
    The vLLM backend built its outgoing request from an ALLOW-LIST of fields it
    recognised. Resumption asks for two beyond an ordinary request --
    continue_final_message, so the model carries on the exact string, and
    add_generation_prompt: false, so no new turn is opened after the prefilled text
    -- and neither was ever forwarded, by this or any previous version.
    
    What reached vLLM was a conversation ending in an assistant turn with
    add_generation_prompt left at its default TRUE. surya-ocr-2's chat template
    then appends a fresh <|im_start|>assistant after the text so far, so the model
    was asked to START A NEW ANSWER about the same image. It did.
    
    And the failure hid behind the guard built to catch it: the new answer repeated
    the beginning of the page, the no-progress check saw text it already had, logged
    "came back repeating itself" and discarded the segment. 253 of 262 resumptions
    on the live install -- read for a day as a model that degenerates on dense
    pages, when it was a model being asked the wrong question, answering correctly,
    and having the answer thrown away. It also explains the dev's table exactly: at
    max_tokens=8000 resumption is barely exercised and OCR holds 93%, while at 4000
    and 2000 nearly every page resumes, each resumption is a full generation that
    produces a restart and is then discarded, and the batch times out.
    
    Fixed by forwarding BY EXCLUSION, which is the only form that stays correct:
    coderai names the fields it CONSUMES (_CHAT_CONSUMED_FIELDS) and everything
    else goes to the server, which is the thing entitled to decide what it
    supports. An allow-list means every field a server gains after the list was
    written is born broken -- the same reasoning as forwarding page diagnostics by
    exclusion. It reaches only a backend whose signature accepts it, so a backend
    that cannot carry an unknown field is unaffected rather than broken, and
    model/messages/stream are never overridable this way.
    
    Also: a 4xx explains itself, if nobody discards it. raise_for_status() reports
    "400 Client Error: Bad Request for url: ..." -- the symptom and nothing else --
    while vLLM puts the reason in the BODY. A whole diagnosis cycle was spent
    guessing at a 400 that had already been explained and thrown away. The error
    now carries the server's own words. Made tolerant of a response object that
    reports no status_code, rather than turning an odd shape into an AttributeError
    inside the request path.
    
    Two findings for the operator, not code: the pods serve max-model-len 32768
    while the model entry asks for ctx 18432, so per-model vllm settings still are
    not reaching the pod; and this fix must be in the POD image, because the proxy
    that dropped the fields runs there.
    
    Tests: 16 new (flags reach the payload, an unknown field is forwarded, no
    passthrough changes nothing, structure not overridable, exclusion at the API
    layer, signature-probed delivery, the error carries the server's words). Full
    suite: 25 failures, all pre-existing and identical to the stashed baseline.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    2a45785d
test_chat_passthrough.py 7.81 KB