• Stefy Lanza (nextime / spora )'s avatar
    models: publish the context window, and say so when nobody does (v0.3.53) · 42356742
    Stefy Lanza (nextime / spora ) authored
    /v1/models now carries `max_model_len`, under vLLM's own name so a client
    written against a vLLM server reads it unchanged.
    
    It is published because a client cannot otherwise plan: anything that sends a
    growing conversation -- a continued generation, a resumed OCR page, a long tool
    loop -- has to know when the window runs out, and the only alternative is
    finding out from a 400 after the work is paid for.
    
    coderai's own OCR page resumption was the first casualty, and the way it failed
    is the point. The 0.3.50 clamp probed /v1/models for this field; coderai did not
    publish it, and the surya worker talks to coderai rather than to raw vLLM. So
    the clamp was INERT from the day it shipped, and said nothing: the log printed
    "? of the context window left" and every resumption ran with no ceiling. Two
    further details made it invisible -- `max_model_len` is not in the OpenAI model
    schema, so the client parks it as an EXTRA field and reading attributes alone
    found nothing; and the probe only logged on an exception, so a server that
    answered WITHOUT the field was indistinguishable from one that worked.
    
      - The number comes from the LOADED backend first, config second. The two
        disagree and only the served one is true: `ctx: 18432` is configured on one
        install while `--max-model-len 32768` is what the pods actually serve, so a
        client trusting config would plan around a window that does not exist.
      - VllmBackend.context_length() asks its own upstream /v1/models, cached: a
        served model cannot change shape under us.
      - Resolution handles BOTH manager shapes. list_models is a method of the
        MULTI manager, which has no .backend of its own -- the per-model manager
        does -- so the first version of this could never have fired either. Found
        by writing the test, not by reading the code.
      - Several spellings read (max_model_len/n_ctx/ctx/context_length, plus the
        vllm/runpod/llamacpp blocks), because several backends own this number.
      - None means UNKNOWN, never unlimited, and the worker now says out loud when
        a server named no window, naming ocr.surya_context_tokens as the answer.
    
    Also, so that "which build is this pod running" stops being an inference: the
    pod boot log prints its coderai version and the short git sha of the app layer
    (new GIT_SHA build arg). A mutable `:latest` is a fine way to publish the
    current release, but it leaves two pods started minutes apart looking identical
    in an account listing -- and on 2026-10-13 that cost two pods being restarted
    that already had the fix, because nothing in their logs named their version.
    
    Tests: 27 new. They pin what the resolver PREFERS (live over config), what it
    tolerates (a stray string, a list-wrapped number, a non-dict backend block),
    that another model's loaded backend is not consulted, that the listing actually
    calls it, and that a server naming no window produces a message rather than a
    silence. Full suite: 25 failures, all pre-existing and identical to baseline.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    42356742
Name
Last commit
Last update
..
audiogenrequest.py Loading commit data...
embedrequest.py Loading commit data...
imagerequest.py Loading commit data...
textrequest.py Loading commit data...
transcriptionrequest.py Loading commit data...
videorequest.py Loading commit data...