-
Stefy Lanza (nextime / spora ) authored
/v1/models now carries `max_model_len`, under vLLM's own name so a client written against a vLLM server reads it unchanged. It is published because a client cannot otherwise plan: anything that sends a growing conversation -- a continued generation, a resumed OCR page, a long tool loop -- has to know when the window runs out, and the only alternative is finding out from a 400 after the work is paid for. coderai's own OCR page resumption was the first casualty, and the way it failed is the point. The 0.3.50 clamp probed /v1/models for this field; coderai did not publish it, and the surya worker talks to coderai rather than to raw vLLM. So the clamp was INERT from the day it shipped, and said nothing: the log printed "? of the context window left" and every resumption ran with no ceiling. Two further details made it invisible -- `max_model_len` is not in the OpenAI model schema, so the client parks it as an EXTRA field and reading attributes alone found nothing; and the probe only logged on an exception, so a server that answered WITHOUT the field was indistinguishable from one that worked. - The number comes from the LOADED backend first, config second. The two disagree and only the served one is true: `ctx: 18432` is configured on one install while `--max-model-len 32768` is what the pods actually serve, so a client trusting config would plan around a window that does not exist. - VllmBackend.context_length() asks its own upstream /v1/models, cached: a served model cannot change shape under us. - Resolution handles BOTH manager shapes. list_models is a method of the MULTI manager, which has no .backend of its own -- the per-model manager does -- so the first version of this could never have fired either. Found by writing the test, not by reading the code. - Several spellings read (max_model_len/n_ctx/ctx/context_length, plus the vllm/runpod/llamacpp blocks), because several backends own this number. - None means UNKNOWN, never unlimited, and the worker now says out loud when a server named no window, naming ocr.surya_context_tokens as the answer. Also, so that "which build is this pod running" stops being an inference: the pod boot log prints its coderai version and the short git sha of the app layer (new GIT_SHA build arg). A mutable `:latest` is a fine way to publish the current release, but it leaves two pods started minutes apart looking identical in an account listing -- and on 2026-10-13 that cost two pods being restarted that already had the fix, because nothing in their logs named their version. Tests: 27 new. They pin what the resolver PREFERS (live over config), what it tolerates (a stray string, a list-wrapped number, a non-dict backend block), that another model's loaded backend is not consulted, that the listing actually calls it, and that a server naming no window produces a message rather than a silence. Full suite: 25 failures, all pre-existing and identical to baseline. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
42356742