models: publish the context window, and say so when nobody does (v0.3.53)

/v1/models now carries `max_model_len`, under vLLM's own name so a client
written against a vLLM server reads it unchanged.

It is published because a client cannot otherwise plan: anything that sends a
growing conversation -- a continued generation, a resumed OCR page, a long tool
loop -- has to know when the window runs out, and the only alternative is
finding out from a 400 after the work is paid for.

coderai's own OCR page resumption was the first casualty, and the way it failed
is the point. The 0.3.50 clamp probed /v1/models for this field; coderai did not
publish it, and the surya worker talks to coderai rather than to raw vLLM. So
the clamp was INERT from the day it shipped, and said nothing: the log printed
"? of the context window left" and every resumption ran with no ceiling. Two
further details made it invisible -- `max_model_len` is not in the OpenAI model
schema, so the client parks it as an EXTRA field and reading attributes alone
found nothing; and the probe only logged on an exception, so a server that
answered WITHOUT the field was indistinguishable from one that worked.

  - The number comes from the LOADED backend first, config second. The two
    disagree and only the served one is true: `ctx: 18432` is configured on one
    install while `--max-model-len 32768` is what the pods actually serve, so a
    client trusting config would plan around a window that does not exist.
  - VllmBackend.context_length() asks its own upstream /v1/models, cached: a
    served model cannot change shape under us.
  - Resolution handles BOTH manager shapes. list_models is a method of the
    MULTI manager, which has no .backend of its own -- the per-model manager
    does -- so the first version of this could never have fired either. Found
    by writing the test, not by reading the code.
  - Several spellings read (max_model_len/n_ctx/ctx/context_length, plus the
    vllm/runpod/llamacpp blocks), because several backends own this number.
  - None means UNKNOWN, never unlimited, and the worker now says out loud when
    a server named no window, naming ocr.surya_context_tokens as the answer.

Also, so that "which build is this pod running" stops being an inference: the
pod boot log prints its coderai version and the short git sha of the app layer
(new GIT_SHA build arg). A mutable `:latest` is a fine way to publish the
current release, but it leaves two pods started minutes apart looking identical
in an account listing -- and on 2026-10-13 that cost two pods being restarted
that already had the fix, because nothing in their logs named their version.

Tests: 27 new. They pin what the resolver PREFERS (live over config), what it
tolerates (a stray string, a list-wrapped number, a non-dict backend block),
that another model's loaded backend is not consulted, that the listing actually
calls it, and that a server naming no window produces a message rather than a
silence. Full suite: 25 failures, all pre-existing and identical to baseline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.52"
+__version__ = "0.3.53"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/backends/vllm.py

@@ -164,6 +164,38 @@ class VllmBackend(ModelBackend):
             raise RuntimeError("vLLM service not started")
         return self._url
 
+    def context_length(self):
+        """The context window the SERVER reports, in tokens, or None.
+
+        Asked of the running server rather than read from config, because the
+        two disagree and only one of them is true: an operator can set a
+        per-model `ctx` that never reaches the launch arguments (measured on one
+        install 2026-10-13 -- `ctx: 18432` configured, `--max-model-len 32768`
+        actually served), and a client that trusted the config would plan around
+        a window the model does not have.
+
+        Cached: a served model cannot change shape under us, so one question per
+        process is enough. Failure is cached as None and not retried hard, since
+        not knowing is survivable and a probe on every listing is not.
+        """
+        if getattr(self, "_ctx_len", "unset") != "unset":
+            return self._ctx_len
+        self._ctx_len = None
+        try:
+            import requests
+            r = requests.get(self._base() + "/v1/models", timeout=5)
+            for m in (r.json().get("data") or []):
+                if m.get("id") != getattr(self, "_served_name", None):
+                    continue
+                for k in ("max_model_len", "max_context_length", "context_length"):
+                    if m.get(k):
+                        self._ctx_len = int(m[k])
+                        break
+                break
+        except Exception:
+            pass
+        return self._ctx_len
+
     def _store_usage(self, usage: dict) -> None:
         if usage:
             self._last_usage = {

codai/models/manager.py

@@ -530,6 +530,86 @@ def _drop_prefix_caches() -> int:
     return n
 
 
+def _live_context_length(manager, model_id: str):
+    """Ask the backend actually serving ``model_id``, or None.
+
+    Two shapes, because the listing is built by the MULTI manager while the
+    backend lives on a per-model one: a `ModelManager` carries `.backend`
+    itself, a `MultiModelManager` carries a dict of loaded sub-managers in
+    `.models`. Handling only the first was the bug -- `list_models` is a method
+    of the second, so the live path could never have fired.
+
+    Matched on what the backend says it SERVES. `current_model` holds a resolved
+    path rather than an id, so it alone would miss almost every time.
+    """
+    def _ask(owner):
+        b = getattr(owner, "backend", None)
+        if b is None or not hasattr(b, "context_length"):
+            return None
+        names = {getattr(owner, "current_model", None),
+                 getattr(b, "_served_name", None),
+                 getattr(b, "_model_id", None)}
+        return b.context_length() if model_id in names else None
+
+    got = _ask(manager)
+    if got:
+        return got
+    loaded = getattr(manager, "models", None)
+    if isinstance(loaded, dict):
+        sub = loaded.get(model_id)
+        if sub is not None:
+            got = _ask(sub)
+            if got:
+                return got
+        # The dict is keyed by the manager's own scheme ("whisper:x", a path),
+        # so a direct hit is not guaranteed even when the model IS loaded.
+        for sub in list(loaded.values()):
+            got = _ask(sub)
+            if got:
+                return got
+    return None
+
+
+def context_window_for(manager, model_id: str, meta: Dict[str, Any]):
+    """The context window to publish for ``model_id``, or None if unknown.
+
+    Order matters. A LOADED backend that can be asked wins, because the
+    configured number and the served number disagree in practice and only the
+    served one is true: an operator can set a per-model ``ctx`` that never
+    reaches the launch arguments (measured on one install 2026-10-13 --
+    ``ctx: 18432`` configured, ``--max-model-len 32768`` actually served), and a
+    client trusting the config would plan around a window the model does not
+    have. Config is the fallback, and ``None`` is a legitimate answer: a client
+    must read it as "unknown", never as "unlimited".
+
+    Module level rather than a closure inside the listing, so it can be tested
+    for what it prefers and what it tolerates without building a whole listing.
+    """
+    try:
+        live = _live_context_length(manager, model_id)
+        if live:
+            return int(live)
+    except Exception:
+        pass
+    # Several spellings, because several backends own this number: the entry's
+    # own keys, then the per-backend blocks a pod or a server is launched from.
+    if not isinstance(meta, dict):
+        return None
+    for src in (meta, meta.get("vllm"), meta.get("runpod"), meta.get("llamacpp")):
+        if not isinstance(src, dict):
+            continue
+        for k in ("max_model_len", "n_ctx", "ctx", "context_length"):
+            v = src.get(k)
+            if isinstance(v, (list, tuple)):
+                v = v[0] if v else None
+            try:
+                if v and int(v) > 0:
+                    return int(v)
+            except (TypeError, ValueError):
+                continue
+    return None
+
+
 class ModelManager:
     """Manages the loaded model and tokenizer."""
 
@@ -5772,6 +5852,7 @@ class MultiModelManager:
                 languages=meta.get("languages") or None,
                 supports_translation=meta.get("supports_translation"),
                 video=video_geom,
+                max_model_len=context_window_for(self, model_id, meta),
             ))
 
         # --- Models from config (the authoritative source) ---

codai/ocr/workers/ocr_worker.py

@@ -106,12 +106,21 @@ def _context_limit(client, model_name):
     if key in _CONTEXT_CACHE:
         return _CONTEXT_CACHE[key]
     limit = 0
+    answered = False
     try:
         for m in (client.models.list().data or []):
             if getattr(m, "id", None) != model_name:
                 continue
+            answered = True
             for attr in ("max_model_len", "max_context_length", "context_length"):
+                # `max_model_len` is not in the OpenAI model schema, so the
+                # client parks it as an EXTRA field rather than an attribute.
+                # Reading only attributes found nothing and said nothing.
                 v = getattr(m, attr, None)
+                if v is None:
+                    extra = getattr(m, "model_extra", None)
+                    if isinstance(extra, dict):
+                        v = extra.get(attr)
                 if v is None and isinstance(getattr(m, "__dict__", None), dict):
                     v = m.__dict__.get(attr)
                 if v:
@@ -127,6 +136,15 @@ def _context_limit(client, model_name):
     if limit:
         print(f"[surya] {model_name} serves a {limit}-token context window",
               file=sys.stderr, flush=True)
+    elif answered:
+        # The case that actually happened, and the one that used to pass in
+        # silence: the server answered, listed the model, and named no window.
+        # Resumption then runs with no ceiling and the log shows '?', with
+        # nothing to say why -- a degradation indistinguishable from working.
+        print(f"[surya] {model_name} does not report a context window on "
+              f"/v1/models; resumption cannot stop before the server does — "
+              f"set ocr.surya_context_tokens to the served --max-model-len",
+              file=sys.stderr, flush=True)
     _CONTEXT_CACHE[key] = limit
     return limit
 

codai/pydantic/textrequest.py

@@ -155,6 +155,22 @@ class ModelInfo(BaseModel):
     languages: Optional[List[str]] = None
     supports_translation: Optional[bool] = None
     speaker_diarization: Optional[bool] = None
+    # The context window this model is being SERVED with, in tokens.
+    #
+    # Named as vLLM names it, so a client written against a vLLM server reads it
+    # without changes. It is published because a client cannot otherwise plan:
+    # anything that sends a growing conversation -- a continued generation, a
+    # resumed OCR page, a long tool loop -- needs to know when the window runs
+    # out, and the alternative is to find out from a 400 after the work is done.
+    # coderai's own OCR page resumption was the first casualty: it probed
+    # /v1/models for this field, coderai did not publish it, so the resumption
+    # budget could not see the ceiling it was about to hit.
+    #
+    # The number is what the model is served with, not what the checkpoint
+    # advertises: an operator may serve a shorter window deliberately. None means
+    # "not established here", which a client must treat as unknown rather than
+    # unlimited.
+    max_model_len: Optional[int] = None
     # Video generation: what this model can actually emit — native frame rate, the
     # legal frame-count grid, how long ONE render may be and whether continuation is
     # native. Published so a client can PLAN around the model (how long a scene can

docs/ocr.md

@@ -1139,8 +1139,14 @@ the full cap dry, and the third segment of any page was **rejected outright**
 So coderai now establishes the window and respects it:
 
 - `surya_context_tokens`, or `0` to read the server's own `max_model_len` from
-  `/v1/models` (vLLM reports it). Configuration wins, because an operator may
-  serve with a shorter `--max-model-len` than the checkpoint advertises.
+  `/v1/models`. Configuration wins, because an operator may serve with a
+  shorter `--max-model-len` than the checkpoint advertises.
+- coderai **publishes** that number on its own `/v1/models`, under vLLM's
+  name, so this probe works when the engine is reached through coderai
+  rather than a raw vLLM. It did not at first, and the probe then degraded
+  in silence: the log printed `? of the context window left` with nothing
+  to say why, and the clamp never fired. A server that answers without the
+  field now says so and names the setting to use instead.
 - Each segment asks for **the smaller of the cap and the room left**, and the
   room left is measured from what the server said the prompt actually cost
   (`usage.prompt_tokens`), not guessed — the guess is used for the first

packaging/runpod/Dockerfile.capability

@@ -29,6 +29,12 @@ ENV PROFILE=${PROFILE}
 # whether the pod ran the new image or an old one. Now it says so at boot.
 ARG IMAGE_TAG=unknown
 ENV CODERAI_IMAGE_TAG=${IMAGE_TAG}
+# The commit the app layer was built from. The version alone cannot distinguish
+# two builds of the same version, which is exactly the case while a fix is being
+# chased: boot.sh prints both so "which code is this pod running" is answerable
+# from the pod's own log instead of inferred from timestamps.
+ARG GIT_SHA=unknown
+ENV CODERAI_GIT_SHA=${GIT_SHA}
 
 WORKDIR /opt/coderai/app
 COPY packaging/runpod/profiles/ /tmp/profiles/

packaging/runpod/boot.sh

@@ -37,6 +37,19 @@ phase "profile=${PROFILE:-unknown} image=${CODERAI_IMAGE_TAG:-unknown}"
 phase "user=$(id -un 2>/dev/null || echo unknown) cwd=$(pwd 2>/dev/null || echo unknown)"
 phase "python=$(python -V 2>&1)"
 
+# WHICH BUILD IS THIS POD RUNNING. A mutable tag like `:latest` is a perfectly
+# good way to publish "the current release", but it makes the question
+# unanswerable from the outside: two pods started minutes apart can carry
+# different code and look identical in the account listing. Working that out by
+# comparing pod creation times against a push time is inference, not fact -- and
+# on 2026-10-13 that inference cost two pods being restarted that already had
+# the fix, because nothing in their logs named their own version.
+#
+# One line, at boot, from the code that is actually loaded (not the tag, which
+# is a label someone else controls, and not a build arg, which records what the
+# builder intended rather than what got layered on).
+phase "coderai=$(python -c 'import codai; print(codai.__version__)' 2>/dev/null || echo unknown) git=${CODERAI_GIT_SHA:-unknown}"
+
 # What the pod was actually told to serve. A pod that answers "not available"
 # for its own model is the single most common failure, and the seed list is the
 # first place to look — it is worth one line at boot rather than an exec into a

packaging/runpod/build_capability_image.sh

@@ -101,6 +101,7 @@ echo "==> building $TAG from profile '$PROFILE' ($DOCKERFILE)"
 DOCKER_BUILDKIT=1 docker build -f "$DOCKERFILE" \
     --build-arg "PROFILE=$PROFILE" \
     --build-arg "IMAGE_TAG=$TAG" \
+    --build-arg "GIT_SHA=$(git rev-parse --short HEAD 2>/dev/null || echo unknown)" \
     --build-arg "CORE_BASE=$CORE_BASE" \
     -t "$TAG" .
 

tests/test_models_context_window.py

+"""`/v1/models` must publish the context window it is serving with.
+
+Anything that sends a growing conversation -- a continued generation, a resumed
+OCR page, a long tool loop -- has to know when the window runs out, and the only
+alternative is finding out from a 400 after the work is paid for.
+
+coderai's OCR page resumption was the first casualty. It probed `/v1/models` for
+`max_model_len`, coderai did not publish it, and the probe degraded in SILENCE:
+the resumption budget ran with no ceiling and the log printed `?` with nothing to
+say why. Two things are asserted here as a result -- that the field is published,
+and that a server which answers WITHOUT it says so out loud.
+
+The name is vLLM's, so a client written against a vLLM server reads it unchanged.
+"""
+import types
+
+import pytest
+
+
+# --- the field exists and means what it says ---------------------------------
+
+
+def test_model_info_publishes_the_context_window():
+    from codai.pydantic.textrequest import ModelInfo
+    m = ModelInfo(id="x", max_model_len=32768)
+    assert m.max_model_len == 32768
+    assert "max_model_len" in m.model_dump()
+
+
+def test_an_unknown_window_is_none_not_zero():
+    """None means "not established here". 0 would read as a window of zero, and
+    a client must not be able to confuse unknown with unlimited either."""
+    from codai.pydantic.textrequest import ModelInfo
+    assert ModelInfo(id="x").max_model_len is None
+
+
+# --- where the number comes from, and in which order -------------------------
+
+
+class _Mgr:
+    """Just enough of a manager for the resolver: a loaded backend, and the
+    resolved-path attribute the real one carries."""
+
+    def __init__(self, backend=None, current_model=None):
+        self.backend = backend
+        self.current_model = current_model
+
+
+@pytest.fixture
+def resolve():
+    from codai.models.manager import context_window_for
+    return context_window_for
+
+
+def test_the_listing_actually_calls_the_resolver():
+    """The resolver being correct is worth nothing if the listing does not use
+    it — which is how `max_model_len` came to be probed for and never
+    published."""
+    import inspect
+    from codai.models.manager import MultiModelManager
+    src = inspect.getsource(MultiModelManager.list_models)
+    assert "max_model_len=context_window_for(self, model_id, meta)" in src
+
+
+def test_a_live_backend_beats_the_configuration(resolve):
+    """The two disagree in practice and only the served one is true: measured
+    2026-10-13, `ctx: 18432` was configured and `--max-model-len 32768` served.
+    A client trusting the config would plan around a window that does not
+    exist."""
+    backend = types.SimpleNamespace(_served_name="m", _model_id="m",
+                                    context_length=lambda: 32768)
+    got = resolve(_Mgr(backend=backend, current_model="/resolved/path"), "m",
+                  {"runpod": {"ctx": 18432}})
+    assert got == 32768
+
+
+def test_the_backend_is_matched_on_what_it_says_it_serves(resolve):
+    """`current_model` holds a RESOLVED PATH, not the id, so matching on it
+    alone misses almost every time."""
+    backend = types.SimpleNamespace(_served_name="m", _model_id="m",
+                                    context_length=lambda: 8192)
+    got = resolve(_Mgr(backend=backend, current_model="/opt/models/m.gguf"),
+                  "m", {})
+    assert got == 8192
+
+
+def test_another_models_backend_is_not_asked(resolve):
+    backend = types.SimpleNamespace(_served_name="other", _model_id="other",
+                                    context_length=lambda: 999999)
+    got = resolve(_Mgr(backend=backend), "m", {"n_ctx": 4096})
+    assert got == 4096
+
+
+def test_a_backend_that_raises_falls_back_to_configuration(resolve):
+    def boom():
+        raise RuntimeError("service not started")
+    backend = types.SimpleNamespace(_served_name="m", _model_id="m",
+                                    context_length=boom)
+    assert resolve(_Mgr(backend=backend), "m", {"ctx": 16384}) == 16384
+
+
+@pytest.mark.parametrize("meta,expected", [
+    ({"max_model_len": 32768}, 32768),
+    ({"n_ctx": 8192}, 8192),
+    ({"ctx": 4096}, 4096),
+    ({"context_length": 2048}, 2048),
+    ({"vllm": {"max_model_len": 65536}}, 65536),
+    ({"runpod": {"ctx": 18432}}, 18432),
+    ({"llamacpp": {"n_ctx": 512}}, 512),
+])
+def test_every_spelling_a_backend_uses_is_read(resolve, meta, expected):
+    """Several backends own this number and each spells it its own way; a
+    resolver that knew one spelling would publish None for the others."""
+    assert resolve(_Mgr(), "m", meta) == expected
+
+
+@pytest.mark.parametrize("meta", [
+    {}, {"ctx": 0}, {"ctx": None}, {"ctx": "wide"}, {"ctx": []},
+    {"vllm": "not-a-dict"},
+])
+def test_nothing_usable_resolves_to_none(resolve, meta):
+    """A stray string or list must not become a window, and must not raise."""
+    assert resolve(_Mgr(), "m", meta) is None
+
+
+def test_a_list_wrapped_number_is_unwrapped(resolve):
+    """An old global default wrapped in a list is a real shape in these configs."""
+    assert resolve(_Mgr(), "m", {"ctx": [16384]}) == 16384
+
+
+# --- the OCR worker reads it, and complains when it is absent ----------------
+
+
+def _worker_module():
+    import importlib
+    return importlib.import_module("codai.ocr.workers.ocr_worker")
+
+
+class _Models:
+    def __init__(self, payload):
+        self._payload = payload
+
+    def list(self):
+        return types.SimpleNamespace(data=self._payload)
+
+
+def _client(payload, base="http://stub/ctx-test"):
+    c = types.SimpleNamespace(models=_Models(payload))
+    c.base_url = base
+    return c
+
+
+def test_the_worker_reads_the_window_from_an_extra_field(capsys):
+    """`max_model_len` is not in the OpenAI model schema, so the client parks it
+    as an EXTRA field. Reading only attributes found nothing -- and said
+    nothing."""
+    mod = _worker_module()
+    mod._CONTEXT_CACHE.clear()
+    m = types.SimpleNamespace(id="m", model_extra={"max_model_len": 32768})
+    assert mod._context_limit(_client([m], "http://stub/extra"), "m") == 32768
+    assert "32768-token context window" in capsys.readouterr().err
+
+
+def test_a_server_that_names_no_window_says_so(capsys):
+    """The failure that actually happened. It must not pass in silence: the log
+    showed '?' and no reason, which is a degradation indistinguishable from
+    working."""
+    mod = _worker_module()
+    mod._CONTEXT_CACHE.clear()
+    m = types.SimpleNamespace(id="m", model_extra={})
+    assert mod._context_limit(_client([m], "http://stub/silent"), "m") == 0
+    err = capsys.readouterr().err
+    assert "does not report a context window" in err
+    assert "surya_context_tokens" in err, "it does not say what to do about it"
+
+
+def test_the_window_is_asked_for_once_per_server(capsys):
+    mod = _worker_module()
+    mod._CONTEXT_CACHE.clear()
+    calls = []
+
+    class _Counting(_Models):
+        def list(self):
+            calls.append(1)
+            return super().list()
+
+    c = types.SimpleNamespace(
+        models=_Counting([types.SimpleNamespace(id="m",
+                                                model_extra={"max_model_len": 4096})]))
+    c.base_url = "http://stub/once"
+    assert mod._context_limit(c, "m") == 4096
+    assert mod._context_limit(c, "m") == 4096
+    assert len(calls) == 1, "it re-probes a model that cannot change shape"
+
+
+def test_the_backend_on_a_loaded_sub_manager_is_found(resolve):
+    """`list_models` is a method of the MULTI manager, which has no `.backend`
+    of its own — the per-model manager does. Handling only the single-manager
+    shape meant the live path could never fire at all."""
+    inner = _Mgr(backend=types.SimpleNamespace(
+        _served_name="m", _model_id="m", context_length=lambda: 32768))
+    outer = _Mgr()
+    outer.models = {"m": inner}
+    assert resolve(outer, "m", {"runpod": {"ctx": 18432}}) == 32768
+
+
+def test_a_sub_manager_under_an_unrelated_key_is_still_found(resolve):
+    """The dict is keyed by the manager's own scheme (a path, "whisper:x"), so a
+    direct lookup is not guaranteed even when the model IS loaded."""
+    inner = _Mgr(backend=types.SimpleNamespace(
+        _served_name="m", _model_id="m", context_length=lambda: 8192))
+    outer = _Mgr()
+    outer.models = {"/opt/models/some-path.gguf": inner}
+    assert resolve(outer, "m", {}) == 8192
+
+
+def test_a_loaded_sub_manager_for_another_model_is_not_used(resolve):
+    inner = _Mgr(backend=types.SimpleNamespace(
+        _served_name="other", _model_id="other", context_length=lambda: 999999))
+    outer = _Mgr()
+    outer.models = {"other": inner}
+    assert resolve(outer, "m", {"ctx": 4096}) == 4096