ocr: the page budget never reached surya, so every page ran at its 12288 (v0.3.57)

Reported from the Digesta side as "the pod ignores max_tokens and uses a fixed
12288 cap, loops, and silently drops dense pages", with a pre-fix pod image
blamed and the work blocked on a versioned one. The pod is not involved: 12288
appears nowhere in coderai. It is `settings.SURYA_MAX_TOKENS_FULL_PAGE` in the
surya library on the ORCHESTRATOR, and surya/recognition builds every full-page
item with `max_tokens=` that value.

surya's own client then resolves the budget as

    max_tokens = item.max_tokens or max_tokens_default

so `item.max_tokens` was always truthy and the `max_tokens_default` the trace
hook rewrote was never read — by anything. `ocr.surya_max_tokens` and a
request's own `max_tokens` were both discarded on the vLLM path, which is the
path in use. Every page ran at 12288, and on a 32,768-token window that leaves
room for about two segments once the ~4,000-token image prefill is counted, so
a dense page reached the ceiling mid-answer. That is the "loops and drops
pages" half of the same report, and it is why the benchmark numbers taken at
"max_tokens=8000/4000/2000" were all the same run.

The hook now applies the cap to the item as well, which is the value that
decides, and says so once per worker rather than once per page:
`[surya] page budget 3072 tokens (overriding surya's own 12288)`. An item that
refuses the assignment says that too, instead of leaving the cap quietly
unreachable. With no cap configured surya's own budget is untouched — raising
it costs tokens on every page.

This was already on the list as "surya's own per-page item.max_tokens overrides
the request cap — flagged, undecided". The decision: an explicit cap is an
instruction and a per-page estimate is a guess, so the instruction wins.

What was NOT true in the report, with evidence: the pod image carries coderai
0.3.53 and `git log 42356742..HEAD -- codai/ocr codai/backends codai/api/text.py
codai/api/ocr.py codai/models/manager.py` is empty, so it contains every OCR
fix; the pod self-reported `coderai=0.3.53 git=42356742` at /boot. A versioned
pod tag is still worth having — a cached `:latest` cannot be told apart from a
current one by looking — but it was not the blocker.

9 tests in tests/test_surya_item_cap.py.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.56"
+__version__ = "0.3.57"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/ocr/workers/ocr_worker.py

@@ -83,6 +83,9 @@ _CONTEXT_CACHE = {}
 #: number only has to be an over-estimate, because its whole job is to keep the
 #: LAST segment inside the window.
 _PROMPT_TOKENS_GUESS = 4096
+#: Said once per worker, not per page: 8 lanes x hundreds of pages would bury
+#: the log, and the number is the same every time.
+_CAP_NOTED = [False]
 
 #: Below this there is no point asking for more: a segment this short cannot
 #: close an open block, so it would spend a round to add nothing. It is a floor
@@ -1147,6 +1150,36 @@ class SuryaWorker:
             cap = _page_override("max_tokens", _MAX_TOKENS[0])
             if cap and "max_tokens_default" in kwargs:
                 kwargs["max_tokens_default"] = int(cap)
+            if cap:
+                # …and the PER-ITEM budget, which is the one that actually
+                # decides. surya's own client does
+                #     max_tokens = item.max_tokens or max_tokens_default
+                # and its full-page path builds every item with
+                #     max_tokens=settings.SURYA_MAX_TOKENS_FULL_PAGE   # 12288
+                # so `item.max_tokens` was always set and the default we just
+                # rewrote was never read. Every configured or per-request
+                # max_tokens was silently discarded, and each page ran at
+                # surya's 12288 — which on a 32k window leaves room for two
+                # segments, so a dense page hit the ceiling mid-answer. It read
+                # from outside like the pod ignoring the field; the pod never
+                # sees it.
+                _it = args[0] if args else kwargs.get("item")
+                _own = int(getattr(_it, "max_tokens", 0) or 0)
+                if _it is not None and _own != int(cap):
+                    try:
+                        _it.max_tokens = int(cap)
+                    except Exception as exc:
+                        # A frozen item would make the cap unreachable again, so
+                        # this must be visible rather than silently ignored.
+                        print(f"[surya] could not apply the {cap}-token cap to "
+                              f"the page item ({exc}) — surya's own {_own} "
+                              f"stands", file=sys.stderr, flush=True)
+                    else:
+                        if not _CAP_NOTED[0]:
+                            _CAP_NOTED[0] = True
+                            print(f"[surya] page budget {cap} tokens "
+                                  f"(overriding surya's own {_own})",
+                                  file=sys.stderr, flush=True)
             result = inner(*args, **kwargs)
             # A reply that ran out of room is finished here, before surya parses
             # it: raising the cap only moves the cliff, and incomplete block

docs/ocr.md

@@ -1265,10 +1265,26 @@ A dense page can legitimately need several thousand tokens, and a batch measured
 on 2026-10-13 averaged **2,644 generated tokens per request**, i.e. above the
 cap.
 
-`ocr.surya_max_tokens` raises it (0 = leave surya's 2048). It is opt-in because
-a higher cap costs tokens on every page and the right value depends on the
-documents. What is **not** optional is being able to tell the two cases apart,
-so every page that hit the cap says so in its own result:
+`ocr.surya_max_tokens` raises it (0 = leave surya's own budget). It is opt-in
+because a higher cap costs tokens on every page and the right value depends on
+the documents.
+
+> **Up to v0.3.56 that setting, and a request's own `max_tokens`, were
+> silently discarded on the vLLM path.** surya resolves a page's budget as
+> `item.max_tokens or max_tokens_default`, and its full-page path builds every
+> item with `max_tokens=settings.SURYA_MAX_TOKENS_FULL_PAGE` — **12288** — so
+> the per-item value was always set and the `max_tokens_default` coderai
+> rewrote was never read. Every page ran at 12288 whatever was configured, and
+> on a 32,768-token window that leaves room for about two segments once the
+> image prefill is counted, so a dense page reached the ceiling mid-answer.
+> From outside it looked exactly like the *server* ignoring `max_tokens` — it
+> was reported that way, against the pod, which never sees the field. v0.3.57
+> applies the cap to the item as well, and says so once per worker:
+> `[surya] page budget 3072 tokens (overriding surya's own 12288)`. With no cap
+> configured, surya's own budget is left alone.
+
+What is **not** optional is being able to tell the two cases apart, so every
+page that hit the cap says so in its own result:
 
 ```json
 "model_attempts": {

tests/test_surya_item_cap.py

+"""The configured max_tokens must beat surya's own per-item budget.
+
+Reported from the Digesta side as "the pod ignores max_tokens and uses a fixed
+12288 cap, loops, and silently drops dense pages", with the pod blamed. The pod
+has nothing to do with it. 12288 appears nowhere in coderai: it is
+`settings.SURYA_MAX_TOKENS_FULL_PAGE` in the surya library running on the
+orchestrator, and `surya/recognition/__init__.py` builds every full-page item
+with `max_tokens=` that value. surya's own client then resolves the budget as
+
+    max_tokens = item.max_tokens or max_tokens_default
+
+so `item.max_tokens` was always truthy and the `max_tokens_default` our hook
+rewrote was never read. Every configured and per-request cap was discarded.
+
+On a 32k window a 12288 budget leaves room for about two segments once the
+image prefill is counted, so a dense page reached the ceiling mid-answer — which
+is the "loops and drops pages" part of the same report.
+"""
+import pathlib
+import sys
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+sys.path.insert(0, str(ROOT / "tests"))
+
+from test_surya_retry_budget import (_fake_surya, _worker,           # noqa: E402
+                                     _worker_module, _Result)
+
+
+class _Item:
+    """A surya request item, which carries its own token budget."""
+
+    def __init__(self, max_tokens=12288):
+        self.max_tokens = max_tokens
+
+
+def _hooked(monkeypatch, cap, item, kw=None):
+    mod = _worker_module()
+    oc = _fake_surya(monkeypatch, mod)
+    mod._CAP_NOTED[0] = False
+    seen = {}
+
+    def _gen(*a, **k):
+        seen["args"], seen["kwargs"] = a, dict(k)
+        return _Result()
+
+    oc._generate_one = _gen
+    w = _worker(mod, token_trace=True, max_tokens=cap)
+    w._install_trace_hook()
+    oc._generate_one(item, **(kw or {}))
+    return seen
+
+
+def test_the_configured_cap_replaces_suryas_12288(monkeypatch):
+    item = _Item(12288)
+    _hooked(monkeypatch, 3072, item, {"max_tokens_default": 2048})
+    assert item.max_tokens == 3072
+
+
+def test_it_wins_even_without_max_tokens_default_in_the_call(monkeypatch):
+    # The guard on the default was `"max_tokens_default" in kwargs`; the item
+    # must be capped whether or not that keyword is there.
+    item = _Item(12288)
+    _hooked(monkeypatch, 3072, item)
+    assert item.max_tokens == 3072
+
+
+def test_a_cap_larger_than_suryas_own_also_applies(monkeypatch):
+    # Raising it is the documented way to fit a denser page in fewer segments.
+    item = _Item(12288)
+    _hooked(monkeypatch, 16384, item, {"max_tokens_default": 2048})
+    assert item.max_tokens == 16384
+
+
+def test_no_cap_leaves_suryas_own_budget_untouched(monkeypatch):
+    """Opt-in: surya's per-page estimate is a reasonable default and changing
+    it costs tokens on every page."""
+    item = _Item(770)
+    _hooked(monkeypatch, 0, item, {"max_tokens_default": 2048})
+    assert item.max_tokens == 770
+
+
+def test_the_per_page_override_reaches_the_item(monkeypatch):
+    """A /v1/ocr request's own max_tokens, not just the configured one."""
+    mod = _worker_module()
+    oc = _fake_surya(monkeypatch, mod)
+    mod._CAP_NOTED[0] = False
+    oc._generate_one = lambda *a, **k: _Result()
+    w = _worker(mod, token_trace=True, max_tokens=0)
+    w._install_trace_hook()
+    item = _Item(12288)
+    # Per-page overrides are thread-local (several lanes share one hook).
+    mod._TRACE.opts = {"max_tokens": 4096}
+    try:
+        oc._generate_one(item, max_tokens_default=2048)
+    finally:
+        mod._TRACE.opts = {}
+    assert item.max_tokens == 4096
+
+
+def test_the_default_is_still_rewritten_for_an_item_without_one(monkeypatch):
+    # An item whose own budget is 0 falls through to the default in surya's
+    # `or`, so that path has to keep working.
+    item = _Item(0)
+    seen = _hooked(monkeypatch, 3072, item, {"max_tokens_default": 2048})
+    assert seen["kwargs"]["max_tokens_default"] == 3072
+    assert item.max_tokens == 3072
+
+
+def test_the_override_is_announced_once_not_once_per_page(monkeypatch, capsys):
+    mod = _worker_module()
+    oc = _fake_surya(monkeypatch, mod)
+    mod._CAP_NOTED[0] = False
+    oc._generate_one = lambda *a, **k: _Result()
+    w = _worker(mod, token_trace=True, max_tokens=3072)
+    w._install_trace_hook()
+    for _ in range(3):
+        oc._generate_one(_Item(12288), max_tokens_default=2048)
+    err = capsys.readouterr().err
+    # Eight lanes over hundreds of pages would bury the log in a line whose
+    # numbers never change.
+    assert err.count("page budget 3072 tokens") == 1
+    assert "overriding surya's own 12288" in err
+
+
+def test_an_unsettable_item_says_so_instead_of_pretending(monkeypatch, capsys):
+    class _Frozen:
+        max_tokens = 12288
+
+        def __setattr__(self, k, v):
+            raise AttributeError("frozen")
+
+    _hooked(monkeypatch, 3072, _Frozen(), {"max_tokens_default": 2048})
+    err = capsys.readouterr().err
+    assert "could not apply the 3072-token cap" in err and "12288 stands" in err
+
+
+def test_a_missing_item_is_not_an_error(monkeypatch):
+    # _generate_one is also called with no positional item in some paths.
+    mod = _worker_module()
+    oc = _fake_surya(monkeypatch, mod)
+    mod._CAP_NOTED[0] = False
+    oc._generate_one = lambda *a, **k: _Result()
+    w = _worker(mod, token_trace=True, max_tokens=3072)
+    w._install_trace_hook()
+    oc._generate_one(max_tokens_default=2048)      # must not raise