-
Stefy Lanza (nextime / spora ) authored
Reported from the Digesta side as "the pod ignores max_tokens and uses a fixed 12288 cap, loops, and silently drops dense pages", with a pre-fix pod image blamed and the work blocked on a versioned one. The pod is not involved: 12288 appears nowhere in coderai. It is `settings.SURYA_MAX_TOKENS_FULL_PAGE` in the surya library on the ORCHESTRATOR, and surya/recognition builds every full-page item with `max_tokens=` that value. surya's own client then resolves the budget as max_tokens = item.max_tokens or max_tokens_default so `item.max_tokens` was always truthy and the `max_tokens_default` the trace hook rewrote was never read — by anything. `ocr.surya_max_tokens` and a request's own `max_tokens` were both discarded on the vLLM path, which is the path in use. Every page ran at 12288, and on a 32,768-token window that leaves room for about two segments once the ~4,000-token image prefill is counted, so a dense page reached the ceiling mid-answer. That is the "loops and drops pages" half of the same report, and it is why the benchmark numbers taken at "max_tokens=8000/4000/2000" were all the same run. The hook now applies the cap to the item as well, which is the value that decides, and says so once per worker rather than once per page: `[surya] page budget 3072 tokens (overriding surya's own 12288)`. An item that refuses the assignment says that too, instead of leaving the cap quietly unreachable. With no cap configured surya's own budget is untouched — raising it costs tokens on every page. This was already on the list as "surya's own per-page item.max_tokens overrides the request cap — flagged, undecided". The decision: an explicit cap is an instruction and a per-page estimate is a guess, so the instruction wins. What was NOT true in the report, with evidence: the pod image carries coderai 0.3.53 and `git log 42356742..HEAD -- codai/ocr codai/backends codai/api/text.py codai/api/ocr.py codai/models/manager.py` is empty, so it contains every OCR fix; the pod self-reported `coderai=0.3.53 git=42356742` at /boot. A versioned pod tag is still worth having — a cached `:latest` cannot be told apart from a current one by looking — but it was not the blocker. 9 tests in tests/test_surya_item_cap.py. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
8fd4ef08