-
Stefy Lanza (nextime / spora ) authored
Two separate reasons the Digesta re-test still saw wrong caps (12288 and 1360, never the 3072 it sent) and "surya-ocr-2 does not report a context window". Neither is the pod image, which was blamed for both. 1. surya's `chat_completions_batch` runs every item through a ThreadPoolExecutor: with ThreadPoolExecutor(max_workers=max_workers) as executor: return list(executor.map(_process, batch)) so the hook around `_generate_one` executes on an EXECUTOR thread, while `_trace_opts` installed the page's options on the lane thread that submitted it — and that state is thread-local, deliberately, so two lanes cannot attribute one page's logprobs to another. Across that boundary every per-request budget read as absent: `max_tokens` was discarded (v0.3.57 fixed which value wins, not whether ours arrives), and so were `max_continuations` and `context_tokens`. The per-page trace went the same way — attempts, token counts, logprobs and accent repairs were recorded on the executor thread and never read, so `model_attempts` could not report what it exists to report. The item is the one object that crosses, so it is the key. The lane binds its page state to the batch's items before surya dispatches them (hooking `chat_completions_batch`, which still runs on the lane thread), and the executor thread adopts it by `id(item)`. The state is shared by reference, so writes land where the lane reads them. Bindings are released in a finally — id() is reused, and a stale entry would hand a new item another page's state. When a surya has no such entry point, that is said rather than silently leaving the budgets unreachable. 2. An attached engine reads `/v1/models` from the GATEWAY — it talks to `…/v1/ocr/attach/<token>/v1` — and the gateway answered with id, object and owned_by, nothing else. So 0.3.53's publishing of `max_model_len` was invisible on the one path whose arithmetic needs it, the worker correctly reported that nobody names a window, and that read as the pod failing to advertise one. The pod was never asked. The gateway now publishes the window it will serve: a value an upstream reported, else the model's configured ctx. And the pod now serves the MODEL's ctx instead of this install's. A runpod block asking for 18432 got a pod serving 32768, because the pod is handed `engine_config_for_pod`, which reads config.json — the local engine's settings. Publishing one number while serving another is worse than publishing none: resumption stops early, or overruns and takes a 400. The per-model value now wins and the difference is logged. Note for operators: a pod will now serve the per-model `ctx` it was always configured with. For surya on the Digesta orchestrator that is 18432, not the 32768 it had been getting by accident — set ctx higher if the larger window was wanted. A live probe of the upstream's own /v1/models was written and dropped: it built a client per page and re-probed for ever against an upstream that reports nothing. `learn_window()` remains as the hook for a value we do learn. 15 tests: tests/test_surya_thread_boundary.py (7) and tests/test_ocr_window_published.py (8). Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
035864a4