• Stefy Lanza (nextime / spora )'s avatar
    ocr: the page budget died at surya's thread pool, and the window at the gateway (v0.3.59) · 035864a4
    Stefy Lanza (nextime / spora ) authored
    Two separate reasons the Digesta re-test still saw wrong caps (12288 and 1360,
    never the 3072 it sent) and "surya-ocr-2 does not report a context window".
    Neither is the pod image, which was blamed for both.
    
    1. surya's `chat_completions_batch` runs every item through a
       ThreadPoolExecutor:
    
           with ThreadPoolExecutor(max_workers=max_workers) as executor:
               return list(executor.map(_process, batch))
    
       so the hook around `_generate_one` executes on an EXECUTOR thread, while
       `_trace_opts` installed the page's options on the lane thread that submitted
       it — and that state is thread-local, deliberately, so two lanes cannot
       attribute one page's logprobs to another. Across that boundary every
       per-request budget read as absent: `max_tokens` was discarded (v0.3.57 fixed
       which value wins, not whether ours arrives), and so were
       `max_continuations` and `context_tokens`. The per-page trace went the same
       way — attempts, token counts, logprobs and accent repairs were recorded on
       the executor thread and never read, so `model_attempts` could not report what
       it exists to report.
    
       The item is the one object that crosses, so it is the key. The lane binds its
       page state to the batch's items before surya dispatches them (hooking
       `chat_completions_batch`, which still runs on the lane thread), and the
       executor thread adopts it by `id(item)`. The state is shared by reference, so
       writes land where the lane reads them. Bindings are released in a finally —
       id() is reused, and a stale entry would hand a new item another page's state.
       When a surya has no such entry point, that is said rather than silently
       leaving the budgets unreachable.
    
    2. An attached engine reads `/v1/models` from the GATEWAY — it talks to
       `…/v1/ocr/attach/<token>/v1` — and the gateway answered with id, object and
       owned_by, nothing else. So 0.3.53's publishing of `max_model_len` was
       invisible on the one path whose arithmetic needs it, the worker correctly
       reported that nobody names a window, and that read as the pod failing to
       advertise one. The pod was never asked. The gateway now publishes the window
       it will serve: a value an upstream reported, else the model's configured ctx.
    
       And the pod now serves the MODEL's ctx instead of this install's. A runpod
       block asking for 18432 got a pod serving 32768, because the pod is handed
       `engine_config_for_pod`, which reads config.json — the local engine's
       settings. Publishing one number while serving another is worse than
       publishing none: resumption stops early, or overruns and takes a 400. The
       per-model value now wins and the difference is logged.
    
       Note for operators: a pod will now serve the per-model `ctx` it was always
       configured with. For surya on the Digesta orchestrator that is 18432, not the
       32768 it had been getting by accident — set ctx higher if the larger window
       was wanted.
    
    A live probe of the upstream's own /v1/models was written and dropped: it built
    a client per page and re-probed for ever against an upstream that reports
    nothing. `learn_window()` remains as the hook for a value we do learn.
    
    15 tests: tests/test_surya_thread_boundary.py (7) and
    tests/test_ocr_window_published.py (8).
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    035864a4
gateway.py 22.9 KB