-
Stefy Lanza (nextime / spora ) authored
A 512-page semaphore, a 48-wide admission gate on every pod, 31 A40s at `Running: 1` with `Waiting: 0`, and the orchestrator delivered 1.65 pages/s. Raising `ocr.max_concurrency` 128 -> 512 changed nothing. Nor did replacing the pods' 2-wide gate with 48. The reason both were inert is that neither was ever the limit: 32 threads / 19.4 s per page = 1.65 pages/s Two defects, and the second is why the first was invisible for every release this code has shipped in: * **The per-page forward ran on the DEFAULT executor.** `asyncio.to_thread` hands work to the loop's default pool -- 32 threads, shared with every other blocking call in the process -- while `codai/api/thread_pool.py` exists for exactly this call, documents it ("a per-page OCR forward to a pod (30-300 seconds each)"), and offers `run_on_pod_pool()`. Nothing called it. It was dead code next to the problem it was written for. * **Every sizing helper read a name that does not exist.** `thread_pool._configured_concurrency()` and `pod_http._pool_size()` both did `from codai.config import config_manager`; the shared manager lives in `codai.admin.routes`. The ImportError was swallowed by a bare except, so both answered 0 and took their FLOORS -- 32 threads, 64 connections -- on every install that has ever run, whatever the config said. `ocr.max_concurrency` has never once reached a thread pool. So the two knobs an operator would reach for were the two that could not help, and the number in the boot line ("default pool 32 workers") was the floor wearing the clothes of a configured value. Fixed: * the page forward and the liveness probe go to the pod pool; * sizing reads the holder that exists, tries both, and `sized_from_config()` reports which -- the boot line now says FLOORS out loud when it could not read the config, because "32 workers" means something entirely different when it is chosen than when it is a fallback nobody saw; * `_MAX_POD` 512 -> 2048: ~100 pages/s at ~20 s per page is ~2000 forwards outstanding, and each is a thread blocked on a socket. Past that an async client is the answer, not more stacks; * `configure()` grows the pod pool when `max_concurrency` changes, so raising it mid-run widens the threads and not just the semaphore. Growing only: shrinking would have to stranded threads mid-page. With max_concurrency 2048 the pools are now (256, 2048) rather than (32, 64). Credit where it is due: the client team measured 41-42 established sockets inside the container's network namespace and a 1.65 pages/s ceiling, and said the serialisation was on the orchestrator's OCR path. It was. The contrary reading here came from running `ss` on the HOST, outside the rootless netns, which can only see forwarded sockets -- a measurement of the wrong thing, reported with more confidence than it had earned. 11 new tests, including the one-liner that pins the original bug (`assert not hasattr(codai.config, "config_manager")`) and a static check that the forward is not back on `asyncio.to_thread`. 3324 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
16ca77bb