• Stefy Lanza (nextime / spora )'s avatar
    the pod path was 32 threads wide, and said 512 (v0.4.8) · 16ca77bb
    Stefy Lanza (nextime / spora ) authored
    A 512-page semaphore, a 48-wide admission gate on every pod, 31 A40s at
    `Running: 1` with `Waiting: 0`, and the orchestrator delivered 1.65 pages/s.
    Raising `ocr.max_concurrency` 128 -> 512 changed nothing. Nor did replacing the
    pods' 2-wide gate with 48. The reason both were inert is that neither was ever
    the limit:
    
        32 threads / 19.4 s per page = 1.65 pages/s
    
    Two defects, and the second is why the first was invisible for every release
    this code has shipped in:
    
    * **The per-page forward ran on the DEFAULT executor.** `asyncio.to_thread`
      hands work to the loop's default pool -- 32 threads, shared with every other
      blocking call in the process -- while `codai/api/thread_pool.py` exists for
      exactly this call, documents it ("a per-page OCR forward to a pod (30-300
      seconds each)"), and offers `run_on_pod_pool()`. Nothing called it. It was
      dead code next to the problem it was written for.
    
    * **Every sizing helper read a name that does not exist.**
      `thread_pool._configured_concurrency()` and `pod_http._pool_size()` both did
      `from codai.config import config_manager`; the shared manager lives in
      `codai.admin.routes`. The ImportError was swallowed by a bare except, so both
      answered 0 and took their FLOORS -- 32 threads, 64 connections -- on every
      install that has ever run, whatever the config said. `ocr.max_concurrency`
      has never once reached a thread pool.
    
    So the two knobs an operator would reach for were the two that could not
    help, and the number in the boot line ("default pool 32 workers") was the
    floor wearing the clothes of a configured value.
    
    Fixed:
    
    * the page forward and the liveness probe go to the pod pool;
    * sizing reads the holder that exists, tries both, and `sized_from_config()`
      reports which -- the boot line now says FLOORS out loud when it could not
      read the config, because "32 workers" means something entirely different
      when it is chosen than when it is a fallback nobody saw;
    * `_MAX_POD` 512 -> 2048: ~100 pages/s at ~20 s per page is ~2000 forwards
      outstanding, and each is a thread blocked on a socket. Past that an async
      client is the answer, not more stacks;
    * `configure()` grows the pod pool when `max_concurrency` changes, so raising
      it mid-run widens the threads and not just the semaphore. Growing only:
      shrinking would have to stranded threads mid-page.
    
    With max_concurrency 2048 the pools are now (256, 2048) rather than (32, 64).
    
    Credit where it is due: the client team measured 41-42 established sockets
    inside the container's network namespace and a 1.65 pages/s ceiling, and said
    the serialisation was on the orchestrator's OCR path. It was. The contrary
    reading here came from running `ss` on the HOST, outside the rootless netns,
    which can only see forwarded sockets -- a measurement of the wrong thing,
    reported with more confidence than it had earned.
    
    11 new tests, including the one-liner that pins the original bug
    (`assert not hasattr(codai.config, "config_manager")`) and a static check that
    the forward is not back on `asyncio.to_thread`. 3324 passed, 13 pre-existing
    failures (broker, studio, manual client) untouched.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    16ca77bb
test_ocr_pod_pool_width.py 6.16 KB