one page waiting for a pod stopped the whole server (v0.4.16)

A 5,000-document run rented 127 pods, reached $79/hr and completed zero
documents in ten minutes. The front had stopped answering entirely:
/v1/runpod/status timed out on every port, which is also what the client
had been reporting as thousands of timeouts.

The pods were fine. Probed from inside the container, both transports
answered /healthz in milliseconds, vLLM was loaded on each one, and the
GPUs sat at KV 0.0% with no surya worker line, because no page had ever
reached them.

`placement.choose` is synchronous and ends in a pool's `acquire()`, which
waits on a condition variable for up to its default 1200 s and may rent a
pod INLINE -- a candidate sweep over every region, then a boot wait, in
whichever thread asked. It was called straight from `_read_on_pods`'s
coroutine, so it ran on the event loop and took the whole process with it
for as long as one page waited for a pod.

It has been there since the pod pipeline was written (0.3.70) and cost
nothing while acquire() returned at once: a pool with a free pod answers in
microseconds. It turns fatal exactly when the pool is the thing under
strain, which is when it matters most.

So the lease runs on the pod pool now, and because a pod takes MINUTES to
boot, three things follow:

  * One attempt is bounded (`POD_LEASE_ATTEMPT_S`, 20 s), threaded through
    `placement.choose(acquire_timeout=)` to `pool.acquire(timeout=)`. An
    unbounded attempt parks a thread for a whole boot, per page; bounded, the
    waiting happens in `_lease_a_pod`'s own loop where it is free.
  * `ocr.pod_pipeline_wait_s` goes 60 s -> 600 s. The 60 s was chosen because
    a waiting page held a thread, so a long ceiling meant thousands of parked
    threads. It no longer does, and at 60 s a cold start handed its whole
    first wave of pages to a GPU-less orchestrator to rasterise -- the exact
    failure `prefer` exists to prevent.
  * The per-page PDF split moves to the DEFAULT executor, not the pod pool:
    pdfium permits one caller at a time, and putting a split behind thousands
    of leases waiting for a boot is a cold start that cannot produce the next
    page while it waits for the pods that page is for.

The forward itself stays on the event loop, which is the point of 0.4.8 and
is now covered by a test that fails if it moves back.

All of this is invisible to a test that awaits one page at a time: one page
blocking the loop for 0.5 s looks exactly like one page taking 0.5 s. The new
tests measure the longest gap between two ticks of a coroutine running
alongside, and the thread-seconds held while waiting out a boot. Five of them
fail on 0.4.15.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.15"
+__version__ = "0.4.16"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/config.py

@@ -1066,11 +1066,21 @@ class OcrConfig:
     # install where the pods exist at all.
     pod_pipeline: str = "prefer"
     # How long `prefer` (and `require`) wait for a pod before giving up on one.
-    # A pod takes minutes to boot, so this is deliberately NOT long enough to
-    # cover a cold pool: it covers a pool that is momentarily full. After a
-    # wait times out the next pages go straight through for a cooldown rather
-    # than paying the ceiling one page at a time.
-    pod_pipeline_wait_s: float = 60.0
+    #
+    # This was 60 s through 0.4.15, deliberately too short to cover a cold pool,
+    # because a waiting page held a POD-POOL THREAD for the whole wait: a long
+    # ceiling meant thousands of parked threads, so the cheaper mistake was to
+    # stop waiting and read the page here.
+    #
+    # It no longer costs a thread. The lease waits on the event loop and only
+    # each attempt touches a thread (`POD_LEASE_ATTEMPT_S`), so a page can wait
+    # out an actual boot for free -- and a boot IS minutes: image pull, vLLM
+    # load, weights. At 60 s a cold start therefore handed the whole first wave
+    # of pages to this box, one GPU-less orchestrator rasterising for a fleet it
+    # was already paying for, which is the failure mode `prefer` exists to
+    # prevent. Ten minutes covers a boot with margin; an install whose pods are
+    # always warm never reaches it, because a free pod answers at once.
+    pod_pipeline_wait_s: float = 600.0
     # --- how a page is handed to the model ---
     # Measured on A4 at 200 DPI (3.4-3.9 Mpix): rendering a page costs 21-46 ms
     # and ENCODING it 86-353 ms, so three quarters of what is called

codai/models/placement.py

@@ -399,8 +399,9 @@ def _spill_pod_candidate(model_id: str) -> Optional[Candidate]:
     if pool is None:
         return None
 
-    def _acquire():
-        pod, url = pool.acquire()
+    def _acquire(timeout: float = 0.0):
+        # The pool's own default (1200 s) unless the caller bounds the attempt.
+        pod, url = pool.acquire(timeout=timeout) if timeout > 0 else pool.acquire()
         key = getattr(pool, "api_key", "") or ""
         return (url.rstrip("/") + "/v1", key, lambda: pool.release(pod),
                 lambda: pool.discard(pod, "connection refused"))
@@ -473,13 +474,18 @@ def _load_of(cand: Candidate) -> int:
     return inflight(cand.key) + _node_reported_load(cand)
 
 
-def choose(model_id: str, spill_after: int = 0) -> Lease:
+def choose(model_id: str, spill_after: int = 0,
+           acquire_timeout: float = 0.0) -> Lease:
     """A lease on the least-loaded server for the model.
 
     ``spill_after`` is the in-flight count at which a fixed server (the local one,
     a node) is considered busy enough to prefer renting instead — the
     local-plus-spillover policy, expressed where OCR can act on it. 0 means never
     spill on load alone; a pool is then used only when it is all there is.
+
+    ``acquire_timeout`` bounds each pool attempt (see :func:`_take`). This
+    function BLOCKS for as long as the pools it tries do, so a caller on an
+    event loop must both run it off the loop and bound it.
     """
     cands = candidates(model_id)
     if not cands:
@@ -505,7 +511,7 @@ def choose(model_id: str, spill_after: int = 0) -> Lease:
     #    pages over the pods it has and grows when they are all at their
     #    threshold. This is "on runpod only", scaling.
     for cand in [c for c in cands if c.primary and c.acquire is not None]:
-        lease = _take(cand)
+        lease = _take(cand, acquire_timeout)
         if lease is not None:
             return lease
         last_exc = "pool unavailable"
@@ -527,7 +533,7 @@ def choose(model_id: str, spill_after: int = 0) -> Lease:
     # 3. Everything we own is busy: rent. This is "local + spillover" — the
     #    overflow goes to a pool instead of queueing behind a saturated GPU.
     for cand in pools:
-        lease = _take(cand)
+        lease = _take(cand, acquire_timeout)
         if lease is not None:
             return lease
         last_exc = "pool unavailable"
@@ -543,12 +549,51 @@ def choose(model_id: str, spill_after: int = 0) -> Lease:
                    f"({last_exc})", status=503)
 
 
-def _take(cand: Candidate) -> Optional[Lease]:
-    """Acquire a pool candidate, or None when it cannot provide a server."""
+def _accepts_timeout(fn) -> bool:
+    """Does this candidate's acquire take a `timeout`?
+
+    Asked of the SIGNATURE rather than by calling it and catching TypeError: a
+    TypeError raised from inside a real acquire would otherwise be read as "it
+    does not accept a timeout" and the pod acquired twice.
+    """
+    try:
+        import inspect
+        return "timeout" in inspect.signature(fn).parameters
+    except Exception:
+        return False
+
+
+#: Pool key -> when its "cannot serve" line may be said again. A bounded
+#: acquire TIMES OUT per page while a pod boots, and a boot is minutes: without
+#: this, a cold start writes one line per page per attempt and buries the
+#: reason it is waiting at all.
+_SAID: dict = {}
+
+
+def _say_once(key: str, message: str, every: float = 30.0) -> None:
+    now = time.time()
+    if now < _SAID.get(key, 0.0):
+        return
+    _SAID[key] = now + every
+    print(message, flush=True)
+
+
+def _take(cand: Candidate, acquire_timeout: float = 0.0) -> Optional[Lease]:
+    """Acquire a pool candidate, or None when it cannot provide a server.
+
+    ``acquire_timeout`` bounds ONE attempt. A pool's acquire waits for a pod and
+    may rent one inline -- minutes, for a cold boot -- and the caller holds a
+    thread for all of it. Bounding the attempt lets the caller do its waiting
+    where waiting is free (the event loop) and come back, instead of parking a
+    thread per page for the length of a boot. 0 keeps the pool's own default.
+    """
     try:
-        got = cand.acquire()
+        got = (cand.acquire(timeout=acquire_timeout)
+               if acquire_timeout > 0 and _accepts_timeout(cand.acquire)
+               else cand.acquire())
     except Exception as exc:
-        print(f"[ocr] {cand.key} cannot serve this page: {exc}", flush=True)
+        _say_once(f"take:{cand.key}",
+                  f"[ocr] {cand.key} cannot serve this page: {exc}")
         return None
     # A pool may also hand back a way to say "this one is dead"; older
     # three-value forms simply have none.

codai/ocr/manager.py

@@ -744,6 +744,14 @@ def _pod_pipeline_policy(cfg) -> tuple:
 #: budget, or at its ceiling, stays that way for minutes: without this every
 #: page of every document would pay the full wait before going local, turning a
 #: 60s ceiling into a 60s tax per page.
+#: How long ONE lease attempt may hold a pod-pool thread. A pool's acquire
+#: waits for a free pod and may rent one inline, so an unbounded attempt parks a
+#: thread for a whole boot -- several minutes on a cold start, times every page
+#: of every document in flight. Bounded, the attempt comes back and the waiting
+#: happens in `_lease_a_pod`'s own loop, on the event loop, where it is free and
+#: where the policy's ceiling is what decides how long to keep trying.
+POD_LEASE_ATTEMPT_S = 20.0
+
 _POD_WAIT_COOLDOWN: dict = {}
 POD_WAIT_COOLDOWN_S = 300.0
 
@@ -778,8 +786,26 @@ async def _lease_a_pod(model_id: str, policy: str, wait_s: float):
     waited = 0.0
     while True:
         try:
-            lease = placement.choose(
-                model_id, spill_after=placement.spill_threshold(model_id))
+            # OFF THE LOOP. `placement.choose` is synchronous and ends in a
+            # pool's `acquire()`, which waits on a condition variable for up to
+            # its default 1200 s and may rent a pod INLINE -- a candidate sweep
+            # over every region, then a boot wait, in whichever thread asked.
+            # Called straight from this coroutine it blocked the whole event
+            # loop, so the process answered nothing at all -- not the client's
+            # next document, not even /v1/runpod/status -- for as long as one
+            # page waited for a pod.
+            #
+            # It has been on the loop since the pod pipeline was written
+            # (0.3.70) and cost nothing for as long as acquire() returned at
+            # once: a pool with a free pod answers in microseconds. It turns
+            # fatal exactly when the pool is the thing under strain, which is
+            # when it matters most. Measured on a 5,000-document run: 116 pods
+            # rented, zero documents completed in ten minutes, and a front that
+            # had stopped answering.
+            lease = await run_on_pod_pool(
+                placement.choose, model_id,
+                spill_after=placement.spill_threshold(model_id),
+                acquire_timeout=POD_LEASE_ATTEMPT_S)
         except OcrError:
             raise
         except Exception as exc:
@@ -2113,7 +2139,17 @@ class OcrManager:
                 f"{POD_PIPELINE_COOLDOWN_S:.0f}s cooldown expires",
                 every=POD_PIPELINE_COOLDOWN_S)
             return None
-        parts = (list(iter_pdf_page_pdfs(data)) if is_pdf else [data])
+        # Also off the loop: pdfium permits exactly ONE caller at a time, so
+        # every document's split serialises against every other one, and doing
+        # that on the loop stalls every request in the process for the duration
+        # rather than only this document.
+        #
+        # The DEFAULT executor, not the pod pool. The pod pool is where leases
+        # wait, and a lease waits for a pod to BOOT -- minutes. Putting a split
+        # behind thousands of those is a cold start that cannot produce the next
+        # page while it waits for the pods that page is for.
+        parts = (await asyncio.to_thread(lambda: list(iter_pdf_page_pdfs(data)))
+                 if is_pdf else [data])
         if not parts:
             return None
         sem = self._sem or asyncio.Semaphore(4)

tests/test_ocr_dual_mode.py

@@ -309,7 +309,7 @@ def _fake_placement(monkeypatch, kind="pod", on_choose=None):
         def release(self):
             pass
 
-    def _choose(mid, spill_after=None):
+    def _choose(mid, spill_after=None, **_kw):
         if on_choose is not None:
             on_choose()
         return _Lease()