A 5,000-document run rented 127 pods, reached $79/hr and completed zero
documents in ten minutes. The front had stopped answering entirely:
/v1/runpod/status timed out on every port, which is also what the client
had been reporting as thousands of timeouts.

The pods were fine. Probed from inside the container, both transports
answered /healthz in milliseconds, vLLM was loaded on each one, and the
GPUs sat at KV 0.0% with no surya worker line, because no page had ever
reached them.

`placement.choose` is synchronous and ends in a pool's `acquire()`, which
waits on a condition variable for up to its default 1200 s and may rent a
pod INLINE -- a candidate sweep over every region, then a boot wait, in
whichever thread asked. It was called straight from `_read_on_pods`'s
coroutine, so it ran on the event loop and took the whole process with it
for as long as one page waited for a pod.

It has been there since the pod pipeline was written (0.3.70) and cost
nothing while acquire() returned at once: a pool with a free pod answers in
microseconds. It turns fatal exactly when the pool is the thing under
strain, which is when it matters most.

So the lease runs on the pod pool now, and because a pod takes MINUTES to
boot, three things follow:

  * One attempt is bounded (`POD_LEASE_ATTEMPT_S`, 20 s), threaded through
    `placement.choose(acquire_timeout=)` to `pool.acquire(timeout=)`. An
    unbounded attempt parks a thread for a whole boot, per page; bounded, the
    waiting happens in `_lease_a_pod`'s own loop where it is free.
  * `ocr.pod_pipeline_wait_s` goes 60 s -> 600 s. The 60 s was chosen because
    a waiting page held a thread, so a long ceiling meant thousands of parked
    threads. It no longer does, and at 60 s a cold start handed its whole
    first wave of pages to a GPU-less orchestrator to rasterise -- the exact
    failure `prefer` exists to prevent.
  * The per-page PDF split moves to the DEFAULT executor, not the pod pool:
    pdfium permits one caller at a time, and putting a split behind thousands
    of leases waiting for a boot is a cold start that cannot produce the next
    page while it waits for the pods that page is for.

The forward itself stays on the event loop, which is the point of 0.4.8 and
is now covered by a test that fails if it moves back.

All of this is invisible to a test that awaits one page at a time: one page
blocking the loop for 0.5 s looks exactly like one page taking 0.5 s. The new
tests measure the longest gap between two ticks of a coroutine running
alongside, and the thread-seconds held while waiting out a boot. Five of them
fail on 0.4.15.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM