Third run, third failure, and this one was mine. A 5,000-document run
dispatched ZERO pages to 50 healthy pods in 41 minutes: not one
`POST /v1/ocr` in any pod's log, every GPU at `KV 0.0%`, 811 pages waiting,
and the scaler reading that unservable queue as demand and renting to 73
pods at $43/hr. `/healthz` answered in 1.6 ms the whole time, so the loop
fixes from 0.4.16/0.4.17 were holding and nothing looked wrong from outside.

0.4.18 bounded a lease attempt at 20 s and ran it on a 256-thread pool. That
made the LEASE STAGE the ceiling on everything: 256 threads at one attempt
per 20 s passes ~13 pages a second however many pods are up, and a thread
that drew the growth branch sat through an entire ~9-minute boot. Against
811 pages each retrying every couple of seconds, nothing got through.

So `acquire(grow_inline=False)`: answer now, or start growth in the
background and raise `NoPodRightNow`. The caller waits on its event loop,
where waiting is free, and `POD_LEASE_ATTEMPT_S` drops 20 s -> 2 s for the
pools whose acquire predates the flag. Threaded through
`placement.choose(grow_inline=)` by signature inspection, so a candidate
that cannot take it keeps working.

Demand had to survive that. `_waiting` only ever counted THREADS parked in
acquire, and the non-blocking path has none, so the pool would have stopped
growing. It now counts DENIALS over a window equal to the caller's retry
interval, which makes the count distinct waiting PAGES rather than the
attempts they have made between them -- a page retrying every 2 s appears
once in any 2-second window. A longer window multiplies demand by the retry
rate, which is how the same run read its own retries as new work.

Two more things found on the way:

  * `pod_weight`/`pod_capacity`/`pod_threshold` depend only on a card's NAME,
    but were recomputed per POD per acquire -- `_pick` once per free pod,
    `_pods_for_demand` once per serving pod, `_preferred_vram_gb` walking
    every pod inside each -- all under the pool lock. Memoised per card:
    measured 286us -> 69us per acquire at 70 pods, 528us -> 102us at 128. The
    memo self-initialises via getattr, because a pool built without __init__
    must not have an optimisation turn into an AttributeError.

  * `_pick` returned a bare None for four different causes -- no pods,
    none healthy, all draining, all at their in-flight cap -- which need
    opposite responses, and the caller could only report the queue it could
    not drain. It now says which, once per 15 s, with every count in the line.

I also mis-read `futex_do_wait` on 324 of 352 threads as lock contention. A
condition-variable wait IS a futex wait, so that is what pages legitimately
waiting for a pod look like. The pod logs settled it instead: zero POSTs
means the dispatch never happened, not that it was slow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM