A client must be able to send too much. The server's job is to carry what
it can and say so about the rest -- not to depend on the client having
been told the right number, which is what the previous fix amounted to.

TWO THINGS WERE MISSING.

1. NOTHING BOUNDED WHAT ONE POD WAS SENT. `max_inflight_per_pod`
   defaults to 0, meaning unlimited, so out of the box every page handed
   to the pool is dispatched. A client offering more than a pod can serve
   therefore does not queue in the orchestrator, where the queue would be
   counted, reported as demand and bounded -- it queues in the POD'S
   LISTEN BACKLOG, and an overflowing backlog makes a healthy pod refuse
   connections. That is what cost eight A40s yesterday.

   The right width is a property of the pod, the model and the card, not
   of the install, so the pod is asked rather than configured. A refusal
   halves what we send that pod, FROM THE IN-FLIGHT COUNT AT THE REFUSAL
   -- its observed breaking point, not a guess -- and successes walk it
   back up one slot at a time (multiplicative decrease, additive
   increase: overshooting costs refused connections and a misdiagnosed
   fleet, undershooting costs a little latency). Floored at 4, never
   above the operator's own ceiling, per pod so one overloaded pod does
   not throttle the fleet, and inert on an install where no pod ever
   refuses. Live numbers: 64 configured, 58 in flight at the refusal,
   width becomes 29 -- the pods were measured serving 29-32.

2. A FULL QUEUE ANSWERED SLOWER INSTEAD OF SAYING NO. The queue itself is
   deliberate: when a pod finishes a page there must already be another
   ready to hand it, or the fleet idles between dispatches and is paid
   for the gap. So the server carries `ocr.max_concurrency` in flight
   PLUS `ocr.max_queued_pages` waiting (0 = follow max_concurrency) --
   and past that answers 429 with Retry-After and the numbers a client
   needs to size its window (X-OCR-Page-Width, X-OCR-Queue-Limit).

   Before, a client offering several times the capacity got no error at
   all: it got replies that took longer and longer, which reads as "the
   pages are getting slower" rather than "stop sending", so it sent more.
   Measured 2026-10-19: 522 pages offered against a fleet carrying ~250,
   mean page latency 238s against a 19s service time, nothing in any
   reply saying the server was over capacity.

   Refused BEFORE the upload is read. The client hears at once instead of
   after transferring a document the server will not look at, and the
   server stops buffering megabytes it is about to discard -- in-flight
   memory here is concurrent requests times upload size, the product that
   has OOM-killed this endpoint before.

Every page now takes its slot through one counted path, which is what
makes the queue depth knowable. Writing that exposed a real defect in
what it replaced: `self._sem or asyncio.Semaphore(4)` built a FRESH
semaphore per page, so an unconfigured manager bounded nothing at all --
forty pages of one document ran at once against what every line of code
called a limit of four. Caught by the existing pod-pipeline gate test.

38 tests. Full suite 3817 passed, 15 pre-existing failures unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM