An OCR pipeline is handed one server address and drives it itself, which is
what keeps its page images, its strict json_schema and its logprobs intact.
One address cannot spread load, and three things followed from that:

  * a pool grows on in-flight requests counted as they pass through it. An
    attached engine never passed through, so the pool saw no load however hard
    OCR ran, never grew, and the engine had one pod's URL anyway;
  * several cluster nodes serving the model resolved to whichever answered
    first, for ever, however busy it became;
  * a local model configured to burst to RunPod never did: that trigger lives
    on the proxy path an attached engine bypasses.

So the engine gets a lease on an address of ours, and codai/ocr/dispatch.py
chooses per request: a pool that is the model's configured home takes every
page (picking the least-loaded pod, renting when they are all at the
threshold); otherwise this machine and the nodes holding the model compete on
load — including the load a node reports for its whole install, so a node busy
with someone else's work is left alone — and the overflow past
ocr.serve_spill_inflight rents instead of queueing. An explicit `backend` pin
still wins over discovery: that was a decision, and load-aware picking must
not quietly repeal it.

The hop is allowed to exist only because it decides nothing: body bytes in,
body bytes out. The losses it is there to avoid came from PARSING a request
and re-emitting it through the backend abstraction.

Two gates stood in the way and neither could be satisfied the usual way: the
engine has no API token, and surya's /health and /v1/models probes send no
headers at all (bare httpx GETs), while refusing to attach if /health is not
200. The lease is therefore read from the URL, and the engine-only internal
gate exempts that path alone — tested to be no wider than that.

olmOCR's model mode asks the same dispatcher, and still takes the in-process
route when the answer is local, where the model manager's loading, eviction
and queueing belong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW