Three corrections to the fan-out, all the same mistake in different places:
the code was deciding things the configuration had already decided.

Nodes are no longer discovered. Asking every cluster node what it holds and
using whoever answered is configuration by probe: it puts work on a machine
nobody chose and moves it the moment a different node replies first. A node
serves a model because the model is PINNED to it — `engine` on the entry, the
same hard constraint the front router already honours, comma-separated when
several nodes should share the work. A pinned node that does not hold the
model is reported, not silently replaced.

The decision is not an OCR decision, so it does not live in codai/ocr: any
model, any caller that needs an endpoint rather than a proxy hop, gets the
same answer from codai/models/placement.py. For requests that pass THROUGH
coderai this was already true — the router honours the pin, treats a node as
an engine by name, reaches RunPod through the backend. This is that decision
for a caller that must hold an address itself, which is why OCR needed it.

And "how busy is too busy" is the model's own scale_up_inflight_per_pod, on
the model page, instead of the parallel ocr.serve_spill_inflight knob I had
added — one number per model, already there, already meaning exactly this. A
model with no RunPod block has nothing to rent, so load alone never moves its
pages.

Finally, the caller: resolve_engine accepted the four pipeline names only, so
a client naming the model it can see in /v1/models got "unknown OCR engine".
Both now work — engine=surya and engine=datalab-to/surya-ocr-2 reach the same
place — because which pipeline reads a page, and where its model runs, are
ours to know and not the client's.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW