-
Stefy Lanza (nextime / spora ) authored
/v1/ocr bypassed the gate entirely because it is not in the router's _INFERENCE_PATHS — and it should not be, since that predicate drives the per-model queue, pins, spillover and keepalive, none of which fit OCR. But OCR is GPU work: an OCR request could start a surya vLLM boot while an LLM was mid-generation on the same card, and vice versa. The gate is what turns that contention into turns. - is_gpu_inference_path() = is_inference_path() + /v1/ocr and /v1/ocr/batch, and that is what the swap gate keys on now. is_inference_path() is unchanged, so no other front machinery starts applying to OCR. - All OCR shares ONE gate owner ("ocr"): the OCR manager holds one set of pools and evicts between them itself, so paddle and surya are the same tenant for the purpose of who owns the card. It also means the front need not read the `engine` field out of a multipart body carrying a 100 MB PDF. - _computed_off_this_gpu(): the gate AND the local wait queue are skipped when the work is computed elsewhere — a cluster node (engine.remote), a RunPod pod (model backend), or an external API endpoint (every enabled OCR engine pointing off-box). Such a request holds no local VRAM, so a turn on this card's gate would add latency and swaps for nothing. Applied at all four queue sites (direct, keepalive, both brokered) and both gate sites. 15 new tests, the front proxy's first: path classification, the owner key, each off-box case, and that an OCR tenant and a model tenant alternate without starving each other. Co-Authored-By:Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
3cb4d23d