• Stefy Lanza (nextime / spora )'s avatar
    front: OCR takes its turn at the shared-GPU swap gate, unless it runs off-box · 3cb4d23d
    Stefy Lanza (nextime / spora ) authored
    /v1/ocr bypassed the gate entirely because it is not in the router's _INFERENCE_PATHS
    — and it should not be, since that predicate drives the per-model queue, pins,
    spillover and keepalive, none of which fit OCR. But OCR is GPU work: an OCR request
    could start a surya vLLM boot while an LLM was mid-generation on the same card, and
    vice versa. The gate is what turns that contention into turns.
    
    - is_gpu_inference_path() = is_inference_path() + /v1/ocr and /v1/ocr/batch, and that
      is what the swap gate keys on now. is_inference_path() is unchanged, so no other
      front machinery starts applying to OCR.
    - All OCR shares ONE gate owner ("ocr"): the OCR manager holds one set of pools and
      evicts between them itself, so paddle and surya are the same tenant for the purpose
      of who owns the card. It also means the front need not read the `engine` field out of
      a multipart body carrying a 100 MB PDF.
    - _computed_off_this_gpu(): the gate AND the local wait queue are skipped when the work
      is computed elsewhere — a cluster node (engine.remote), a RunPod pod (model backend),
      or an external API endpoint (every enabled OCR engine pointing off-box). Such a
      request holds no local VRAM, so a turn on this card's gate would add latency and
      swaps for nothing. Applied at all four queue sites (direct, keepalive, both brokered)
      and both gate sites.
    
    15 new tests, the front proxy's first: path classification, the owner key, each off-box
    case, and that an OCR tenant and a model tenant alternate without starving each other.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    3cb4d23d
router.py 13 KB