• Stefy Lanza (nextime / spora )'s avatar
    per-model queueing: parallel serving across co-loaded models · e9131f38
    Stefy Lanza (nextime / spora ) authored
    Multiple loaded models must each have their own queue — one model's
    backlog or GPU ownership must never block requests for another model
    that is loaded and idle:
    
    - queue admission is per model (is_full_for): 429 only when the
      REQUEST'S model already has queue_max_size waiters (global 4×
      backstop bounds memory); previously ONE hot model's backlog 429'd
      every other model.
    - GpuSwapGate: requests for a model already RESIDENT on the engine
      bypass the ownership gate — no swap is needed and its own per-model
      queue governs concurrency. The gate now serializes only requests
      that would actually trigger a model swap (its real purpose). Three
      co-resident embedders can each serve a request in parallel.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
    e9131f38
__init__.py 4.11 KB