-
Stefy Lanza (nextime / spora ) authored
Multiple loaded models must each have their own queue — one model's backlog or GPU ownership must never block requests for another model that is loaded and idle: - queue admission is per model (is_full_for): 429 only when the REQUEST'S model already has queue_max_size waiters (global 4× backstop bounds memory); previously ONE hot model's backlog 429'd every other model. - GpuSwapGate: requests for a model already RESIDENT on the engine bypass the ownership gate — no swap is needed and its own per-model queue governs concurrency. The gate now serializes only requests that would actually trigger a model swap (its real purpose). Three co-resident embedders can each serve a request in parallel. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
e9131f38