• Stefy Lanza (nextime / spora )'s avatar
    per-model queueing: parallel serving across co-loaded models · e9131f38
    Stefy Lanza (nextime / spora ) authored
    Multiple loaded models must each have their own queue — one model's
    backlog or GPU ownership must never block requests for another model
    that is loaded and idle:
    
    - queue admission is per model (is_full_for): 429 only when the
      REQUEST'S model already has queue_max_size waiters (global 4×
      backstop bounds memory); previously ONE hot model's backlog 429'd
      every other model.
    - GpuSwapGate: requests for a model already RESIDENT on the engine
      bypass the ownership gate — no swap is needed and its own per-model
      queue governs concurrency. The gate now serializes only requests
      that would actually trigger a model swap (its real purpose). Three
      co-resident embedders can each serve a request in parallel.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
    e9131f38
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...