-
Stefy Lanza (nextime / spora ) authored
Multiple loaded models must each have their own queue — one model's backlog or GPU ownership must never block requests for another model that is loaded and idle: - queue admission is per model (is_full_for): 429 only when the REQUEST'S model already has queue_max_size waiters (global 4× backstop bounds memory); previously ONE hot model's backlog 429'd every other model. - GpuSwapGate: requests for a model already RESIDENT on the engine bypass the ownership gate — no swap is needed and its own per-model queue governs concurrency. The gate now serializes only requests that would actually trigger a model swap (its real purpose). Three co-resident embedders can each serve a request in parallel. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
e9131f38
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| admin | ||
| api | ||
| backends | ||
| broker | ||
| frontproxy | ||
| models | ||
| openai | ||
| pydantic | ||
| queue | ||
| tasks | ||
| __init__.py | ||
| cli.py | ||
| config.py | ||
| main.py | ||
| platform_paths.py | ||
| system_app.py |