• Stefy Lanza (nextime / spora )'s avatar
    per-model queueing: parallel serving across co-loaded models · e9131f38
    Stefy Lanza (nextime / spora ) authored
    Multiple loaded models must each have their own queue — one model's
    backlog or GPU ownership must never block requests for another model
    that is loaded and idle:
    
    - queue admission is per model (is_full_for): 429 only when the
      REQUEST'S model already has queue_max_size waiters (global 4×
      backstop bounds memory); previously ONE hot model's backlog 429'd
      every other model.
    - GpuSwapGate: requests for a model already RESIDENT on the engine
      bypass the ownership gate — no swap is needed and its own per-model
      queue governs concurrency. The gate now serializes only requests
      that would actually trigger a model swap (its real purpose). Three
      co-resident embedders can each serve a request in parallel.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
    e9131f38
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...