• Stefy Lanza (nextime / spora )'s avatar
    front: own the generation queue (per-model admission gate) · e7001716
    Stefy Lanza (nextime / spora ) authored
    Move request queuing from the engine to the front so the engine does only
    generation. New codai/frontproxy/reqqueue.py FrontQueue: a per-model concurrency
    gate sized to the model's max_instances (per-model from models.json, else the
    global models.max_model_instances). A text-generation request acquires a slot
    before dispatch; when all slots are busy it waits (FIFO) and shows as "queued"
    with its position on the Tasks page; beyond server.queue_max_size waiting it gets
    HTTP 503. A client disconnect while queued cancels the wait and drops it from the
    queue with no slot leak.
    
    Wired into both dispatch paths in app.py (direct proxy + broker route) so direct
    and brokered requests share one queue, scoped to text generation only (images/
    audio/embeddings pass through unqueued). Slots are released on stream completion,
    send failure, or disconnect. Queue key is the canonical model id so all aliases/
    paths of a model share one gate. Front never over-subscribes the engine, so the
    engine's own queue stays a safety net that effectively never fills.
    
    The engine-side QueueManager + KV-affinity instance pool are intentionally left
    in place for now (trimming them is a follow-up once this is verified live).
    
    Verified: FrontQueue unit tests (capacity, FIFO hand-off, QueueFull, cancel
    cleanup, position, capacity>1) and a FrontProxy integration test (capacity
    resolution, alias-key collapsing, queued task surfacing).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
    e7001716
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...