• Stefy Lanza (nextime / spora )'s avatar
    front: own the generation queue (per-model admission gate) · e7001716
    Stefy Lanza (nextime / spora ) authored
    Move request queuing from the engine to the front so the engine does only
    generation. New codai/frontproxy/reqqueue.py FrontQueue: a per-model concurrency
    gate sized to the model's max_instances (per-model from models.json, else the
    global models.max_model_instances). A text-generation request acquires a slot
    before dispatch; when all slots are busy it waits (FIFO) and shows as "queued"
    with its position on the Tasks page; beyond server.queue_max_size waiting it gets
    HTTP 503. A client disconnect while queued cancels the wait and drops it from the
    queue with no slot leak.
    
    Wired into both dispatch paths in app.py (direct proxy + broker route) so direct
    and brokered requests share one queue, scoped to text generation only (images/
    audio/embeddings pass through unqueued). Slots are released on stream completion,
    send failure, or disconnect. Queue key is the canonical model id so all aliases/
    paths of a model share one gate. Front never over-subscribes the engine, so the
    engine's own queue stays a safety net that effectively never fills.
    
    The engine-side QueueManager + KV-affinity instance pool are intentionally left
    in place for now (trimming them is a follow-up once this is verified live).
    
    Verified: FrontQueue unit tests (capacity, FIFO hand-off, QueueFull, cancel
    cleanup, position, capacity>1) and a FrontProxy integration test (capacity
    resolution, alias-key collapsing, queued task surfacing).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
    e7001716
Name
Last commit
Last update
..
__init__.py Loading commit data...
app.py Loading commit data...
assignment.py Loading commit data...
engine_supervisor.py Loading commit data...
gpu_detect.py Loading commit data...
registry.py Loading commit data...
reqqueue.py Loading commit data...
router.py Loading commit data...
ui_pages.py Loading commit data...