• Stefy Lanza (nextime / spora )'s avatar
    front: own the generation queue (per-model admission gate) · e7001716
    Stefy Lanza (nextime / spora ) authored
    Move request queuing from the engine to the front so the engine does only
    generation. New codai/frontproxy/reqqueue.py FrontQueue: a per-model concurrency
    gate sized to the model's max_instances (per-model from models.json, else the
    global models.max_model_instances). A text-generation request acquires a slot
    before dispatch; when all slots are busy it waits (FIFO) and shows as "queued"
    with its position on the Tasks page; beyond server.queue_max_size waiting it gets
    HTTP 503. A client disconnect while queued cancels the wait and drops it from the
    queue with no slot leak.
    
    Wired into both dispatch paths in app.py (direct proxy + broker route) so direct
    and brokered requests share one queue, scoped to text generation only (images/
    audio/embeddings pass through unqueued). Slots are released on stream completion,
    send failure, or disconnect. Queue key is the canonical model id so all aliases/
    paths of a model share one gate. Front never over-subscribes the engine, so the
    engine's own queue stays a safety net that effectively never fills.
    
    The engine-side QueueManager + KV-affinity instance pool are intentionally left
    in place for now (trimming them is a follow-up once this is verified live).
    
    Verified: FrontQueue unit tests (capacity, FIFO hand-off, QueueFull, cancel
    cleanup, position, capacity>1) and a FrontProxy integration test (capacity
    resolution, alias-key collapsing, queued task surfacing).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
    e7001716
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...