-
Stefy Lanza (nextime / spora ) authored
Move request queuing from the engine to the front so the engine does only generation. New codai/frontproxy/reqqueue.py FrontQueue: a per-model concurrency gate sized to the model's max_instances (per-model from models.json, else the global models.max_model_instances). A text-generation request acquires a slot before dispatch; when all slots are busy it waits (FIFO) and shows as "queued" with its position on the Tasks page; beyond server.queue_max_size waiting it gets HTTP 503. A client disconnect while queued cancels the wait and drops it from the queue with no slot leak. Wired into both dispatch paths in app.py (direct proxy + broker route) so direct and brokered requests share one queue, scoped to text generation only (images/ audio/embeddings pass through unqueued). Slots are released on stream completion, send failure, or disconnect. Queue key is the canonical model id so all aliases/ paths of a model share one gate. Front never over-subscribes the engine, so the engine's own queue stays a safety net that effectively never fills. The engine-side QueueManager + KV-affinity instance pool are intentionally left in place for now (trimming them is a follow-up once this is verified live). Verified: FrontQueue unit tests (capacity, FIFO hand-off, QueueFull, cancel cleanup, position, capacity>1) and a FrontProxy integration test (capacity resolution, alias-key collapsing, queued task surfacing). Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
e7001716