• Stefy Lanza (nextime / spora )'s avatar
    front: burst to RunPod when the local model is busy, and to pods · c104bcc9
    Stefy Lanza (nextime / spora ) authored
    Spillover already existed, but not in the shape that is actually useful:
    it fired only when the WAIT QUEUE overflowed (on_concurrency_full) or
    when no GPU could serve the model, and it could only reach a serverless
    endpoint — the code said outright that a pods target "goes through the
    engine and is not handled here", which meant bursting to our own pods was
    not implemented at all.
    
    Two additions:
    
    `on_busy` — offload the moment there is no free local slot, instead of
    queueing behind the running generation. Built on FrontQueue.try_acquire,
    a non-blocking take: the front has to learn the model is occupied WITHOUT
    committing to wait for it. Wired into both the buffered and the SSE
    dispatch paths.
    
    `target.mode: "pods"` — borrow a pod from the managed pool (provision on
    demand, health-gate, scale, reap, bill against the same caps) for one
    request and hand it straight back, so an overlap that never recurs is
    reaped rather than left running. Serverless and pods now share one
    `_spill_open` primitive, with a Response wrapper for the buffered path
    and a chunk generator for the SSE path.
    
    Local stays the safety net in both directions: a burst that cannot be
    opened falls back to the local queue, never the reverse.
    
    Two bugs caught before shipping, both in my own first cut: the direct
    path referenced _swap_tok before it is assigned (NameError on every
    burst), and the SSE path left _qkey set after bursting, so cleanup would
    have released a slot it never held — handing a phantom slot to the next
    waiter and over-subscribing the model.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    c104bcc9
reqqueue.py 12.4 KB