-
Stefy Lanza (nextime / spora ) authored
Spillover already existed, but not in the shape that is actually useful: it fired only when the WAIT QUEUE overflowed (on_concurrency_full) or when no GPU could serve the model, and it could only reach a serverless endpoint — the code said outright that a pods target "goes through the engine and is not handled here", which meant bursting to our own pods was not implemented at all. Two additions: `on_busy` — offload the moment there is no free local slot, instead of queueing behind the running generation. Built on FrontQueue.try_acquire, a non-blocking take: the front has to learn the model is occupied WITHOUT committing to wait for it. Wired into both the buffered and the SSE dispatch paths. `target.mode: "pods"` — borrow a pod from the managed pool (provision on demand, health-gate, scale, reap, bill against the same caps) for one request and hand it straight back, so an overlap that never recurs is reaped rather than left running. Serverless and pods now share one `_spill_open` primitive, with a Response wrapper for the buffered path and a chunk generator for the SSE path. Local stays the safety net in both directions: a burst that cannot be opened falls back to the local queue, never the reverse. Two bugs caught before shipping, both in my own first cut: the direct path referenced _swap_tok before it is assigned (NameError on every burst), and the SSE path left _qkey set after bursting, so cleanup would have released a slot it never held — handing a phantom slot to the next waiter and over-subscribing the model. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
c104bcc9