-
Stefy Lanza (nextime / spora ) authored
Three reports from the same run, all of them about giving pods back. SCALE BLOCKED THE WHOLE SERVER. The endpoint called into the pool straight from the coroutine, so while it walked dozens of (card, region) candidates -- a network round trip each -- nothing else was served. /v1/runpod/status stopped answering during exactly the operation someone would poll it about, which makes a slow rental look identical to an outage. It now runs off the event loop and the reply is bounded: after `wait_s` (default 20) it answers `pending` with a request id, and the SCALING CONTINUES -- shielded, because cancelling it would leave a half-made lease and possibly a pod. GET /v1/runpod/scale/{id} has the result for a caller that gave up. A BOOT COULD NOT BE CANCELLED. `_booting` is an integer, not pod objects -- a pod being created does not exist here yet -- so `shed_now` could not see one, count one or stop one. A client that asked for a pod, got four on the way and released them was told `kept:0, needed:1, floor:1` and paid every boot out in full. A shed now bumps a cancel epoch; an ANTICIPATORY boot (warm floor, background growth) checks it during the model load and again just before it joins the fleet, and hands its pod straight back. A boot a request is blocked on is never cancellable, whatever arrives meanwhile: the request is what the pod is for. Booting pods also count as capacity in the surplus arithmetic, and the shed reports how many it stopped. A POD COULD NOT BE RELEASED unless some pool floor could be lowered to lose it. `'pool:digesta-qwen38' has no runpod block` came from code whose own comment says the caller cannot look that key up -- so a live pool is now addressable by the key /v1/runpod/status reports. And nothing anywhere could terminate ONE pod by id: not an adopted pod, not one left by a crash, not one whose pool was reconfigured underneath it. POST /v1/runpod/release does, admin-scoped (leasing is bounded and expires; terminating a named pod is neither), attributing the spend before it drops the pod so the money does not land nowhere. And the over-provisioning: `ensure_ready` incremented `_booting` by ONE and then started `want` boots, so with min_pods 3 three cards were on the way while the 15s scaler, `_should_grow` and the parallel-boot cap all believed there was one -- each topping up against a deficit already being filled. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
764b5c59