-
Stefy Lanza (nextime / spora ) authored
`discard()` terminated the pod, on the reasoning that a connection can only be refused by a pod that no longer exists. A pod whose listen backlog is full refuses connections while being perfectly alive and busy -- and refuses them on /healthz too, so the "is anything still there?" probe added to protect against exactly this agrees it has vanished. Observed 2026-10-19, mid-campaign, ~58 concurrent connections per pod: 08:36:13 pod 2n69g0vt47h0hl terminated (connection refused) 08:36:30 pod c79lv31ayh28l1 terminated (connection refused) ... eight healthy A40s in 69 seconds ... 08:37:22 pod n9fpbskd7owl2h terminated (connection refused) 522 pages were in flight on them. The warm floor then re-rented eight more, which booted ~12 minutes, began refusing, and were killed again: $3.72 of pods billed -- more than the campaign's whole spend -- on boots that never produced a page, while throughput read 0.577 pages/s against a measured 0.557 PER POD. A pod is now QUARANTINED instead: out of rotation, still ours, still counted against max_pods because it is still billing. `maintain` re-probes it when the quarantine expires (45s) and * it answers -> straight back into rotation, which is the common case and the whole reason this stopped terminating; * it has work in flight -> left alone whatever the probe says, and it accrues no strikes while excused, so it does not die the moment the last page drains. Killing a pod with pages on it loses them AND pays for them; * it fails 3 probes with nothing in flight -> terminated. ~2 minutes of a pod that exists, bills and will not answer while carrying no load is dead in every sense that matters, so this does not trade a terminating bug for a leaking one. A quarantined pod also counts toward the warm-floor deficit. Without that, eight set-aside pods read as a deficit of eight and the pool rents ~700s of boot to cover a 45s gap, ending with two cards where the floor asked for one -- the re-rent loop in one line. The snapshot reports `quarantined` as its own state with the seconds remaining, and `pods_quarantined` in the summary. It is NOT counted as booting, which told an operator a settled fleet was still starting and told a client sizing from pods_ready to wait for pods already present. `discard` has exactly one caller (RunpodBackend._unreachable, reason "connection refused"), so nothing else loses a termination it needed. 19 tests. Full suite 3777 passed, 15 pre-existing failures unchanged. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
83e40d03