• Stefy Lanza (nextime / spora )'s avatar
    a refused connection is backpressure, not a dead pod (v0.4.37) · 83e40d03
    Stefy Lanza (nextime / spora ) authored
    `discard()` terminated the pod, on the reasoning that a connection can
    only be refused by a pod that no longer exists. A pod whose listen
    backlog is full refuses connections while being perfectly alive and busy
    -- and refuses them on /healthz too, so the "is anything still there?"
    probe added to protect against exactly this agrees it has vanished.
    
    Observed 2026-10-19, mid-campaign, ~58 concurrent connections per pod:
    
        08:36:13  pod 2n69g0vt47h0hl terminated (connection refused)
        08:36:30  pod c79lv31ayh28l1 terminated (connection refused)
        ... eight healthy A40s in 69 seconds ...
        08:37:22  pod n9fpbskd7owl2h terminated (connection refused)
    
    522 pages were in flight on them. The warm floor then re-rented eight
    more, which booted ~12 minutes, began refusing, and were killed again:
    $3.72 of pods billed -- more than the campaign's whole spend -- on boots
    that never produced a page, while throughput read 0.577 pages/s against
    a measured 0.557 PER POD.
    
    A pod is now QUARANTINED instead: out of rotation, still ours, still
    counted against max_pods because it is still billing. `maintain`
    re-probes it when the quarantine expires (45s) and
    
      * it answers -> straight back into rotation, which is the common case
        and the whole reason this stopped terminating;
      * it has work in flight -> left alone whatever the probe says, and it
        accrues no strikes while excused, so it does not die the moment the
        last page drains. Killing a pod with pages on it loses them AND pays
        for them;
      * it fails 3 probes with nothing in flight -> terminated. ~2 minutes of
        a pod that exists, bills and will not answer while carrying no load
        is dead in every sense that matters, so this does not trade a
        terminating bug for a leaking one.
    
    A quarantined pod also counts toward the warm-floor deficit. Without
    that, eight set-aside pods read as a deficit of eight and the pool rents
    ~700s of boot to cover a 45s gap, ending with two cards where the floor
    asked for one -- the re-rent loop in one line.
    
    The snapshot reports `quarantined` as its own state with the seconds
    remaining, and `pods_quarantined` in the summary. It is NOT counted as
    booting, which told an operator a settled fleet was still starting and
    told a client sizing from pods_ready to wait for pods already present.
    
    `discard` has exactly one caller (RunpodBackend._unreachable, reason
    "connection refused"), so nothing else loses a termination it needed.
    
    19 tests. Full suite 3777 passed, 15 pre-existing failures unchanged.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    83e40d03
test_runpod_client_scale.py 23.8 KB