• Stefy Lanza (nextime / spora )'s avatar
    quarantine re-admits a busy pod, and a restart re-adopts every pod (v0.4.46) · e4554b0e
    Stefy Lanza (nextime / spora ) authored
    Three defects, all measured on a live 32-pod fleet on 2026-10-19.
    
    QUARANTINE KEPT HEALTHY PODS OUT. The re-check skipped its probe while
    a pod had pages in flight -- meant to stop a busy pod being terminated,
    it also stopped it being RE-ADMITTED. A pod that tripped once on a
    connection got no new pages until all 64 drained, minutes later. Two of
    four serving pods sat quarantined while answering /healthz in 0.1 s.
    Now the probe always runs and a pod that answers is back at once.
    
    A HUNG POD HELD ITS PAGES FOR 15 MINUTES. The same rule meant a pod
    that answered nothing was never terminated while it had pages charged,
    so they waited for the in-flight timeout. 64 pages sat on one. Now three
    failed probes (~2 min) terminate it regardless; busy pods answer
    /healthz in a tenth of a second at full load, so this cannot catch one.
    
    A RESTART UNDER LOAD ORPHANED 29 PODS ($17.11/hr). Adoption happened
    only on demand -- a warm floor, or a waiting request -- and the restart
    dropped the client's lease, so nothing asked. And when asked, the probe
    was /v1/models with a 4 s timeout, through an engine still finishing 64
    pages. Now every maintenance pass sweeps the registry (rate-limited to
    once a minute) and adopts every surviving pod whatever the floor says;
    the engine probe gets 15 s, and a pod slow there but alive on /healthz
    is adopted when the token it was launched with is on record.
    Co-Authored-By: 's avatarClaude Opus 5.5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    e4554b0e
test_runpod_refusal_is_not_death.py 10.5 KB