-
Stefy Lanza (nextime / spora ) authored
Three defects, all measured on a live 32-pod fleet on 2026-10-19. QUARANTINE KEPT HEALTHY PODS OUT. The re-check skipped its probe while a pod had pages in flight -- meant to stop a busy pod being terminated, it also stopped it being RE-ADMITTED. A pod that tripped once on a connection got no new pages until all 64 drained, minutes later. Two of four serving pods sat quarantined while answering /healthz in 0.1 s. Now the probe always runs and a pod that answers is back at once. A HUNG POD HELD ITS PAGES FOR 15 MINUTES. The same rule meant a pod that answered nothing was never terminated while it had pages charged, so they waited for the in-flight timeout. 64 pages sat on one. Now three failed probes (~2 min) terminate it regardless; busy pods answer /healthz in a tenth of a second at full load, so this cannot catch one. A RESTART UNDER LOAD ORPHANED 29 PODS ($17.11/hr). Adoption happened only on demand -- a warm floor, or a waiting request -- and the restart dropped the client's lease, so nothing asked. And when asked, the probe was /v1/models with a 4 s timeout, through an engine still finishing 64 pages. Now every maintenance pass sweeps the registry (rate-limited to once a minute) and adopts every surviving pod whatever the floor says; the engine probe gets 15 s, and a pod slow there but alive on /healthz is adopted when the token it was launched with is on record. Co-Authored-By:
Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
e4554b0e