• Stefy Lanza (nextime / spora )'s avatar
    runpod: a restart mid-boot froze the pool and hid its own pods (v0.3.68) · 143b86fa
    Stefy Lanza (nextime / spora ) authored
    Observed live: six pods billing on the RunPod account, none in coderai's
    interface, and "another engine is booting a pod for 'datalab-to/surya-ocr-2' --
    waiting for it" every fifteen seconds for seven minutes. Nothing was booting. A
    deploy had restarted coderai mid-boot, leaving that pod's URL-less registry
    entry with no owner.
    
    Three fixes, three different mistakes:
    
    1. sibling_is_provisioning() judged the entry's liveness against
       REAP_GRACE_SECONDS (1200s). That constant answers how long the REAPER waits
       before killing an unowned pod -- a different question. The registry has a
       60s heartbeat, so a dead owner is detectable in seconds. Now
       PROVISION_STALE_SECONDS (180s = three missed beats).
    
    2. The sibling check ran BEFORE _adopt_shared_pod(), so the recovery path for
       orphaned pods sat behind the marker that orphaning creates. Five running,
       registered, billing pods were never adopted. Adoption now runs first: a pod
       already serving this pool beats waiting for a boot and beats renting.
    
    3. A pod registered at CREATION recorded no $/hr -- the rate was passed only at
       the ready-registration 80 lines later -- so a boot interrupted before ready
       could only be reaped with "the registry records no rate for it". The price is
       known at creation. Also, hourly_usd is now carried over on re-registration
       like image and gpu already were: an adoption that did not know the rate
       turned a billing pod free.
    
    14 tests, including the exact live entry (url-less, 910s stale) as a case.
    test_runpod_client_scale's adopt-before-rent assertion was pinned to the old
    source spelling; updated to assert the intent, which is now stronger.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    143b86fa
test_runpod_restart_orphans.py 6.9 KB