• Stefy Lanza (nextime / spora )'s avatar
    runpod: a restart mid-boot froze the pool and hid its own pods (v0.3.68) · 143b86fa
    Stefy Lanza (nextime / spora ) authored
    Observed live: six pods billing on the RunPod account, none in coderai's
    interface, and "another engine is booting a pod for 'datalab-to/surya-ocr-2' --
    waiting for it" every fifteen seconds for seven minutes. Nothing was booting. A
    deploy had restarted coderai mid-boot, leaving that pod's URL-less registry
    entry with no owner.
    
    Three fixes, three different mistakes:
    
    1. sibling_is_provisioning() judged the entry's liveness against
       REAP_GRACE_SECONDS (1200s). That constant answers how long the REAPER waits
       before killing an unowned pod -- a different question. The registry has a
       60s heartbeat, so a dead owner is detectable in seconds. Now
       PROVISION_STALE_SECONDS (180s = three missed beats).
    
    2. The sibling check ran BEFORE _adopt_shared_pod(), so the recovery path for
       orphaned pods sat behind the marker that orphaning creates. Five running,
       registered, billing pods were never adopted. Adoption now runs first: a pod
       already serving this pool beats waiting for a boot and beats renting.
    
    3. A pod registered at CREATION recorded no $/hr -- the rate was passed only at
       the ready-registration 80 lines later -- so a boot interrupted before ready
       could only be reaped with "the registry records no rate for it". The price is
       known at creation. Also, hourly_usd is now carried over on re-registration
       like image and gpu already were: an adoption that did not know the rate
       turned a billing pod free.
    
    14 tests, including the exact live entry (url-less, 910s stale) as a case.
    test_runpod_client_scale's adopt-before-rent assertion was pinned to the old
    source spelling; updated to assert the intent, which is now stronger.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    143b86fa
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
compose Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
util Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...