• Stefy Lanza (nextime / spora )'s avatar
    runpod: a restart mid-boot froze the pool and hid its own pods (v0.3.68) · 143b86fa
    Stefy Lanza (nextime / spora ) authored
    Observed live: six pods billing on the RunPod account, none in coderai's
    interface, and "another engine is booting a pod for 'datalab-to/surya-ocr-2' --
    waiting for it" every fifteen seconds for seven minutes. Nothing was booting. A
    deploy had restarted coderai mid-boot, leaving that pod's URL-less registry
    entry with no owner.
    
    Three fixes, three different mistakes:
    
    1. sibling_is_provisioning() judged the entry's liveness against
       REAP_GRACE_SECONDS (1200s). That constant answers how long the REAPER waits
       before killing an unowned pod -- a different question. The registry has a
       60s heartbeat, so a dead owner is detectable in seconds. Now
       PROVISION_STALE_SECONDS (180s = three missed beats).
    
    2. The sibling check ran BEFORE _adopt_shared_pod(), so the recovery path for
       orphaned pods sat behind the marker that orphaning creates. Five running,
       registered, billing pods were never adopted. Adoption now runs first: a pod
       already serving this pool beats waiting for a boot and beats renting.
    
    3. A pod registered at CREATION recorded no $/hr -- the rate was passed only at
       the ready-registration 80 lines later -- so a boot interrupted before ready
       could only be reaped with "the registry records no rate for it". The price is
       known at creation. Also, hourly_usd is now carried over on re-registration
       like image and gpu already were: an adoption that did not know the rate
       turned a billing pod free.
    
    14 tests, including the exact live entry (url-less, 910s stale) as a case.
    test_runpod_client_scale's adopt-before-rent assertion was pinned to the old
    source spelling; updated to assert the intent, which is now stronger.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    143b86fa
Name
Last commit
Last update
..
superpowers Loading commit data...
README.md Loading commit data...
admin-forward-auth.md Loading commit data...
cluster.md Loading commit data...
deepseek-ds4.md Loading commit data...
dtype-auto-selection.md Loading commit data...
expressive-tts.md Loading commit data...
frontend-engine-split.md Loading commit data...
gguf-process-isolation.md Loading commit data...
glm-colibri.md Loading commit data...
install-from-packages.md Loading commit data...
kimi-k3.md Loading commit data...
ktransformers.md Loading commit data...
longcat-video.md Loading commit data...
minimax-h3.md Loading commit data...
modelli-digesta-e-container.md Loading commit data...
ocr.md Loading commit data...
orchestrator-runpod-guide.md Loading commit data...
piano-inferenza-su-coderai-prod.md Loading commit data...
process-isolation-plans.md Loading commit data...
reference-projects.md Loading commit data...
remote-execution.md Loading commit data...
reverse-proxy-nginx.md Loading commit data...
runpod-regional-storage-plan.md Loading commit data...
runpod.md Loading commit data...
tools-as-services.md Loading commit data...
unlimited-ocr-plan.md Loading commit data...
video-compose.md Loading commit data...
vllm.md Loading commit data...
zimage-lora-training.md Loading commit data...