• Stefy Lanza (nextime / spora )'s avatar
    runpod: a restart mid-boot froze the pool and hid its own pods (v0.3.68) · 143b86fa
    Stefy Lanza (nextime / spora ) authored
    Observed live: six pods billing on the RunPod account, none in coderai's
    interface, and "another engine is booting a pod for 'datalab-to/surya-ocr-2' --
    waiting for it" every fifteen seconds for seven minutes. Nothing was booting. A
    deploy had restarted coderai mid-boot, leaving that pod's URL-less registry
    entry with no owner.
    
    Three fixes, three different mistakes:
    
    1. sibling_is_provisioning() judged the entry's liveness against
       REAP_GRACE_SECONDS (1200s). That constant answers how long the REAPER waits
       before killing an unowned pod -- a different question. The registry has a
       60s heartbeat, so a dead owner is detectable in seconds. Now
       PROVISION_STALE_SECONDS (180s = three missed beats).
    
    2. The sibling check ran BEFORE _adopt_shared_pod(), so the recovery path for
       orphaned pods sat behind the marker that orphaning creates. Five running,
       registered, billing pods were never adopted. Adoption now runs first: a pod
       already serving this pool beats waiting for a boot and beats renting.
    
    3. A pod registered at CREATION recorded no $/hr -- the rate was passed only at
       the ready-registration 80 lines later -- so a boot interrupted before ready
       could only be reaped with "the registry records no rate for it". The price is
       known at creation. Also, hourly_usd is now carried over on re-registration
       like image and gpu already were: an adoption that did not know the rate
       turned a billing pod free.
    
    14 tests, including the exact live entry (url-less, 910s stale) as a case.
    test_runpod_client_scale's adopt-before-rent assertion was pinned to the old
    source spelling; updated to assert the intent, which is now stronger.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    143b86fa
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...