• Stefy Lanza (nextime / spora )'s avatar
    runpod: retry stuck-boot pods on another machine + self-diagnosing vLLM logs · 0c8417fa
    Stefy Lanza (nextime / spora ) authored
    Live testing showed RunPod sometimes ACCEPTS a pod create but the machine never
    starts (port never exposed) — 2 of 4 pods hung this way, burning the full 900s
    wait. Provisioning is now resilient and diagnosable:
    
    - Boot failure retries on the NEXT ranked GPU candidate (MAX_PROVISION_ATTEMPTS=3)
      instead of failing the request; _create_with_fallback returns the next index so
      each attempt uses a different machine/type.
    - Timeouts split + tightened: port-exposed 300s (was 900 — a pod that hasn't
      exposed its port in 5min never will), vLLM /v1/models health 600s (model load).
      Cuts worst-case billing on a bad machine from ~15min to ~5min per attempt.
    - Self-diagnosing: on any boot failure the pod's container/vLLM log is fetched and
      printed to coderai's log before teardown (the user's #1 cause is a vLLM OOM /
      launch error, only visible in that log), plus the console link.
    - Pod id + console URL are now logged immediately after create (previously only at
      "ready", so a stuck pod was untraceable).
    - Booting pods appear in pods_status/stats page as state="provisioning" with their
      console link, so a slow boot is visible live.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    0c8417fa
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...