• Stefy Lanza (nextime / spora )'s avatar
    runpod: retry stuck-boot pods on another machine + self-diagnosing vLLM logs · 0c8417fa
    Stefy Lanza (nextime / spora ) authored
    Live testing showed RunPod sometimes ACCEPTS a pod create but the machine never
    starts (port never exposed) — 2 of 4 pods hung this way, burning the full 900s
    wait. Provisioning is now resilient and diagnosable:
    
    - Boot failure retries on the NEXT ranked GPU candidate (MAX_PROVISION_ATTEMPTS=3)
      instead of failing the request; _create_with_fallback returns the next index so
      each attempt uses a different machine/type.
    - Timeouts split + tightened: port-exposed 300s (was 900 — a pod that hasn't
      exposed its port in 5min never will), vLLM /v1/models health 600s (model load).
      Cuts worst-case billing on a bad machine from ~15min to ~5min per attempt.
    - Self-diagnosing: on any boot failure the pod's container/vLLM log is fetched and
      printed to coderai's log before teardown (the user's #1 cause is a vLLM OOM /
      launch error, only visible in that log), plus the console link.
    - Pod id + console URL are now logged immediately after create (previously only at
      "ready", so a stuck pod was untraceable).
    - Booting pods appear in pods_status/stats page as state="provisioning" with their
      console link, so a slow boot is visible live.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    0c8417fa
runpod.html 7.67 KB