• Stefy Lanza (nextime / spora )'s avatar
    runpod: retry stuck-boot pods on another machine + self-diagnosing vLLM logs · 0c8417fa
    Stefy Lanza (nextime / spora ) authored
    Live testing showed RunPod sometimes ACCEPTS a pod create but the machine never
    starts (port never exposed) — 2 of 4 pods hung this way, burning the full 900s
    wait. Provisioning is now resilient and diagnosable:
    
    - Boot failure retries on the NEXT ranked GPU candidate (MAX_PROVISION_ATTEMPTS=3)
      instead of failing the request; _create_with_fallback returns the next index so
      each attempt uses a different machine/type.
    - Timeouts split + tightened: port-exposed 300s (was 900 — a pod that hasn't
      exposed its port in 5min never will), vLLM /v1/models health 600s (model load).
      Cuts worst-case billing on a bad machine from ~15min to ~5min per attempt.
    - Self-diagnosing: on any boot failure the pod's container/vLLM log is fetched and
      printed to coderai's log before teardown (the user's #1 cause is a vLLM OOM /
      launch error, only visible in that log), plus the console link.
    - Pod id + console URL are now logged immediately after create (previously only at
      "ready", so a stuck pod was untraceable).
    - Booting pods appear in pods_status/stats page as state="provisioning" with their
      console link, so a slow boot is visible live.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    0c8417fa
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...