runpod: a pod that EXITED is dead, not pulling — and the boot script must never be the reason

A speaker pod reported EXITED with uptime 0s from its very first poll. The loop
waited out the full budget on it — three machines in a row — printing 'image
pull in progress', which it never was: wait_ready treated only TERMINATED and
FAILED as terminal. EXITED past a short restart grace now fails fast and says
what it means: the image pulled and the process died on boot.

The budget it waited out was 300s, not the 900s set on the pool class: the
dataclass default was never updated. Two numbers for one thing, and the smaller
one silently won. There is one now.

Why the process died: boot.sh ran under set -u, and every line before the exec
is a diagnostic — it exists to describe the pod, and a diagnostic must never be
able to stop the boot it describes. The image boots fine locally; on RunPod an
environment difference tripped a strict-mode abort before uvicorn was reached.
That is the best-supported explanation, not a proven one — RunPod publishes no
log API to confirm it — so the script is now unable to abort before its exec
under any environment, verified against a bare sh with nothing set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY

codai/api/runpod_client.py

@@ -433,16 +433,31 @@ class RunpodClient:
             last = info
             if info.get("status") in ("TERMINATED", "FAILED"):
                 raise RunpodError(f"Pod {pod_id} entered status {info.get('status')} before ready.")
+            if info.get("status") == "EXITED" and time.time() - started > 45:
+                # The container started and died. A pod reported EXITED with
+                # uptime 0s from its very first poll and the loop waited out the
+                # full budget on it — three machines in a row — while printing
+                # "image pull in progress", which it never was. A short grace
+                # period covers RunPod's own restart-on-exit; past that, EXITED
+                # means the process crashed on boot and no amount of waiting
+                # changes that. Fail fast so the retry lands on the real cause.
+                raise RunpodError(
+                    f"Pod {pod_id} container EXITED on boot (uptime "
+                    f"{info.get('uptime_s')}s) — the image pulled but the process "
+                    f"died immediately. Not a pull problem: check the image's "
+                    f"entrypoint and the console log.")
             if info.get("ready"):
                 return pod_proxy_url(pod_id, port)
             # A multi-GB image pull is minutes of silence otherwise, which reads
             # exactly like a hung pod. Say what is actually being waited on.
             if time.time() >= next_report:
                 next_report = time.time() + 60.0
+                why = ("image pull in progress" if info.get("status") != "EXITED"
+                       else "container has EXITED — waiting for a restart")
                 print(f"[runpod] pod {pod_id}: still no port after "
                       f"{time.time() - started:.0f}s "
                       f"(status={info.get('status')} uptime={info.get('uptime_s')}s) "
-                      f"— image pull in progress", flush=True)
+                      f"— {why}", flush=True)
             time.sleep(poll_every)
         raise RunpodError(
             f"Pod {pod_id} did not expose port {port} within {ready_timeout}s "

codai/api/runpod_worker.py

@@ -138,7 +138,11 @@ class RunpodModelConfig:
     sticky_sessions: bool = True
     idle_timeout_s: int = 300                # destroy a pod this long after its last request
     # Boot budgets — bigger models need longer (image pull + weight download).
-    boot_timeout_s: int = 300                # until the pod exposes its port
+    # Until the pod exposes its port. The port appears only AFTER the image is
+    # pulled, so this covers the download: 15 GB at a cold machine's ~25 MB/s
+    # is ten minutes. 900 was set on the pool class and this default was left at
+    # 300 — two numbers for one thing, and the smaller one silently won.
+    boot_timeout_s: int = 900
     load_timeout_s: int = 600                # until vLLM answers /v1/models
     # --- serverless ---
     endpoint_id: str = ""                    # reference an existing serverless endpoint

packaging/runpod/boot.sh

@@ -14,7 +14,12 @@
 # when the log API does not), and the same phases written to a file the app
 # serves at /boot, so the moment the port opens we can ask the pod where it is
 # instead of guessing.
-set -u
+# NOT set -u, and NOT set -e. Every line before the exec is a diagnostic: it
+# exists to tell us what the pod is, and a diagnostic must never be able to
+# stop the boot it describes. A speaker pod died with EXITED at uptime 0s on
+# three machines in a row — the image booted fine locally — because an
+# environment difference tripped a strict-mode abort before uvicorn was ever
+# reached. The one line that matters is the last one.
 
 BOOT_LOG="${CODERAI_BOOT_LOG:-/tmp/coderai-boot.log}"
 : > "$BOOT_LOG"
@@ -29,6 +34,7 @@ phase() {
 
 phase "container started"
 phase "profile=${PROFILE:-unknown} image=${CODERAI_IMAGE_TAG:-unknown}"
+phase "user=$(id -un 2>/dev/null || echo unknown) cwd=$(pwd 2>/dev/null || echo unknown)"
 phase "python=$(python -V 2>&1)"
 
 # What the pod was actually told to serve. A pod that answers "not available"
@@ -42,7 +48,7 @@ else
 fi
 
 if command -v nvidia-smi >/dev/null 2>&1; then
-    phase "gpu: $(nvidia-smi --query-gpu=name,memory.total --format=csv,noheader 2>/dev/null | head -1)"
+    phase "gpu: $(nvidia-smi --query-gpu=name,memory.total --format=csv,noheader 2>/dev/null | head -1 || echo 'nvidia-smi query failed')"
 else
     phase "gpu: nvidia-smi not present"
 fi