runpod: fetch lastStatusChange, the field that says why a pod is EXITED

A dead pod produced three hundred seconds of 'still no port' and no reason at
all, because the one Pod field that carries the reason was never queried. It
now travels into the progress line and the failure message, so an EXITED pod
says what RunPod says about it instead of leaving the cause to be guessed.

The strict-mode removal in boot.sh was one such guess: the image boots cleanly
here with a minimal environment and with a GPU exposed, so it does not
reproduce the exit. This is how the real cause gets found.
Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
parent baa0d92a
...@@ -325,6 +325,7 @@ class RunpodClient: ...@@ -325,6 +325,7 @@ class RunpodClient:
pod(input: $input) { pod(input: $input) {
id id
desiredStatus desiredStatus
lastStatusChange
costPerHr costPerHr
runtime { runtime {
uptimeInSeconds uptimeInSeconds
...@@ -339,6 +340,11 @@ class RunpodClient: ...@@ -339,6 +340,11 @@ class RunpodClient:
return { return {
"id": p.get("id") or pod_id, "id": p.get("id") or pod_id,
"status": p.get("desiredStatus"), "status": p.get("desiredStatus"),
# The one field that says WHY a pod is EXITED. It reads like
# "Rented by User: ..." or "container exited with code 1" — and it
# was never fetched, so a dead pod produced three hundred seconds
# of "still no port" and no reason at all.
"last_change": p.get("lastStatusChange") or "",
"uptime_s": rt.get("uptimeInSeconds") or 0, "uptime_s": rt.get("uptimeInSeconds") or 0,
"cost_per_hr": p.get("costPerHr"), "cost_per_hr": p.get("costPerHr"),
"ports": ports, "ports": ports,
...@@ -444,8 +450,8 @@ class RunpodClient: ...@@ -444,8 +450,8 @@ class RunpodClient:
raise RunpodError( raise RunpodError(
f"Pod {pod_id} container EXITED on boot (uptime " f"Pod {pod_id} container EXITED on boot (uptime "
f"{info.get('uptime_s')}s) — the image pulled but the process " f"{info.get('uptime_s')}s) — the image pulled but the process "
f"died immediately. Not a pull problem: check the image's " f"died immediately. RunPod says: "
f"entrypoint and the console log.") f"{info.get('last_change') or '(no reason given)'}")
if info.get("ready"): if info.get("ready"):
return pod_proxy_url(pod_id, port) return pod_proxy_url(pod_id, port)
# A multi-GB image pull is minutes of silence otherwise, which reads # A multi-GB image pull is minutes of silence otherwise, which reads
...@@ -453,7 +459,7 @@ class RunpodClient: ...@@ -453,7 +459,7 @@ class RunpodClient:
if time.time() >= next_report: if time.time() >= next_report:
next_report = time.time() + 60.0 next_report = time.time() + 60.0
why = ("image pull in progress" if info.get("status") != "EXITED" why = ("image pull in progress" if info.get("status") != "EXITED"
else "container has EXITED — waiting for a restart") else f"container has EXITED ({info.get('last_change') or 'no reason given'})")
print(f"[runpod] pod {pod_id}: still no port after " print(f"[runpod] pod {pod_id}: still no port after "
f"{time.time() - started:.0f}s " f"{time.time() - started:.0f}s "
f"(status={info.get('status')} uptime={info.get('uptime_s')}s) " f"(status={info.get('status')} uptime={info.get('uptime_s')}s) "
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment