A request for an embedding model rented a pod, which began pulling a 9.65 GB
image that had been pushed twenty minutes earlier — a five-minute cold pull, and
the ORDINARY case for the first pod on a newly tagged image. Two minutes in,
RunPod's control plane answered HTTP 503 once. Three defects then fired in
sequence, and the pod that was coming up fine became an untracked GPU billing at
$1.09/hr:

  21:36:58  pod created — booting
  21:38:00  still no port after 62s (status=RUNNING) — image pull in progress
  21:38:50  boot FAILED: RunPod API HTTP 503: Service Unavailable
  21:38:53  POST /v1/embeddings  502

1. THE BOOT WAIT CONDEMNED THE POD FOR THE API'S FAILURE. Any exception out of
   the wait is read as "this pod failed to boot", and `get_pod` raised straight
   through the poll loop. A 503 from the control plane means the question went
   unanswered; it says nothing about the machine. RunpodTransient (a subclass,
   so every existing `except RunpodError` is unchanged) now marks 5xx/408/425/429
   and transport errors, and the wait keeps polling to the deadline the caller
   set, saying once that the API is quiet and again when it comes back. A wait
   that never got a single answer raises transient too — "unknown, not
   known-bad" — rather than handing the teardown path a verdict nobody reached.
   401 and 4xx stay final: those will say the same thing in five seconds.

2. THE TEARDOWN RAN THROUGH THE SAME DEAD API, INSIDE `except Exception: pass`.
   The attempt most likely to fail is the one that matters most, and its failure
   was dropped on the floor — no log line anywhere named the pod. terminate_pod
   now retries with backoff while the error is transient, and the boot handler
   prints what is still billing, at what rate, and how to stop it. A pod that
   could not be terminated KEEPS its registry entry: that entry is the only
   record it exists. `_give_back` had the mirror image of the bug — it
   unregistered unconditionally, throwing away the record of a live pod — and
   now forgets only what is actually gone. A boot that failed also ledgers the
   minutes it burned; no pool ever held that pod, so nothing else would.

3. THE REAPER COULD NEVER HAVE CLEANED UP AFTER EITHER. Pod age came from
   `runtime.uptimeInSeconds`, which is the CURRENT RUN's uptime and resets on
   every container restart. The leaked pod reported 234s while it was 11 minutes
   old, and its uptime reset every few minutes, so it could never cross the
   1200s grace — the pod that most needs reaping, one nothing tracks that is
   failing over and over, was the one pod the reaper was structurally blind to.
   `lastStatusChange` carries the rental time ("Rented by User: Thu Oct 15 2026
   19:36:58 GMT+0000") and was already in the response; list_pods now asks for
   it. Age is the larger of rental age and uptime — uptime cannot exceed the
   age, so the pair is still a true lower bound, and a pod whose stamp will not
   parse is no worse off than before. A future stamp is refused rather than
   making every pod ancient, and no age at all is still None: the reaper fails
   closed on that.

Found because the image update was the trigger: a new tag is cold in every
datacenter, so the first boot after every release spends minutes exposed to
exactly this. The next pod proved the image itself was fine — 301s of pull, 22s
of warm-up, 200 OK.

27 tests. Suite: 3132 passed, the same 13 known-stale failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM