runpod: the reaper's grace period has never protected a single pod (v0.3.14)

Observed live on the Digesta orchestrator while testing OCR: pod created
19:24:56, "terminated STALE pod … not tracked by any pool" 19:24:57. The pool
then waited out its 900s boot timeout on a pod that no longer existed, every
OCR request 503'd, and the next attempt rented another.

This is the race the source already documents from an earlier incident — "the
nvidia engine created a pod and the radeon engine terminated it four seconds
later, then the reverse — a mutual kill loop that rented and destroyed pods
until stopped" — and the defence against it was supposed to be the age guard,
"never reap a pod younger than this, whatever the registry says".

It was written as `age is not None and age < REAP_GRACE_SECONDS`, and
list_pods() returned {id, name, status, cost_per_hr, image} with no timestamp
of any kind. So age was None for every pod, the condition was never true, and
the guard abstained on exactly the pods it existed to protect. A guard that
fails open is worse than none: it reads as covered.

Two changes. The guard now fails CLOSED — an undatable pod is left alone and
says so — and the listing carries runtime.uptimeInSeconds so it can measure
instead of abstaining. Not reaping costs one more cycle; reaping a live pod
costs the work on it and bills in a loop that never serves a request.

What made it recur here is worth recording: this install has TWO config dirs,
so the cross-process registry that shields a new pod exists twice and each
process reads the one the other does not write. That is the deeper fault and is
not fixed by this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.13"
+__version__ = "0.3.14"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/runpod_client.py

@@ -469,19 +469,34 @@ class RunpodClient:
 
     def list_pods(self) -> list:
         """List the account's pods: [{id, name, status, cost_per_hr, image}]."""
+        # runtime.uptimeInSeconds is what makes the reaper's grace period real:
+        # without an age every pod looked undatable, and the "never reap a pod
+        # younger than GRACE" backstop abstained on all of them.
         q = """
         query Pods {
           myself {
-            pods { id name desiredStatus costPerHr imageName }
+            pods { id name desiredStatus costPerHr imageName
+                   runtime { uptimeInSeconds } }
           }
         }"""
         data = self._gql(q)
         me = data.get("myself") or {}
-        return [{
-            "id": p.get("id"), "name": p.get("name"),
-            "status": p.get("desiredStatus"), "cost_per_hr": p.get("costPerHr"),
-            "image": p.get("imageName"),
-        } for p in (me.get("pods") or [])]
+        out = []
+        for p in (me.get("pods") or []):
+            row = {
+                "id": p.get("id"), "name": p.get("name"),
+                "status": p.get("desiredStatus"), "cost_per_hr": p.get("costPerHr"),
+                "image": p.get("imageName"),
+            }
+            # Absent while a pod is still being scheduled, which is exactly when
+            # it must not be mistaken for a leftover — the reaper now treats a
+            # missing age as "too young to judge".
+            rt = p.get("runtime") or {}
+            up = rt.get("uptimeInSeconds")
+            if isinstance(up, (int, float)) and up >= 0:
+                row["uptime_s"] = float(up)
+            out.append(row)
+        return out
 
     def get_pod_logs(self, pod_id: str, tail: int = 200) -> dict:
         """Best-effort container/vLLM logs for a pod via the REST API.

codai/api/runpod_worker.py

@@ -3618,7 +3618,25 @@ def reap_orphans() -> int:
         if status in ("TERMINATED", "EXITED"):
             continue
         age = _pod_age_seconds(p)
-        if age is not None and age < REAP_GRACE_SECONDS:
+        if age is None:
+            # Fail CLOSED. This guard is documented as holding "whatever the
+            # registry says", and it was written as `age is not None and
+            # age < GRACE` — so an unknown age skipped it entirely. list_pods()
+            # returned no timestamp at all until the query below was widened,
+            # which means the backstop never protected a single pod: a pod one
+            # second old, invisible to a sibling's registry, was terminated as a
+            # leftover. Observed live on the Digesta orchestrator — created
+            # 19:24:56, reaped 19:24:57, and OCR then waited out the boot
+            # timeout on a pod that no longer existed.
+            #
+            # Not reaping costs one more cycle. Reaping a live pod costs the work
+            # on it and, when the pool immediately rents a replacement, bills in
+            # a loop that never serves a request.
+            print(f"[runpod-reaper] pod {pid} ({name}) has no age from RunPod — "
+                  f"leaving it alone rather than guessing it is a leftover",
+                  flush=True)
+            continue
+        if age < REAP_GRACE_SECONDS:
             continue                      # too young to be a leftover
         try:
             client.terminate_pod(pid)

tests/test_runpod_reaper_grace.py

+"""The stale-pod reaper must not kill a pod that was just created.
+
+coderai runs a front plus an engine per GPU, each with its own pool registry and
+its own reaper, so a pod one process created is an orphan in another's view until
+the shared registry catches up. That race is already documented in the source,
+from a live incident: "the nvidia engine created a pod and the radeon engine
+terminated it four seconds later, then the reverse — a mutual kill loop that
+rented and destroyed pods until stopped."
+
+The defence against it is an age guard — "never reap a pod younger than this,
+whatever the registry says". It was written as:
+
+    age = _pod_age_seconds(p)
+    if age is not None and age < REAP_GRACE_SECONDS:
+        continue
+
+and `list_pods()` returned {id, name, status, cost_per_hr, image} — no timestamp
+of any kind. So `age` was None for every pod, the condition was never true, and
+the backstop had never protected anything. It recurred on the Digesta
+orchestrator, which has two config dirs and therefore two registries: pod created
+19:24:56, reaped by a sibling 19:24:57, after which OCR waited out its 900s boot
+timeout on a pod that no longer existed and the pool rented another.
+"""
+
+import pathlib
+import sys
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from codai.api import runpod_worker as rw                          # noqa: E402
+
+
+def test_an_undatable_pod_is_left_alone():
+    """The only state the guard was ever actually in. Not reaping costs one more
+    cycle; reaping a live pod costs the work on it and bills in a loop."""
+    assert rw._pod_age_seconds({"id": "p1", "name": "coderai-x-ocr"}) is None
+
+
+def test_uptime_is_read_when_runpod_reports_it():
+    assert rw._pod_age_seconds({"uptime_s": 12.0}) == 12.0
+
+
+def test_the_grace_period_is_longer_than_any_boot():
+    """A pod still pulling a 12 GB image must not look like a leftover: the
+    image pull alone ran past 128s in the incident."""
+    assert rw.REAP_GRACE_SECONDS >= 900
+
+
+def test_the_listing_now_carries_an_age():
+    """Without a timestamp in the query the guard can only ever abstain."""
+    src = (ROOT / "codai/api/runpod_client.py").read_text()
+    blk = src.split("def list_pods")[1].split("def ")[0]
+    assert "uptimeInSeconds" in blk
+    assert '"uptime_s"' in blk
+
+
+def test_an_unknown_age_does_not_fall_through_to_termination():
+    """The shape of the bug: `age is not None and age < GRACE` skipped the guard
+    for exactly the pods it existed to protect."""
+    src = (ROOT / "codai/api/runpod_worker.py").read_text()
+    blk = src.split("def reap_orphans")[1]
+    assert "if age is None:" in blk
+    assert "age is not None and age < REAP_GRACE_SECONDS" not in blk