-
Stefy Lanza (nextime / spora ) authored
Observed live on the Digesta orchestrator while testing OCR: pod created 19:24:56, "terminated STALE pod … not tracked by any pool" 19:24:57. The pool then waited out its 900s boot timeout on a pod that no longer existed, every OCR request 503'd, and the next attempt rented another. This is the race the source already documents from an earlier incident — "the nvidia engine created a pod and the radeon engine terminated it four seconds later, then the reverse — a mutual kill loop that rented and destroyed pods until stopped" — and the defence against it was supposed to be the age guard, "never reap a pod younger than this, whatever the registry says". It was written as `age is not None and age < REAP_GRACE_SECONDS`, and list_pods() returned {id, name, status, cost_per_hr, image} with no timestamp of any kind. So age was None for every pod, the condition was never true, and the guard abstained on exactly the pods it existed to protect. A guard that fails open is worse than none: it reads as covered. Two changes. The guard now fails CLOSED — an undatable pod is left alone and says so — and the listing carries runtime.uptimeInSeconds so it can measure instead of abstaining. Not reaping costs one more cycle; reaping a live pod costs the work on it and bills in a loop that never serves a request. What made it recur here is worth recording: this install has TWO config dirs, so the cross-process registry that shields a new pod exists twice and each process reads the one the other does not write. That is the deeper fault and is not fixed by this commit. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
5fa59441