• Stefy Lanza (nextime / spora )'s avatar
    runpod: the reaper's grace period has never protected a single pod (v0.3.14) · 5fa59441
    Stefy Lanza (nextime / spora ) authored
    Observed live on the Digesta orchestrator while testing OCR: pod created
    19:24:56, "terminated STALE pod … not tracked by any pool" 19:24:57. The pool
    then waited out its 900s boot timeout on a pod that no longer existed, every
    OCR request 503'd, and the next attempt rented another.
    
    This is the race the source already documents from an earlier incident — "the
    nvidia engine created a pod and the radeon engine terminated it four seconds
    later, then the reverse — a mutual kill loop that rented and destroyed pods
    until stopped" — and the defence against it was supposed to be the age guard,
    "never reap a pod younger than this, whatever the registry says".
    
    It was written as `age is not None and age < REAP_GRACE_SECONDS`, and
    list_pods() returned {id, name, status, cost_per_hr, image} with no timestamp
    of any kind. So age was None for every pod, the condition was never true, and
    the guard abstained on exactly the pods it existed to protect. A guard that
    fails open is worse than none: it reads as covered.
    
    Two changes. The guard now fails CLOSED — an undatable pod is left alone and
    says so — and the listing carries runtime.uptimeInSeconds so it can measure
    instead of abstaining. Not reaping costs one more cycle; reaping a live pod
    costs the work on it and bills in a loop that never serves a request.
    
    What made it recur here is worth recording: this install has TWO config dirs,
    so the cross-process registry that shields a new pod exists twice and each
    process reads the one the other does not write. That is the deeper fault and is
    not fixed by this commit.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    5fa59441
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
compose Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
util Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...