• Stefy Lanza (nextime / spora )'s avatar
    runpod: a warm floor is not satisfied by a pod that no longer exists (v0.3.44) · 9c82c3f1
    Stefy Lanza (nextime / spora ) authored
    `keep_warm` stopped re-renting whenever a pod went away behind the pool's
    back — terminated from the console, reclaimed as spot, lost with its host —
    and the scaler said nothing about it. Both halves had one cause: nothing in
    a pool ever re-checks a pod's health. A handle's `healthy` flag goes false
    only when something tried to USE the pod and failed, or when the pool itself
    retired it, and a warm-floor pod is precisely the pod with no traffic to
    fail. So `maintain()` counted the handle, found `healthy_n >= keep_min`, and
    neither topped up nor had anything to log. The floor stayed "satisfied" by a
    machine that had not existed for hours, and the first real request paid the
    full ~12-minute cold boot that keep_warm is bought to avoid.
    
    (The class docstring claimed the scaler "re-checks health" for months. It
    never has. Corrected rather than implemented: a timer-probe would have to
    decide what an unanswered probe means, and that decision is what killed
    seven A40s in v0.3.41.)
    
    `reconcile_pools` instead reads the account listing the stale-pod reaper
    already fetches every ~30s — authoritative, free, and unable to mistake a
    busy pod for a dead one. Only positive evidence acts:
    
      * a dead status (TERMINATED/EXITED/DEAD/FAILED) drops the pod at once;
      * a mere absence drops it only after two listings in a row, because the
        API is eventually consistent and one page omitting a live pod used to be
        enough to unregister it — a reappearance resets the count;
      * a pod that served a request in the last 60s is kept whatever the listing
        says, which cannot mask this bug since a warm-floor pod has no traffic;
      * a provisioning pod is shielded, and a FAILED listing never gets here at
        all, so an API outage cannot empty every pool and re-rent the fleet.
    
    `PodPool.forget()` is `discard()` minus the kill: nothing is left to
    terminate, but the bill is closed in the ledger exactly as a real
    termination closes it, since the pod was billed up to its death. A borrowed
    `(shared)` pod is dropped without a ledger entry — its owner holds the bill.
    The registry is pruned through the same two-sweep rule it used to skip.
    
    The top-up now announces itself before spending, because a warm floor
    silently failing is indistinguishable from a warm floor that is fine.
    
    21 new tests in tests/test_runpod_warm_floor_rerent.py pin the caution, not
    just the fix: the failure mode in the other direction is forgetting a pod
    that is alive and billing. test_runpod_client_scale's source assertion moved
    to the new seam.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    9c82c3f1
test_runpod_client_scale.py 21 KB