-
Stefy Lanza (nextime / spora ) authored
`keep_warm` stopped re-renting whenever a pod went away behind the pool's back — terminated from the console, reclaimed as spot, lost with its host — and the scaler said nothing about it. Both halves had one cause: nothing in a pool ever re-checks a pod's health. A handle's `healthy` flag goes false only when something tried to USE the pod and failed, or when the pool itself retired it, and a warm-floor pod is precisely the pod with no traffic to fail. So `maintain()` counted the handle, found `healthy_n >= keep_min`, and neither topped up nor had anything to log. The floor stayed "satisfied" by a machine that had not existed for hours, and the first real request paid the full ~12-minute cold boot that keep_warm is bought to avoid. (The class docstring claimed the scaler "re-checks health" for months. It never has. Corrected rather than implemented: a timer-probe would have to decide what an unanswered probe means, and that decision is what killed seven A40s in v0.3.41.) `reconcile_pools` instead reads the account listing the stale-pod reaper already fetches every ~30s — authoritative, free, and unable to mistake a busy pod for a dead one. Only positive evidence acts: * a dead status (TERMINATED/EXITED/DEAD/FAILED) drops the pod at once; * a mere absence drops it only after two listings in a row, because the API is eventually consistent and one page omitting a live pod used to be enough to unregister it — a reappearance resets the count; * a pod that served a request in the last 60s is kept whatever the listing says, which cannot mask this bug since a warm-floor pod has no traffic; * a provisioning pod is shielded, and a FAILED listing never gets here at all, so an API outage cannot empty every pool and re-rent the fleet. `PodPool.forget()` is `discard()` minus the kill: nothing is left to terminate, but the bill is closed in the ledger exactly as a real termination closes it, since the pod was billed up to its death. A borrowed `(shared)` pod is dropped without a ledger entry — its owner holds the bill. The registry is pruned through the same two-sweep rule it used to skip. The top-up now announces itself before spending, because a warm floor silently failing is indistinguishable from a warm floor that is fine. 21 new tests in tests/test_runpod_warm_floor_rerent.py pin the caution, not just the fix: the failure mode in the other direction is forgetting a pod that is alive and billing. test_runpod_client_scale's source assertion moved to the new seam. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
9c82c3f1