runpod: a warm floor is not satisfied by a pod that no longer exists (v0.3.44)

`keep_warm` stopped re-renting whenever a pod went away behind the pool's
back — terminated from the console, reclaimed as spot, lost with its host —
and the scaler said nothing about it. Both halves had one cause: nothing in
a pool ever re-checks a pod's health. A handle's `healthy` flag goes false
only when something tried to USE the pod and failed, or when the pool itself
retired it, and a warm-floor pod is precisely the pod with no traffic to
fail. So `maintain()` counted the handle, found `healthy_n >= keep_min`, and
neither topped up nor had anything to log. The floor stayed "satisfied" by a
machine that had not existed for hours, and the first real request paid the
full ~12-minute cold boot that keep_warm is bought to avoid.

(The class docstring claimed the scaler "re-checks health" for months. It
never has. Corrected rather than implemented: a timer-probe would have to
decide what an unanswered probe means, and that decision is what killed
seven A40s in v0.3.41.)

`reconcile_pools` instead reads the account listing the stale-pod reaper
already fetches every ~30s — authoritative, free, and unable to mistake a
busy pod for a dead one. Only positive evidence acts:

  * a dead status (TERMINATED/EXITED/DEAD/FAILED) drops the pod at once;
  * a mere absence drops it only after two listings in a row, because the
    API is eventually consistent and one page omitting a live pod used to be
    enough to unregister it — a reappearance resets the count;
  * a pod that served a request in the last 60s is kept whatever the listing
    says, which cannot mask this bug since a warm-floor pod has no traffic;
  * a provisioning pod is shielded, and a FAILED listing never gets here at
    all, so an API outage cannot empty every pool and re-rent the fleet.

`PodPool.forget()` is `discard()` minus the kill: nothing is left to
terminate, but the bill is closed in the ledger exactly as a real
termination closes it, since the pod was billed up to its death. A borrowed
`(shared)` pod is dropped without a ledger entry — its owner holds the bill.
The registry is pruned through the same two-sweep rule it used to skip.

The top-up now announces itself before spending, because a warm floor
silently failing is indistinguishable from a warm floor that is fine.

21 new tests in tests/test_runpod_warm_floor_rerent.py pin the caution, not
just the fix: the failure mode in the other direction is forgetting a pod
that is alive and billing. test_runpod_client_scale's source assertion moved
to the new seam.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.43"
+__version__ = "0.3.44"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

docs/runpod.md

@@ -334,6 +334,56 @@ Consequences worth knowing:
 
 `atexit` also tears down every pool on a clean shutdown.
 
+### The same sweep reconciles the pools (a pod that dies behind our back)
+
+The listing is not only used to find pods that should not exist. It is also the
+only place a pod that **should** exist and no longer does becomes visible.
+
+Nothing in a pool probes its own pods on a timer. A `PodHandle`'s `healthy` flag
+goes false when something tried to **use** the pod and failed, or when the pool
+itself retired it — which means a pod with no traffic on it is never discovered
+dead. That is exactly a warm-floor pod. So a `keep_warm` pod terminated from the
+RunPod console, reclaimed as spot, or lost with its host left the pool believing
+its floor was satisfied, **indefinitely and silently**: `maintain()` counted the
+handle, found `healthy >= min_pods`, and neither topped up nor logged. The first
+real request then paid a full cold boot — around 12 minutes on the Digesta
+orchestrator — which is the precise cost `keep_warm` is bought to avoid.
+
+Each successful listing is now reconciled against what the pools believe:
+
+- A pod RunPod reports as `TERMINATED`/`EXITED`/`DEAD`/`FAILED` is dropped
+  immediately. That is RunPod telling us directly; a second opinion adds nothing.
+- A pod simply **missing** from the listing is dropped only after it is missing
+  from **two listings in a row** (~30s apart). The API is eventually consistent
+  and pages; one response that omits a running pod used to be enough to
+  unregister it, and believing it costs a second rented GPU for work already
+  being done. A pod that reappears resets the count.
+- A pod that **served a request in the last 60s** is kept whatever the listing
+  says — traffic is better evidence than an API page. This cannot mask the bug
+  above, because a warm-floor pod has no traffic to speak for it.
+- A pod still being **provisioned** is shielded, and a **failed** listing never
+  reaches the reconcile at all: `reap_orphans` returns early on a `RunpodError`,
+  so an API outage cannot empty every pool and re-rent the fleet.
+
+Dropping such a handle does **not** call the API — there is nothing left to
+terminate — but it does close the bill: the ledger entry is written exactly as a
+real termination writes it, estimated up to the moment we noticed, because a
+pod that died out of band was billed right up to its death. A borrowed
+(`(shared)`) pod is dropped without a ledger entry; its owner holds the bill.
+
+Once the handle is gone, the next scaler pass sees the pool below its floor and
+says so before it spends:
+
+```
+[runpod] pod qb9ap… for 'pool:digesta-qwen38' is gone from RunPod (status TERMINATED)
+         — dropped from the pool; billed ~$0.412
+[runpod] 'pool:digesta-qwen38': warm floor 1, 0 pod(s) healthy — renting one
+```
+
+A pool with `min_pods: 0` re-rents nothing: a pod dying out of band there is
+simply gone, and renting a replacement nobody asked for is how an idle install
+bills overnight.
+
 ---
 
 ## 5. Budgets and the ledger

tests/test_runpod_client_scale.py

@@ -467,10 +467,18 @@ def test_a_booting_pod_from_a_previous_process_counts():
 
 def test_the_reaper_prunes_entries_for_pods_that_are_gone():
     """A terminated-but-registered id stays 'known', so it is shielded from
-    reaping and accumulates for the life of the install."""
+    reaping and accumulates for the life of the install.
+
+    The pruning moved into `reconcile_pools`, which judges the pools' own pods
+    and the registry from the same listing — and gates an ABSENCE on two
+    listings in a row, because the API is eventually consistent and one page
+    that omits a live pod used to be enough to unregister it. Behaviour pinned
+    in tests/test_runpod_warm_floor_rerent.py; what matters here is that
+    reap_orphans still feeds it the listing and acts on what comes back."""
     src = __import__("pathlib").Path(rw.__file__).read_text()
     blk = src.split("def reap_orphans")[1].split("def _scaler_loop")[0]
-    assert "known - live" in blk and "unregister_pod(gone)" in blk
+    assert "reconcile_pools(live_status, known)" in blk
+    assert "unregister_pod(gone)" in blk
 
 
 # ------------------------------- the pid in a container is not an identity