A qwen38 pool pod (A40, $0.59/hr) was found running at 21:30, outside its
08:00-20:00 schedule window, hours after it should have been down. Its config
was correct and the pool said so every pass — "outside its warm-pod window —
warm floor 0" — while the pod billed on.

Two mechanisms each did their job and left a gap between them. The 21:17
unattended upgrade restarted coderai; the pod survived it. With a warm floor of
0 nothing provisioned, and adoption (`find_shared_pod`) only runs when a pool
actually needs a pod — so no pool tracked it, and no pool applied
`idle_timeout_s` to it. Meanwhile the reaper skips everything in
`_known_pod_ids()`, and the dead process's registry entry was still in there,
because `registered_pod_ids()` returned every key in the file regardless of who
wrote it or whether that process still existed. The pod was simultaneously
unmanaged and shielded. Nothing in the registry could tell the two apart: `at`
was written once at registration and never updated.

`at` is now a heartbeat. `touch_registered_pods()` refreshes this process's own
entries from the scaler loop (every fourth 15 s pass), `live_registered_pod_ids()`
counts ours unconditionally and a sibling's only while its heartbeat is fresh,
and `_known_pod_ids()` uses that. An entry whose owner has been silent for
REAP_GRACE_SECONDS stops shielding its pod, and the reaper treats it like any
other leftover. The grace is 1200 s against a 60 s heartbeat, so a sibling
between passes is never at risk.

The trade this accepts: a process still running but whose maintenance loop has
died stops heartbeating, and a sibling will reap its pods after the grace.
Twenty minutes of silence from the loop that is supposed to manage a pod is
itself the failure.

Also: a pod the reaper kills is now recorded in the ledger, computed from a new
`created` field (distinct from the heartbeat, which moves). Those hours were
invisible before — only `_terminate` recorded spend, and that only runs for a
pod a pool was tracking, which is exactly not this case. And the reaper
unregisters what it terminates, so it cannot try twice.

The pod itself was terminated by hand and its stale entry removed; the account
is back to zero pods.

11 tests in tests/test_runpod_registry_heartbeat.py, including the whole bug in
one assertion: an orphan id is in `registered_pod_ids()` and not in
`_known_pod_ids()`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW