The first real pod test found this in seconds: coderai runs one engine
per GPU, each with its own pool registry AND its own stale-pod reaper, so
each engine sees its siblings' pods as untracked orphans. The nvidia
engine created a pod; the radeon engine terminated it four seconds later;
radeon created one, nvidia killed that — a mutual kill loop that rented
and destroyed pods until I disabled RunPod. Every boot then "failed"
after its 300s port timeout, because the pod had been deleted underneath
it, and the pool dutifully retried on the next machine.

Two fixes, because one is not enough:

A shared registry (config_dir/runpod-pods.json, flock'd) records every
pod this DEPLOYMENT owns. _known_pod_ids now reads it, so an engine with
no pools of its own still knows its siblings' pods are not orphans.

A grace period (REAP_GRACE_SECONDS, 20 min) refuses to reap any pod
younger than a boot budget, whatever the registry says — the backstop for
the race a registry cannot close, such as a process that died between
creating a pod and recording it.

No pods were left billing: the mutual reaping cleaned up after itself and
the account shows zero pods.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY