runpod: hours nobody ledgered, and a flag the ledger threw away (v0.3.63)

Two gaps I left open last night, both about the ledger telling the truth.

**A pod that went away without a pool noticing was never billed to anyone.** A
pool closes its own pods' bills — `_terminate` for one it retires, `forget` for
one RunPod says is gone — and neither runs for a pod only the REGISTRY knew
about: one orphaned by a restart, or terminated from the console or straight
through the RunPod API. Observed: pod lgdgfs749chkib, killed by hand, never
appeared in the ledger while two pods that went through coderai did.
`close_bill_from_registry()` now computes the hours from the registry's own
`created` — the only record of when a pod started once no handle exists — and
records them against the pool it belonged to. Both reaper paths use it, so the
"terminated as stale" and "vanished from the account" cases cannot drift apart,
and a pod a pool still holds is deliberately skipped: `forget` ledgers that one,
and counting it here as well would double-count every pod RunPod reclaims.

**`record_spend` dropped the `reaped` flag it was being handed.** It keeps a
deliberate allow-list, which is right — the file is persisted and pruned for
every pod — but the list has to carry what distinguishes one kind of charge from
another. A pod the reaper killed looked in the file exactly like one a pool
retired on purpose. `reaped` and `reason` are kept now.

Writing the test for the first fix found a third thing, and it was mine: v0.3.56
made a stale registry entry stop counting as "known" so the reaper could kill the
pod it named — but the reconcile was only ever asked about known ids, so that
same change stopped stale entries being RECONCILED. An entry for a pod that no
longer exists was then never pruned and its hours never attributed. The reaper
now reconciles against the whole registry; `known` keeps its one job, which is
deciding what is shielded from reaping.

9 tests in tests/test_ledger_out_of_band.py, including the double-count guard
and the pruning regression. test_runpod_client_scale's source anchor moves with
the change and says why.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.62"
+__version__ = "0.3.63"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/runpod_ledger.py

@@ -80,7 +80,17 @@ def record_spend(model_key: str, usd: float, meta: dict = None) -> None:
     now = time.time()
     rec = {"ts": now, "usd": float(usd)}
     if meta:
-        rec.update({k: meta[k] for k in ("pod_id", "gpu", "is_spot") if k in meta})
+        # A deliberate allow-list, not the whole of ``meta``: the ledger is
+        # persisted and pruned, and anything kept here is kept for every pod for
+        # as long as the window lasts. It has to include the fields a reader
+        # needs to tell one kind of charge from another, though — `reaped` was
+        # being passed and silently dropped, so a pod the reaper killed looked
+        # in the file exactly like one a pool retired on purpose, and `reason`
+        # was not recorded at all. Both answer "why am I being billed for this",
+        # which is the question a ledger exists for.
+        rec.update({k: meta[k] for k in
+                    ("pod_id", "gpu", "is_spot", "reaped", "reason")
+                    if k in meta})
     with _lock:
         data = _load()
         for key in (model_key or "?", _GLOBAL_KEY):

codai/api/runpod_worker.py

@@ -4669,6 +4669,43 @@ _absent_before: set = set()
 _DEAD_STATUSES = ("TERMINATED", "EXITED", "DEAD", "FAILED")
 
 
+def close_bill_from_registry(pod_id: str, reason: str) -> float:
+    """Ledger the hours of a pod NO POOL was holding. Returns what was recorded.
+
+    A pool closes its own pods' bills — ``_terminate`` for one it retires,
+    ``forget`` for one RunPod says is gone. Neither runs for a pod that only the
+    registry knew about: one orphaned by a restart, or killed from the console or
+    straight through the RunPod API. Those hours were billed and went into the
+    ledger as nothing at all, which is the one place they must appear: a cost
+    that is invisible cannot be budgeted against, and the surprise always
+    arrives later, on the invoice.
+
+    Computed from the registry's own ``created`` — the only record of when the
+    pod started once no handle exists — so it survives the owner that rented it.
+    """
+    info = registry_entry(pod_id)
+    rate = float(info.get("hourly_usd") or 0.0)
+    began = float(info.get("created") or info.get("at") or 0.0)
+    if not (rate and began):
+        print(f"[runpod] cannot attribute {pod_id}'s spend: the registry records "
+              f"{'no rate' if not rate else 'no start time'} for it", flush=True)
+        return 0.0
+    cost = (time.time() - began) / 3600.0 * rate
+    try:
+        from codai.api import runpod_ledger
+        runpod_ledger.record_spend(
+            str(info.get("pool") or "(orphan)"), cost,
+            {"pod_id": str(pod_id), "gpu": str(info.get("gpu") or "(orphan)"),
+             "reaped": True, "reason": reason})
+    except Exception as exc:
+        print(f"[runpod] could not write {pod_id}'s spend to the ledger: {exc}",
+              flush=True)
+        return 0.0
+    print(f"[runpod] pod {pod_id} ({reason}) billed ~${cost:.3f} — recorded "
+          f"against {info.get('pool') or '(orphan)'}", flush=True)
+    return cost
+
+
 def reconcile_pools(live_status: dict, candidates=None) -> set:
     """Reconcile pools (and the registry) against what RunPod actually has.
 
@@ -4830,7 +4867,21 @@ def reap_orphans() -> int:
     live_status = {str(p.get("id")): str(p.get("status") or "")
                    for p in pods if p.get("id")}
     try:
-        for gone in reconcile_pools(live_status, known):
+        # Whose bill a POOL will close, read before the reconcile removes them
+        # from their pools: `forget` ledgers those, and attributing them again
+        # here would double-count every pod RunPod reclaims.
+        with _pools_lock:
+            _pooled = {str(h.pod_id) for p in _pools.values()
+                       for h in list(p.pods)}
+        # Judged against the WHOLE registry, not just the entries that still
+        # have an owner. `known` is about shielding a pod from the reaper;
+        # deciding whether a pod still exists is this function's own job, and
+        # when 0.3.56 made a stale entry stop counting as known it also stopped
+        # it being reconciled — so an entry for a pod that no longer exists was
+        # never cleaned up and its hours were never attributed.
+        for gone in reconcile_pools(live_status, known | registered_pod_ids()):
+            if str(gone) not in _pooled:
+                close_bill_from_registry(gone, "gone from the account listing")
             unregister_pod(gone)
     except Exception as exc:
         print(f"[runpod-reaper] reconcile: {exc}", flush=True)
@@ -4865,26 +4916,13 @@ def reap_orphans() -> int:
         if age < REAP_GRACE_SECONDS:
             continue                      # too young to be a leftover
         try:
-            info = registry_entry(pid)
             client.terminate_pod(pid)
             reaped += 1
-            # A pod the reaper kills billed real money, and until now it left no
-            # trace in the ledger: the spend was recorded by _terminate, which
-            # only runs for a pod a pool was tracking. An orphan's hours were
-            # therefore invisible in exactly the case where they are most
-            # surprising.
-            try:
-                from codai.api import runpod_ledger
-                rate = float(info.get("hourly_usd") or 0.0)
-                began = float(info.get("created") or info.get("at") or 0.0)
-                if rate and began:
-                    runpod_ledger.record_spend(
-                        str(info.get("pool") or "(orphan)"),
-                        (time.time() - began) / 3600.0 * rate,
-                        {"pod_id": pid, "gpu": "(orphan)", "reaped": True})
-            except Exception as exc:
-                print(f"[runpod-reaper] could not attribute {pid}'s spend: {exc}",
-                      flush=True)
+            # A pod the reaper kills billed real money, and it used to leave no
+            # trace: the spend was recorded by _terminate, which only runs for a
+            # pod a pool was tracking. Same helper as the vanished case, so the
+            # two paths cannot drift apart.
+            close_bill_from_registry(pid, "reaped as a stale pod")
             unregister_pod(pid)
             print(f"[runpod-reaper] terminated STALE pod {pid} ({name}) — "
                   "not tracked by any pool and its owner stopped reporting",

docs/runpod.md

@@ -359,6 +359,14 @@ an entry recording neither image nor card — written before this check existed
 is refused rather than trusted: one extra pod once, against a silent answer from
 the wrong code. Every refusal says which pod and why.
 
+Every pod's hours reach the ledger, whichever way it ends: a pool retires one
+(`_terminate`), RunPod takes one away (`forget`), the reaper kills a stale one,
+or it disappears entirely — terminated from the console, or straight through the
+RunPod API. The last two have no pool to close the bill, so it is computed from
+the registry's own `created` and recorded against the pool the pod belonged to,
+marked `reaped` with the reason. A cost that is invisible cannot be budgeted
+against, and the surprise arrives later, on the invoice.
+
 Consequences worth knowing:
 
 - Pods from **another** `deployment_id` are never touched, so two CoderAI

tests/test_ledger_out_of_band.py

+"""Hours nobody ledgered: a pod that went away without a pool noticing.
+
+A pool closes its own pods' bills — `_terminate` for one it retires, `forget`
+for one RunPod says is gone. Neither runs for a pod only the REGISTRY knew
+about: one orphaned by a restart, or terminated from the console or straight
+through the RunPod API. Those hours were billed and recorded as nothing.
+
+Observed on 2026-10-13: pod lgdgfs749chkib was killed by hand through the RunPod
+API and never appeared in the ledger, while two pods that went through coderai
+did. A cost that is invisible cannot be budgeted against, and the surprise
+arrives later, on the invoice.
+
+The second half: `record_spend` kept a fixed set of fields and dropped `reaped`,
+which was being passed — so in the file a pod the reaper killed looked exactly
+like one a pool retired on purpose.
+"""
+import json
+import pathlib
+import sys
+import time
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from codai.api import runpod_ledger as ledger                      # noqa: E402
+from codai.api import runpod_worker as rw                          # noqa: E402
+
+
+def _registry(monkeypatch, tmp_path, entries):
+    path = tmp_path / "runpod-pods.json"
+    path.write_text(json.dumps(entries))
+    monkeypatch.setattr(rw, "_pod_registry_path", lambda: str(path))
+    return path
+
+
+def _spend_sink(monkeypatch):
+    seen = []
+    monkeypatch.setattr(ledger, "record_spend",
+                        lambda key, usd, meta=None: seen.append((key, usd, meta)))
+    return seen
+
+
+# --- the fields the ledger keeps -------------------------------------------
+
+def test_the_ledger_keeps_the_reaped_flag(monkeypatch, tmp_path):
+    monkeypatch.setattr(ledger, "_PATH", str(tmp_path / "led.json"), raising=False)
+    monkeypatch.setattr(ledger, "_ledger_path", lambda: str(tmp_path / "led.json"),
+                        raising=False)
+    kept = {}
+    monkeypatch.setattr(ledger, "_load", lambda: {})
+    monkeypatch.setattr(ledger, "_save", lambda d: kept.update(d))
+    ledger.record_spend("m", 0.5, {"pod_id": "p1", "gpu": "(orphan)",
+                                   "reaped": True, "reason": "stale",
+                                   "ignored": "x"})
+    rec = kept["m"][0]
+    assert rec["reaped"] is True and rec["reason"] == "stale"
+    assert rec["pod_id"] == "p1" and rec["gpu"] == "(orphan)"
+    # Still an allow-list: the ledger is persisted for every pod, so it does not
+    # grow a field because a caller passed one.
+    assert "ignored" not in rec
+
+
+# --- closing a bill from the registry alone --------------------------------
+
+def test_a_pods_hours_are_computed_from_when_it_was_created(monkeypatch, tmp_path):
+    born = time.time() - 3600
+    _registry(monkeypatch, tmp_path,
+              {"p1": {"pool": "m", "proc": "dead", "created": born, "at": born,
+                      "hourly_usd": 0.59, "url": "u", "key": "k"}})
+    seen = _spend_sink(monkeypatch)
+    cost = rw.close_bill_from_registry("p1", "gone from the account listing")
+    assert 0.58 < cost < 0.60                  # an hour at $0.59
+    key, usd, meta = seen[0]
+    assert key == "m" and abs(usd - cost) < 1e-9
+    assert meta["pod_id"] == "p1" and meta["reaped"] is True
+    assert meta["reason"] == "gone from the account listing"
+
+
+def test_an_entry_with_no_rate_says_so_instead_of_recording_zero(monkeypatch,
+                                                                 tmp_path, capsys):
+    _registry(monkeypatch, tmp_path,
+              {"p1": {"pool": "m", "created": time.time() - 60}})
+    seen = _spend_sink(monkeypatch)
+    assert rw.close_bill_from_registry("p1", "reaped") == 0.0
+    assert seen == []
+    # Silence would leave an unexplained gap between the invoice and the ledger.
+    assert "cannot attribute p1's spend" in capsys.readouterr().out
+
+
+def test_an_unknown_pod_is_not_invented(monkeypatch, tmp_path):
+    _registry(monkeypatch, tmp_path, {})
+    seen = _spend_sink(monkeypatch)
+    assert rw.close_bill_from_registry("nope", "reaped") == 0.0 and seen == []
+
+
+def test_a_pod_with_no_pool_recorded_is_attributed_to_orphan(monkeypatch, tmp_path):
+    born = time.time() - 1800
+    _registry(monkeypatch, tmp_path,
+              {"p1": {"created": born, "at": born, "hourly_usd": 0.59}})
+    seen = _spend_sink(monkeypatch)
+    rw.close_bill_from_registry("p1", "reaped as a stale pod")
+    assert seen[0][0] == "(orphan)"
+
+
+# --- the reaper uses it, once, for both of its paths ----------------------
+
+def test_the_reaper_attributes_a_pod_that_vanished(monkeypatch, tmp_path):
+    """RunPod reclaimed it, or someone killed it from the console: no pool had
+    it, so nothing closed the bill."""
+    born = time.time() - 7200
+    _registry(monkeypatch, tmp_path,
+              {"p1": {"pool": "m", "proc": "dead", "created": born, "at": born,
+                      "hourly_usd": 0.59, "url": "u", "key": "k"}})
+    seen = _spend_sink(monkeypatch)
+
+    class _Client:
+        def list_pods(self):
+            return []                      # the account no longer has it
+
+    import codai.api.runpod_client as rc
+    monkeypatch.setattr(rc, "RunpodClient", lambda *a, **k: _Client())
+    monkeypatch.setattr(rw, "_deploy_tag", lambda *a, **k: "dep")
+
+    class _Acct:
+        enabled = True
+        api_key = "k"
+        deployment_id = "dep"
+
+    import codai.models.manager as mm
+    monkeypatch.setattr(mm, "get_active_runpod_config", lambda: _Acct())
+    # Absent from TWO listings in a row before anything acts — one API page is
+    # never evidence (the rule reconcile_pools exists to enforce).
+    rw.reap_orphans()
+    assert seen == []
+    rw.reap_orphans()
+    assert seen and seen[0][2]["reason"] == "gone from the account listing"
+
+
+def test_a_pod_a_pool_still_holds_is_not_counted_twice(monkeypatch, tmp_path):
+    """`forget` closes that one's bill; attributing it here as well would
+    double-count every pod RunPod reclaims."""
+    born = time.time() - 600
+    _registry(monkeypatch, tmp_path,
+              {"p1": {"pool": "m", "proc": rw._PROC_ID, "created": born,
+                      "at": born, "hourly_usd": 0.59, "url": "u", "key": "k"}})
+    calls = []
+    monkeypatch.setattr(rw, "close_bill_from_registry",
+                        lambda pid, reason: calls.append(pid))
+    monkeypatch.setattr(rw, "reconcile_pools", lambda live, known: {"p1"})
+    monkeypatch.setattr(rw, "unregister_pod", lambda pid: None)
+
+    class _Pool:
+        model_key = "m"
+        pods = [rw.PodHandle(pod_id="p1", url="u", hourly_usd=0.59,
+                             started_at=born)]
+
+        def __init__(self):
+            import threading
+            self._cv = threading.Condition()
+
+    class _Client:
+        def list_pods(self):
+            return [{"id": "zzz", "name": "coderai-dep-other", "status": "RUNNING",
+                     "uptime_s": 10}]
+
+    import codai.api.runpod_client as rc
+    monkeypatch.setattr(rc, "RunpodClient", lambda *a, **k: _Client())
+    monkeypatch.setattr(rw, "_deploy_tag", lambda *a, **k: "dep")
+
+    class _Acct:
+        enabled = True
+        api_key = "k"
+        deployment_id = "dep"
+
+    import codai.models.manager as mm
+    monkeypatch.setattr(mm, "get_active_runpod_config", lambda: _Acct())
+    with rw._pools_lock:
+        rw._pools["m"] = _Pool()
+    try:
+        rw.reap_orphans()
+    finally:
+        with rw._pools_lock:
+            rw._pools.pop("m", None)
+    assert calls == [], "a pool-held pod was ledgered twice"
+
+
+def test_both_reaper_paths_share_one_implementation():
+    src = (ROOT / "codai/api/runpod_worker.py").read_text()
+    assert src.count("close_bill_from_registry(") >= 3    # def + two call sites
+    # The old inline copy must not come back.
+    assert 'runpod_ledger.record_spend(\n                        str(info.get("pool")' not in src
+
+
+def test_a_registry_entry_for_a_pod_that_no_longer_exists_is_cleaned_up(
+        monkeypatch, tmp_path):
+    """Found by the test above, and caused by v0.3.56.
+
+    Making a stale entry stop counting as "known" (so the reaper could kill the
+    pod it named) also stopped it being RECONCILED, because the reconcile was
+    only asked about known ids. An entry for a pod that no longer exists was
+    then never removed — it would sit in the registry for the life of the
+    install, and id() reuse aside, it is exactly the kind of record that later
+    gets believed.
+    """
+    born = time.time() - 7200
+    path = _registry(monkeypatch, tmp_path,
+                     {"p1": {"pool": "m", "proc": "dead", "created": born,
+                             "at": born, "hourly_usd": 0.59, "url": "u",
+                             "key": "k"}})
+    _spend_sink(monkeypatch)
+
+    class _Client:
+        def list_pods(self):
+            return []
+
+    import codai.api.runpod_client as rc
+    monkeypatch.setattr(rc, "RunpodClient", lambda *a, **k: _Client())
+    monkeypatch.setattr(rw, "_deploy_tag", lambda *a, **k: "dep")
+
+    class _Acct:
+        enabled = True
+        api_key = "k"
+        deployment_id = "dep"
+
+    import codai.models.manager as mm
+    monkeypatch.setattr(mm, "get_active_runpod_config", lambda: _Acct())
+    rw.reap_orphans()
+    rw.reap_orphans()                       # the two-sweep rule
+    assert json.loads(path.read_text()) == {}

tests/test_runpod_client_scale.py

@@ -481,7 +481,13 @@ def test_the_reaper_prunes_entries_for_pods_that_are_gone():
     reap_orphans still feeds it the listing and acts on what comes back."""
     src = __import__("pathlib").Path(rw.__file__).read_text()
     blk = src.split("def reap_orphans")[1].split("def _scaler_loop")[0]
-    assert "reconcile_pools(live_status, known)" in blk
+    # Against the WHOLE registry since v0.3.63, not only the entries that
+    # still have an owner: `known` decides what is shielded from the
+    # reaper, while deciding what still EXISTS is the reconcile's own
+    # job. When v0.3.56 made a stale entry stop counting as known, it
+    # also stopped it being reconciled — so the entry was never pruned
+    # and the pod's hours were never attributed.
+    assert "reconcile_pools(live_status, known | registered_pod_ids())" in blk
     assert "unregister_pod(gone)" in blk