every restart left the fleet it had already paid for (v0.3.103)

Adoption existed, but only ON DEMAND: it runs inside `acquire()`, when a
request needs a pod. Nothing reconciled the registry at boot. So after every
restart this deployment's surviving pods sat unowned until either a request
happened to want that pool or the reaper's 20-minute grace expired — and an
hourly unattended upgrade could pay for a fleet twice, once for the pods it
abandoned and again for the ones it rented beside them.

Observed 2026-10-16. A forced upgrade at 00:01 orphaned three pods. At 00:04
the surya pool rented a NEW pod while a healthy surya pod from before the
restart was still running and still in the registry. The three were reaped at
00:21 having billed $1.11 between them: "not tracked by any pool and its owner
stopped reporting".

THREE FIXES.

1. A POD THAT WAS BOOTING COULD NEVER BE ADOPTED. register_pod writes the url
   only once the pod is ready — `# url once ready` — and find_shared_pod
   skipped a url-less entry in SILENCE. These images take 6-14 minutes to boot,
   so a restart has a good chance of landing inside one, and the pod it lands
   on is unadoptable for the rest of its paid life. The pod id is enough to ask
   RunPod where it went, so an entry with no address now recovers one.

2. TWO OF THE THREE REFUSALS SAID NOTHING. No url and a failed health probe
   both `continue`d without a word, so "rented beside a pod it could have
   adopted" and "had nothing to adopt" produced identical logs. That is why
   tonight's case cannot be pinned down: the evidence was never written. Every
   refusal now names the pod and the reason.

3. NOTHING RECLAIMED AT STARTUP. `reclaim_registered_pods()` walks the registry
   when the scaler starts and takes this deployment's pods back into their
   pools — before `reap_orphans`, deliberately: a pod we are already paying for
   should rejoin its pool, not be terminated and rented again a minute later.
   It takes back EVERY pod of a pool, not one; the first version of this
   reclaimed one of sixteen and left fifteen billing, which is the same bug
   with extra steps. Bounded by how many the registry holds for that key, since
   adoption removes nothing by itself and the loop must end whatever it answers.
   A `pool:` key is left to the on-demand path: its shape comes from whichever
   model loads first, and guessing would be worse.

Also: `scale_up_max_parallel_boots` is now editable in the admin model form
(and survives the save). It decides how many GPUs start BILLING together when a
pool grows — a floor of 16 fills in waves of 4 at the default — which is
exactly the kind of thing an operator should not have to edit models.json for.

17 tests. Suite: 3183 passed, the same 13 known-stale failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.102"
+__version__ = "0.3.103"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -3166,7 +3166,10 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                           if isinstance(sv, str) else bool(sv))
         # ints
         for k in ("container_disk_gb", "volume_gb", "port", "ctx", "min_pods",
-                  "max_pods", "scale_up_inflight_per_pod", "scale_down_after_s",
+                  "max_pods", "scale_up_inflight_per_pod",
+                  # How many GPUs may start billing together when the pool
+                  # grows. Settable in the form, so it must survive the save.
+                  "scale_up_max_parallel_boots", "scale_down_after_s",
                   "scale_down_costliest_first",
                   "max_inflight_per_pod",
                   "idle_timeout_s",

codai/admin/templates/models.html

@@ -1131,6 +1131,11 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
               <input type="number" id="cfg-rp-max-inflight" class="form-input" min="0" placeholder="0 = unlimited">
               <span class="form-hint" style="font-size:11px">Above this a request waits for a slot instead of piling on.</span>
             </div>
+            <div class="form-row" style="margin:0">
+              <label class="form-label">Boots at once <span class="muted">(parallel)</span></label>
+              <input type="number" id="cfg-rp-parallel-boots" class="form-input" min="1" placeholder="4">
+              <span class="form-hint" style="font-size:11px">How many pods may be booting together when the pool grows. This is also how many GPUs start <b>billing</b> together, which is why it is not simply <code>max_pods</code>: a pool filling a floor of 16 four at a time reaches it in waves, and a mistake costs one wave rather than the whole fleet. Raise it when a batch is waiting and the ramp is the bottleneck.</span>
+            </div>
             <div class="form-row" style="margin:0;align-self:end">
               <label style="display:flex;align-items:center;gap:.3rem;font-size:12px"><input type="checkbox" id="cfg-rp-sticky" style="width:auto" checked> Sticky sessions</label>
               <label style="display:flex;align-items:center;gap:.3rem;font-size:12px" title="When shedding a surplus pod, give back the dearest one first instead of the least loaded. Only a mixed fleet has anything to choose between — pods take whatever the market offered when they were rented. Note that an on-demand pod costs more than a spot one, so this prefers to keep the interruptible ones."><input type="checkbox" id="cfg-rp-costliest" style="width:auto"> Scale down the most expensive first</label>
@@ -4952,6 +4957,7 @@ function _populateRunpod(s){
   set('cfg-rp-health-path', rp.health_path || '');
   set('cfg-rp-docker-args', rp.docker_args || '');
   set('cfg-rp-scale-up', rp.scale_up_inflight_per_pod != null ? rp.scale_up_inflight_per_pod : '');
+  set('cfg-rp-parallel-boots', rp.scale_up_max_parallel_boots != null ? rp.scale_up_max_parallel_boots : '');
   set('cfg-rp-scale-down', rp.scale_down_after_s != null ? rp.scale_down_after_s : '');
   set('cfg-rp-max-inflight', rp.max_inflight_per_pod != null ? rp.max_inflight_per_pod : '');
   chk('cfg-rp-sticky', rp.sticky_sessions !== false);
@@ -5135,6 +5141,7 @@ function _collectRunpod(){
     health_path:        (val('cfg-rp-health-path') || '').trim(),
     docker_args:        (val('cfg-rp-docker-args') || '').trim(),
     scale_up_inflight_per_pod: intOrU('cfg-rp-scale-up'),
+    scale_up_max_parallel_boots: intOrU('cfg-rp-parallel-boots'),
     scale_down_after_s:        intOrU('cfg-rp-scale-down'),
     max_inflight_per_pod:      intOrU('cfg-rp-max-inflight'),
     sticky_sessions:    document.getElementById('cfg-rp-sticky').checked,

codai/api/runpod_worker.py

@@ -5374,7 +5374,20 @@ def find_shared_pod(pool_key: str, health_path: str = "/v1/models",
             continue                       # our own; the local pool handles it
         url = str(info.get("url") or "")
         if not url:
-            continue
+            # A pod that was still BOOTING when its owner died. register_pod
+            # writes the url only once the pod is ready, so a restart landing
+            # inside a boot — and these images take 6-14 minutes to boot — left
+            # an entry nothing could ever adopt: skipped here in silence, then
+            # billed until the reaper's grace expired. The pod id is enough to
+            # ask RunPod where it is.
+            url = _recover_pod_url(pod_id, info)
+            if not url:
+                print(f"[runpod] not adopting pod {pod_id}: it has no address "
+                      f"recorded and RunPod does not give one yet (it was "
+                      f"still booting when its owner stopped)", flush=True)
+                continue
+            print(f"[runpod] pod {pod_id} was mid-boot when its owner stopped; "
+                  f"recovered its address from RunPod: {url}", flush=True)
         if image:
             theirs = str(info.get("image") or "")
             if theirs != str(image):
@@ -5392,9 +5405,48 @@ def find_shared_pod(pool_key: str, health_path: str = "/v1/models",
         key = str(info.get("key") or "") or api_key
         if _pod_health_ok(url, path=health_path, api_key=key):
             return pod_id, url, key, float(info.get("hourly_usd") or 0.0)
+        # The last silent refusal. A pod that does not answer may be dead, may
+        # be mid-warm-up, or may want a token we do not have — and renting
+        # beside it looks identical from the log either way.
+        print(f"[runpod] not adopting pod {pod_id}: it did not answer "
+              f"{health_path} at {url} (dead, still loading, or it wants a "
+              f"token this pool does not have)", flush=True)
     return None, "", "", 0.0
 
 
+def _recover_pod_url(pod_id: str, info: dict) -> str:
+    """Where a registered pod is now, asked of RunPod rather than remembered.
+
+    Only for an entry with no url: the owner died between creating the pod and
+    it becoming ready. Best effort — a pod that still has no port gives nothing
+    and the caller says so.
+    """
+    try:
+        from codai.models.manager import get_active_runpod_config
+        from codai.api.runpod_client import RunpodClient, direct_tcp_url, pod_proxy_url
+        acct = get_active_runpod_config()
+        if acct is None or not getattr(acct, "api_key", ""):
+            return ""
+        live = RunpodClient(acct).get_pod(str(pod_id))
+    except Exception as exc:
+        print(f"[runpod] could not ask RunPod where pod {pod_id} is: {exc}",
+              flush=True)
+        return ""
+    if str(live.get("status") or "").upper() in ("TERMINATED", "EXITED", "FAILED"):
+        return ""
+    ports = live.get("ports") or []
+    if not ports:
+        return ""
+    # The same two shapes _provision_one chooses between, decided the same way:
+    # a recorded direct-TCP address is https to a bare IP.
+    try:
+        port = int(info.get("port") or 0) or 8000
+    except Exception:
+        port = 8000
+    direct = direct_tcp_url(ports, port, tls=True)
+    return direct or pod_proxy_url(str(pod_id), port)
+
+
 def unregister_pod(pod_id: str) -> None:
     """Forget a pod we terminated, so the registry does not grow without bound."""
     if not pod_id:
@@ -5764,8 +5816,89 @@ def reap_orphans() -> int:
     return reaped
 
 
+def reclaim_registered_pods() -> int:
+    """Take this deployment's surviving pods back into their pools, at startup.
+
+    Adoption existed, but only ON DEMAND: it runs inside ``acquire`` when a
+    request needs a pod. Nothing reconciled the registry at boot, so after a
+    restart the pods this deployment was already paying for sat unowned until
+    either a request happened to want that pool or the reaper's 20-minute grace
+    expired. "Every restart leaves orphaned pods" was an accurate description
+    of the design rather than a bug in one branch: an hourly unattended upgrade
+    could pay for a fleet twice over, once for the pods it abandoned and again
+    for the ones it rented beside them.
+
+    Reclaiming here costs one registry read and one health probe per pod, and
+    saves both the double billing and a cold boot the fleet had already paid
+    for. Returns how many were taken back.
+
+    Only pods whose pool key names a model this install configures: a shared
+    pool key (``pool:…``) has no single entry to rebuild its shape from, and
+    guessing would be worse than the on-demand path those already have.
+    """
+    try:
+        from codai.models.manager import get_active_runpod_config
+        acct = get_active_runpod_config()
+    except Exception:
+        acct = None
+    if acct is None or not getattr(acct, "enabled", False):
+        return 0
+    try:
+        data = _registry_update(lambda d: False) or {}
+    except Exception as exc:
+        print(f"[runpod] reclaim: cannot read the registry ({exc})", flush=True)
+        return 0
+    keys = {}
+    for pod_id, info in data.items():
+        if not isinstance(info, dict) or _is_this_process(info):
+            continue
+        key = str(info.get("pool") or "")
+        if key:
+            keys[key] = keys.get(key, 0) + 1
+    if not keys:
+        return 0
+    taken = 0
+    for key in keys:
+        if key.startswith("pool:"):
+            print(f"[runpod] reclaim: leaving shared pool {key!r} to the "
+                  f"on-demand path — its shape comes from whichever model "
+                  f"loads first, not from the registry", flush=True)
+            continue
+        entry = _model_entry(key)
+        if not entry:
+            print(f"[runpod] reclaim: {key!r} has pods registered but this "
+                  f"install no longer configures it — leaving them to the "
+                  f"reaper", flush=True)
+            continue
+        try:
+            mcfg = parse_model_runpod(entry.get("runpod") or {})
+            served = str((entry.get("runpod") or {}).get("served_model")
+                         or entry.get("id") or key)
+            pool = get_pod_pool(key, acct, mcfg, served, entry=entry)
+            # EVERY pod of this pool, not one. A fleet of sixteen reclaimed one
+            # pod and left fifteen billing, which is the bug with extra steps.
+            # Bounded by how many the registry actually holds for this key, so
+            # a pool that keeps answering "yes" cannot spin: adoption removes
+            # nothing by itself, and the loop must end whatever it returns.
+            for _ in range(keys[key]):
+                if not pool._adopt_shared_pod():
+                    break
+                taken += 1
+                print(f"[runpod] reclaim: took back a pod for {key!r} that "
+                      f"outlived the process that rented it", flush=True)
+        except Exception as exc:
+            print(f"[runpod] reclaim: {key!r}: {exc}", flush=True)
+    return taken
+
+
 def _scaler_loop():
     tick = 0
+    # Take our own pods back BEFORE reaping: a pod this deployment is already
+    # paying for should rejoin its pool, not be killed and rented again.
+    try:
+        reclaim_registered_pods()
+    except Exception as exc:
+        print(f"[runpod] reclaim: startup: {exc}", flush=True)
     # Reap once promptly on start to catch orphans left by a crash/restart.
     try:
         reap_orphans()

tests/test_runpod_restart_reclaim.py

+"""A restart should pick up the fleet it already paid for.
+
+Adoption existed but ran only inside `acquire()`, when a request needed a pod.
+Nothing reconciled the registry at boot, so after every restart the pods this
+deployment was already paying for sat unowned until a request happened to want
+that pool or the reaper's 20-minute grace expired. Observed 2026-10-16: a
+forced upgrade at 00:01 orphaned three pods; at 00:04 the surya pool rented a
+NEW pod while a healthy surya pod from before the restart was still running and
+still in the registry; the three were reaped at 00:21 having billed $1.11
+between them, "not tracked by any pool and its owner stopped reporting".
+
+And a pod that was still BOOTING when its owner died could never be adopted at
+all: register_pod writes the url only once the pod is ready, and
+find_shared_pod skipped a url-less entry in silence. These images take 6-14
+minutes to boot, so a restart has a good chance of landing inside one.
+"""
+
+import types
+
+import pytest
+
+from codai.api import runpod_worker as rw
+
+
+@pytest.fixture
+def registry(monkeypatch, tmp_path):
+    """A registry backed by a dict, not a file."""
+    store = {}
+
+    def _update(fn):
+        fn(store)
+        return dict(store)
+
+    monkeypatch.setattr(rw, "_registry_update", _update)
+    return store
+
+
+def _entry(pool="m", url="https://1.2.3.4:1", image="img", gpu="A40", key="k"):
+    return {"pool": pool, "url": url, "image": image, "gpu": gpu, "key": key,
+            "pid": 1, "proc": "1-deadbeef", "hourly_usd": 0.5}
+
+
+# ----------------------------------------------- a mid-boot pod is adoptable #
+def test_a_pod_with_no_url_is_recovered_from_runpod(registry, monkeypatch):
+    """The case that could never be adopted: killed mid-boot, url never written."""
+    registry["p1"] = _entry(url="")
+    monkeypatch.setattr(rw, "_recover_pod_url",
+                        lambda pid, info: "https://5.6.7.8:9")
+    monkeypatch.setattr(rw, "_pod_health_ok", lambda url, **kw: True)
+    pid, url, key, rate = rw.find_shared_pod("m", image="img", gpus=["A40"])
+    assert pid == "p1"
+    assert url == "https://5.6.7.8:9"
+
+
+def test_a_pod_that_runpod_cannot_place_is_refused_out_loud(registry, monkeypatch, capsys):
+    registry["p1"] = _entry(url="")
+    monkeypatch.setattr(rw, "_recover_pod_url", lambda pid, info: "")
+    pid, url, _k, _r = rw.find_shared_pod("m", image="img", gpus=["A40"])
+    assert pid is None
+    out = capsys.readouterr().out
+    assert "no address recorded" in out
+    assert "still booting" in out
+
+
+# --------------------------------------------- every refusal says why it refused #
+def test_a_pod_that_does_not_answer_says_so(registry, monkeypatch, capsys):
+    """Was silent. Renting beside a pod that did not answer looked identical in
+    the log to renting because there was nothing to adopt."""
+    registry["p1"] = _entry()
+    monkeypatch.setattr(rw, "_pod_health_ok", lambda url, **kw: False)
+    pid, _u, _k, _r = rw.find_shared_pod("m", image="img", gpus=["A40"])
+    assert pid is None
+    out = capsys.readouterr().out
+    assert "did not answer" in out
+    assert "p1" in out
+
+
+def test_the_image_and_card_refusals_still_speak(registry, monkeypatch, capsys):
+    registry["p1"] = _entry(image="other")
+    registry["p2"] = _entry(gpu="L40S")
+    monkeypatch.setattr(rw, "_pod_health_ok", lambda url, **kw: True)
+    rw.find_shared_pod("m", image="img", gpus=["A40"])
+    out = capsys.readouterr().out
+    assert "not adopting pod p1" in out
+    assert "not adopting pod p2" in out
+
+
+def test_a_healthy_matching_pod_is_still_adopted(registry, monkeypatch):
+    """The fix must not make adoption pickier than it was."""
+    registry["p1"] = _entry()
+    monkeypatch.setattr(rw, "_pod_health_ok", lambda url, **kw: True)
+    pid, url, key, rate = rw.find_shared_pod("m", image="img", gpus=["A40"])
+    assert (pid, url, key, rate) == ("p1", "https://1.2.3.4:1", "k", 0.5)
+
+
+def test_our_own_pods_are_left_to_their_pool(registry, monkeypatch):
+    registry["p1"] = dict(_entry(), proc=rw._PROC_ID)
+    monkeypatch.setattr(rw, "_pod_health_ok", lambda url, **kw: True)
+    assert rw.find_shared_pod("m", image="img", gpus=["A40"])[0] is None
+
+
+# ------------------------------------------------------ reclaim at startup #
+def _acct(enabled=True):
+    return types.SimpleNamespace(enabled=enabled, api_key="k")
+
+
+def test_startup_reclaims_a_pod_that_outlived_its_process(registry, monkeypatch):
+    registry["p1"] = _entry(pool="mymodel")
+    monkeypatch.setattr(rw, "_model_entry",
+                        lambda k: {"id": "mymodel", "runpod": {"mode": "pods"}})
+    adopted = []
+
+    class _Pool:
+        def _adopt_shared_pod(self):
+            adopted.append(True)
+            return True
+
+    monkeypatch.setattr(rw, "get_pod_pool", lambda *a, **k: _Pool())
+    import codai.models.manager as mgr
+    monkeypatch.setattr(mgr, "get_active_runpod_config", lambda: _acct())
+    assert rw.reclaim_registered_pods() == 1
+    assert adopted
+
+
+def test_reclaim_runs_before_the_reaper_can_kill_them():
+    """Order matters: a pod we are already paying for should rejoin its pool,
+    not be terminated and rented again a minute later."""
+    import pathlib
+    src = pathlib.Path(rw.__file__).read_text()
+    body = src.split("def _scaler_loop")[1].split("while True:")[0]
+    assert body.index("reclaim_registered_pods") < body.index("reap_orphans")
+
+
+def test_reclaim_leaves_a_shared_pool_alone(registry, monkeypatch, capsys):
+    """A `pool:` key has no single model entry to rebuild its shape from."""
+    registry["p1"] = _entry(pool="pool:shared")
+    import codai.models.manager as mgr
+    monkeypatch.setattr(mgr, "get_active_runpod_config", lambda: _acct())
+    assert rw.reclaim_registered_pods() == 0
+    assert "leaving shared pool" in capsys.readouterr().out
+
+
+def test_reclaim_says_when_a_model_is_gone(registry, monkeypatch, capsys):
+    registry["p1"] = _entry(pool="retired-model")
+    monkeypatch.setattr(rw, "_model_entry", lambda k: {})
+    import codai.models.manager as mgr
+    monkeypatch.setattr(mgr, "get_active_runpod_config", lambda: _acct())
+    assert rw.reclaim_registered_pods() == 0
+    assert "no longer configures it" in capsys.readouterr().out
+
+
+def test_reclaim_skips_our_own_entries(registry, monkeypatch):
+    registry["p1"] = dict(_entry(pool="mymodel"), proc=rw._PROC_ID)
+    import codai.models.manager as mgr
+    monkeypatch.setattr(mgr, "get_active_runpod_config", lambda: _acct())
+    assert rw.reclaim_registered_pods() == 0
+
+
+def test_reclaim_is_a_no_op_when_runpod_is_off(registry, monkeypatch):
+    registry["p1"] = _entry(pool="mymodel")
+    import codai.models.manager as mgr
+    monkeypatch.setattr(mgr, "get_active_runpod_config", lambda: _acct(enabled=False))
+    assert rw.reclaim_registered_pods() == 0
+
+
+def test_reclaim_survives_one_bad_pool(registry, monkeypatch):
+    """One model's failure must not stop the others being reclaimed."""
+    registry["p1"] = _entry(pool="bad")
+    registry["p2"] = _entry(pool="good")
+    monkeypatch.setattr(rw, "_model_entry", lambda k: {"id": k, "runpod": {}})
+
+    def _pool(key, *a, **k):
+        if key == "bad":
+            raise RuntimeError("no config")
+        return types.SimpleNamespace(_adopt_shared_pod=lambda: True)
+
+    monkeypatch.setattr(rw, "get_pod_pool", _pool)
+    import codai.models.manager as mgr
+    monkeypatch.setattr(mgr, "get_active_runpod_config", lambda: _acct())
+    assert rw.reclaim_registered_pods() == 1
+
+
+# ----------------------------------------------------- the whole fleet, not one #
+def test_reclaim_takes_back_every_pod_of_a_pool(registry, monkeypatch):
+    """A fleet of sixteen reclaimed ONE pod and left fifteen billing in the
+    first version of this — the bug with extra steps."""
+    for i in range(16):
+        registry[f"p{i}"] = _entry(pool="mymodel")
+    monkeypatch.setattr(rw, "_model_entry", lambda k: {"id": k, "runpod": {}})
+    taken = {"n": 0}
+
+    class _Pool:
+        def _adopt_shared_pod(self):
+            taken["n"] += 1
+            return taken["n"] <= 16
+
+    monkeypatch.setattr(rw, "get_pod_pool", lambda *a, **k: _Pool())
+    import codai.models.manager as mgr
+    monkeypatch.setattr(mgr, "get_active_runpod_config", lambda: _acct())
+    assert rw.reclaim_registered_pods() == 16
+
+
+def test_a_pool_that_keeps_saying_yes_cannot_spin(registry, monkeypatch):
+    """Adoption removes nothing by itself, so the loop is bounded by what the
+    registry holds rather than by the answer."""
+    registry["p1"] = _entry(pool="mymodel")
+    monkeypatch.setattr(rw, "_model_entry", lambda k: {"id": k, "runpod": {}})
+    monkeypatch.setattr(rw, "get_pod_pool", lambda *a, **k: types.SimpleNamespace(
+        _adopt_shared_pod=lambda: True))          # always yes
+    import codai.models.manager as mgr
+    monkeypatch.setattr(mgr, "get_active_runpod_config", lambda: _acct())
+    assert rw.reclaim_registered_pods() == 1      # one entry -> one attempt
+
+
+# ------------------------------------------- the knob is reachable from the UI #
+def test_parallel_boots_is_editable_in_the_admin_form():
+    """It decides how many GPUs start billing together, which is exactly the
+    kind of thing an operator should not have to edit models.json for."""
+    import pathlib
+    from codai import admin
+    html = (pathlib.Path(admin.__file__).parent / "templates" / "models.html").read_text()
+    assert 'id="cfg-rp-parallel-boots"' in html
+    assert "rp.scale_up_max_parallel_boots" in html
+    assert "scale_up_max_parallel_boots: intOrU('cfg-rp-parallel-boots')" in html
+
+
+def test_parallel_boots_survives_the_save():
+    """A field the form sends and the save path drops is worse than no field."""
+    import pathlib
+    from codai.admin import routes
+    src = pathlib.Path(routes.__file__).read_text()
+    assert '"scale_up_max_parallel_boots"' in src