ocr+runpod: a hardcoded "why", and growth that rented beside its own pod (v0.3.60)

Two reports from the Digesta side, both of which my own code caused.

**"resumption is disabled server-side (ocr.surya_max_continuations = 0)".** It
is not, and never was. Both config files leave the key unset, the dataclass
default is 8, and the running process resolves 8 for either candidate config
dir — checked inside the container through the app's own loader. The number came
from this, in ocr_worker:

    elif capped and not resumed:
        info["why"] = (f"the reply stopped at the {cap}-token cap and "
                       f"resumption is off (ocr.surya_max_continuations = 0) …")

a hardcoded string in a branch that is reached four different ways: budget zero,
a generation already repeating itself, no room left in the context window, or a
reply that could not be resumed at all. It asserted a configuration value it
never read, and it was believed — twice — and acted on. A diagnostic that names
the wrong cause is worse than one that says nothing.

So the reason is now recorded where it is known (`_no_resume`) and reported as
itself. The budget actually in force goes into `model_attempts` as
`max_continuations`, because printing the cap but not the budget is why a caller
could not tell a request that landed from one that was ignored. The old message
survives for the one case where it was true, and says it is about this page
rather than the server.

**Growth rented a card beside a pod it already owned.** Adoption was attempted
in three places — `ensure_ready`, `acquire` with nothing healthy, and the warm
floor — and the growth path was not one of them. Live at 23:10: a run adopted
one surviving pod, grew by one, and rented an RTX A6000 (after walking thirty
candidates) while a second healthy pod of its own sat idle in the registry,
billing for nothing until the reaper's grace would have expired.
`_grow_in_background` now adopts first, and a source-anchored test pins all
five provisioning sites so a sixth cannot quietly forget. `seed_region` stays
excluded by design: it exists to boot a pod in a NAMED region, and adopting one
elsewhere would not fill that region's volume.

tests/test_pool_growth_adopts.py (4), four more in
tests/test_surya_thread_boundary.py, one in tests/test_runpod_scale_out.py —
whose pool fake now stubs adoption, since left real it reads this machine's own
registry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.59"
+__version__ = "0.3.60"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/runpod_worker.py

@@ -3202,7 +3202,17 @@ class RunpodPodPool:
         """
         def _run():
             try:
-                self._provision_one()
+                # Adopt before renting, here too. Only the cold paths did —
+                # `ensure_ready`, `acquire` with nothing healthy, and the warm
+                # floor — so GROWTH always rented, even with a healthy pod for
+                # this very pool sitting in the registry under another process.
+                # Seen live: a run adopted one surviving pod, grew by one, and
+                # rented a card beside a second idle pod it already owned, which
+                # then billed for nothing. Adoption refuses by itself when it
+                # would mix two tokens in one pool, so this cannot make that
+                # worse.
+                if not self._adopt_shared_pod():
+                    self._provision_one()
             except Exception as exc:
                 print(f"[runpod] scale-up for {self.model_key!r} failed: {exc}", flush=True)
             finally:

codai/ocr/workers/ocr_worker.py

@@ -215,6 +215,8 @@ def _trace_begin():
     st["tokens"] = []
     st["continued"] = 0
     st["still_truncated"] = False
+    st.pop("no_resume", None)
+    st.pop("budget", None)
     return st["calls"], st["repairs"]
 
 
@@ -299,6 +301,23 @@ def _attempt_tokens(n: int) -> None:
         pass
 
 
+def _no_resume(reason: str) -> None:
+    """Why a reply that hit its cap was NOT resumed.
+
+    The report used to assert one: "resumption is off
+    (ocr.surya_max_continuations = 0)" was a hardcoded string in the branch for
+    "hit the cap and did not resume", which is reached for four different
+    reasons. A caller read it exactly as written and went looking for a setting
+    that was not set — the real cause was a looping generation, and the budget
+    was never 0. A diagnostic that names the wrong cause is worse than one that
+    says nothing, because it is acted on.
+    """
+    try:
+        _state()["no_resume"] = str(reason)
+    except Exception:
+        pass
+
+
 def _page_override(name: str, default):
     """A per-request value for the page this THREAD is reading, else the default.
 
@@ -975,6 +994,7 @@ def _continue_reply(oc, result, item, client, model_name, cap, temperature,
         base_messages = oc._build_messages(oc.scale_to_fit(item.image), prompt)
     except Exception as exc:
         print(f"[surya] cannot resume a cut-off page: {exc}", file=sys.stderr, flush=True)
+        _no_resume(f"the reply could not be resumed ({exc})")
         return result, 0, True
 
     # A reply that is already repeating itself did not run out of room -- it ran
@@ -990,6 +1010,10 @@ def _continue_reply(oc, result, item, client, model_name, cap, temperature,
                   f"itself — it stopped at the {cap}-token cap because it is "
                   f"looping, not because the page is long",
                   file=sys.stderr, flush=True)
+            _no_resume("the generation was already repeating itself, so it "
+                       "stopped at the cap because it is LOOPING, not because "
+                       "the page is long; resuming would only extend the loop "
+                       "and surya re-runs the page anyway")
             return result, 0, False
     except Exception:
         pass
@@ -1254,6 +1278,11 @@ class SuryaWorker:
             try:
                 budget = int(_page_override("max_continuations",
                                             _CONTINUE_BUDGET[0]) or 0)
+                _state()["budget"] = budget
+                if not budget:
+                    _no_resume("resumption is off for this page "
+                               "(max_continuations = 0, from the request or "
+                               "ocr.surya_max_continuations)")
                 item = args[0] if args else kwargs.get("item")
                 eff_cap = int(getattr(item, "max_tokens", 0) or 0) or int(
                     kwargs.get("max_tokens_default") or 0)
@@ -1466,6 +1495,12 @@ class SuryaWorker:
         incomplete = bool(_state().get("still_truncated"))
         if tries > 1 or capped or resumed:
             info = {"calls": tries or 1, "tokens": toks, "cap": cap}
+            # The budget actually in force for this page. Printing the cap but
+            # not this is why a caller could not tell a request that landed from
+            # one that was ignored.
+            _budget = _state().get("budget")
+            if _budget is not None:
+                info["max_continuations"] = int(_budget)
             if resumed:
                 info["resumed"] = resumed
                 info["why"] = (f"the reply ran past the {cap}-token cap and was "
@@ -1483,11 +1518,14 @@ class SuryaWorker:
             elif capped and not resumed:
                 info["truncated"] = True
                 info["complete"] = False
+                why = _state().get("no_resume")
                 info["why"] = (
-                    f"the reply stopped at the {cap}-token cap and resumption is "
-                    f"off (ocr.surya_max_continuations = 0), so this page is "
-                    f"INCOMPLETE. A cut-off reply, NOT a page the model could "
-                    f"not read.")
+                    f"the reply stopped at the {cap}-token cap and was not "
+                    f"resumed: {why}. So this page is INCOMPLETE."
+                    if why else
+                    f"the reply stopped at the {cap}-token cap and was not "
+                    f"resumed. So this page is INCOMPLETE. A cut-off reply, NOT "
+                    f"a page the model could not read.")
             elif not resumed:
                 info["why"] = ("surya re-ran this page because the generation "
                                "repeated itself; each re-run is a full "

tests/test_pool_growth_adopts.py

+"""Growing a pool must try adoption before renting, like every other path.
+
+Live, 2026-10-13 23:10. A run started with two of this pool's pods already
+running (ours, orphaned by a restart). The cold path adopted one — correctly —
+and then the pool grew:
+
+  23:10:12 [runpod] adopting pod sncrmkqdnd7c57's bearer token
+  23:10:12 [runpod] reusing the pod already running … instead of renting another
+  23:10:12 'datalab-to/surya-ocr-2': growing to meet 1 pod(s) of demand
+  23:10:13 provisioning pod … (a 30-candidate capacity walk, then an RTX A6000)
+
+…renting a card beside a second, healthy, idle pod it already owned. Adoption
+was attempted in three places — `ensure_ready`, `acquire` with nothing healthy,
+and the warm floor — and the growth path was not one of them, so growth always
+rented. The second pod billed for nothing until the reaper's grace expired.
+"""
+import pathlib
+import sys
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from codai.api import runpod_worker as rw                          # noqa: E402
+
+
+class _Pool:
+    """Just the growth entry point, over recorded decisions."""
+
+    model_key = "m"
+    _grow_in_background = rw.RunpodPodPool._grow_in_background
+
+    def __init__(self, adopts):
+        import threading
+        self._cv = threading.Condition()
+        self._booting = 1
+        self._adopts = adopts
+        self.calls = []
+
+    def _adopt_shared_pod(self):
+        self.calls.append("adopt")
+        return self._adopts
+
+    def _provision_one(self):
+        self.calls.append("rent")
+
+
+def _grow(adopts):
+    import time
+    pool = _Pool(adopts)
+    pool._grow_in_background()
+    for _ in range(200):                     # it runs on its own thread
+        if pool.calls and (adopts or "rent" in pool.calls):
+            break
+        time.sleep(0.01)
+    time.sleep(0.05)
+    return pool
+
+
+def test_growth_adopts_a_running_pod_instead_of_renting():
+    pool = _grow(True)
+    assert pool.calls == ["adopt"], "a card was rented beside a pod we own"
+
+
+def test_growth_still_rents_when_there_is_nothing_to_adopt():
+    pool = _grow(False)
+    assert pool.calls == ["adopt", "rent"]
+
+
+def test_the_booting_count_is_released_either_way():
+    # Left standing, it would block every later growth: _should_grow refuses
+    # while `_booting >= scale_up_max_parallel_boots`.
+    assert _grow(True)._booting == 0
+    assert _grow(False)._booting == 0
+
+
+def test_every_provisioning_path_tries_adoption_first():
+    """Pinned against the source: a fourth path added later must not forget.
+
+    `seed_region` is deliberately excluded — it exists to fill a region's volume
+    by booting a pod THERE, and adopting a pod elsewhere would not do that.
+    """
+    src = (ROOT / "codai/api/runpod_worker.py").read_text()
+    calls = src.count("self._provision_one()")
+    guarded = src.count("if not self._adopt_shared_pod():")
+    adopt_first = src.count("if self._adopt_shared_pod():")
+    # 5 call sites: ensure_ready, acquire, growth, warm floor, seed_region.
+    assert calls == 5
+    assert guarded + adopt_first == 4

tests/test_runpod_global_volume.py

-"""A global volume must not pin the pod to a region — that is the whole point.
-
-RunPod's Global Volumes (beta, Sept 2026) are region-independent storage backed
-by object storage. A NETWORK volume lives in one data centre and a pod elsewhere
-cannot attach it, which is why coderai forces the pod into the volume's region.
-Applying that same pinning to a global volume would throw away the only reason
-to use one: store the weights once, rent a GPU in whichever region has one.
-
-So the two are resolved separately, and these tests hold them apart.
-"""
-import pathlib
-
-from codai.api import runpod_worker as rw
-
-ROOT = pathlib.Path(rw.__file__).resolve().parents[2]
-
-
-class _Acct:
-    def __init__(self, net="", glob="", dc="", mount="/workspace"):
-        self.network_volume_id = net
-        self.global_volume_id = glob
-        self.data_center = dc
-        self.volume_mount_path = mount
-
-
-def test_the_account_setting_applies_to_every_model():
-    mcfg = rw.parse_model_runpod({})
-    assert rw.global_volume_for(mcfg, _Acct(glob="gv-123")) == "gv-123"
-
-
-def test_a_model_can_override_the_account_volume():
-    mcfg = rw.parse_model_runpod({"global_volume_id": "gv-model"})
-    assert rw.global_volume_for(mcfg, _Acct(glob="gv-account")) == "gv-model"
-
-
-def test_no_volume_configured_is_empty_not_an_error():
-    assert rw.global_volume_for(rw.parse_model_runpod({}), _Acct()) == ""
-
-
-def test_a_global_volume_does_not_pin_the_data_centre():
-    """The network volume's region wins over everything; a global volume must
-    leave the choice alone so the scaler can rank cards across all regions."""
-    pool = rw.RunpodPodPool.__new__(rw.RunpodPodPool)
-    pool.account = _Acct(glob="gv-123")
-    pool.mcfg = rw.parse_model_runpod({"global_volume_id": "gv-123"})
-    pool._volume_dc = None
-    assert pool._data_center() == ""      # free to go anywhere
-    pool.mcfg.data_center = "EU-RO-1"
-    assert pool._data_center() == "EU-RO-1"   # only because the model asked
-
-
-def test_only_the_network_volume_triggers_the_region_lookup():
-    src = (ROOT / "codai/api/runpod_worker.py").read_text()
-    blk = src.split("vol_id, vol_mount = volume_for")[1].split("self._tls = False")[0]
-    # The region lookup sits under the network-volume branch, and the global
-    # branch that follows must not contain it.
-    net, glob = blk.split("global_vol = global_volume_for")
-    assert "_data_center_for_volume" in net
-    assert "_data_center_for_volume" not in glob
-
-
-def test_the_two_volumes_are_not_attached_together_by_accident():
-    """With both attached RunPod moves the global one to /workspace-global, so
-    pointing the pod env at /workspace would aim it at the wrong volume."""
-    src = (ROOT / "codai/api/runpod_worker.py").read_text()
-    blk = src.split("global_vol = global_volume_for")[1][:600]
-    assert "if global_vol and not vol_id:" in blk
-    assert "/workspace-global" in src
-
-
-def test_the_client_sends_the_field_on_both_api_shapes():
-    """create_pod has a GraphQL body and a REST body; a volume wired into only
-    one of them attaches on some code paths and silently not on others."""
-    src = (ROOT / "codai/api/runpod_client.py").read_text()
-    assert src.count("GLOBAL_VOLUME_FIELD] = global_volume_id") == 2
-    assert "global_volume_id: str = \"\"" in src
-
-
-def test_the_field_name_is_overridable_because_it_is_undocumented():
-    """RunPod documents global volumes as console-only and the v1 validator
-    accepts unknown body keys silently, so a wrong name cannot be detected by
-    probing — it yields a pod with no volume that re-downloads its weights."""
-    from codai.api import runpod_client
-    assert runpod_client.GLOBAL_VOLUME_FIELD
-    src = (ROOT / "codai/api/runpod_client.py").read_text()
-    assert "CODERAI_RUNPOD_GLOBAL_VOLUME_FIELD" in src
-    assert "VERIFY" in src        # says out loud that it is unconfirmed
-
-
-def test_the_config_documents_that_it_cannot_be_created_by_api():
-    cfgsrc = (ROOT / "codai/config.py").read_text()
-    blk = cfgsrc.split("global_volume_id")[0][-1400:]
-    assert "console" in blk.lower()
-    assert "STANDARD" in blk      # the enum the API actually allows
-
-
-# ------------------------------------------- "anywhere in the EU", said safely
-
-class _AcctDC(_Acct):
-    def __init__(self, dcs=None, dc="", **kw):
-        super().__init__(**kw)
-        self.data_centers = dcs or []
-        self.data_center = dc
-
-
-def _pool(mcfg_block=None, acct=None):
-    pool = rw.RunpodPodPool.__new__(rw.RunpodPodPool)
-    pool.account = acct or _AcctDC()
-    pool.mcfg = rw.parse_model_runpod(mcfg_block or {})
-    pool._volume_dc = None
-    return pool
-
-
-def test_a_region_list_is_parsed_from_a_string_or_a_list():
-    for value in ("EU-RO-1,EU-FR-1", "EU-RO-1 EU-FR-1", ["eu-ro-1", "EU-FR-1"]):
-        assert rw._parse_dc_list(value) == ["EU-RO-1", "EU-FR-1"], value
-    assert rw._parse_dc_list("") == []
-    assert rw._parse_dc_list(None) == []
-
-
-def test_the_order_is_the_fallback_order_and_is_preserved():
-    """Not sorted: the first entry is the preferred region and the scaler only
-    moves on when it cannot get a card there."""
-    assert rw._parse_dc_list("EU-RO-1,EU-FR-1,EUR-NO-1")[0] == "EU-RO-1"
-    assert rw._parse_dc_list("EU-FR-1,EU-RO-1")[0] == "EU-FR-1"
-
-
-def test_duplicates_collapse():
-    assert rw._parse_dc_list("EU-RO-1, eu-ro-1 ,EU-FR-1") == ["EU-RO-1", "EU-FR-1"]
-
-
-def test_the_model_list_beats_the_account_list():
-    pool = _pool({"data_centers": "EU-FR-1"}, _AcctDC(dcs=["EU-RO-1"]))
-    assert pool._data_centers() == ["EU-FR-1"]
-
-
-def test_a_single_data_center_still_works_exactly_as_before():
-    pool = _pool({"data_center": "EU-SE-1"})
-    assert pool._data_centers() == ["EU-SE-1"]
-    assert pool._data_center() == "EU-SE-1"
-
-
-def test_nothing_configured_still_means_runpod_chooses():
-    pool = _pool()
-    assert pool._data_centers() == [""]
-    assert pool._data_center() == ""
-
-
-def test_a_network_volume_overrides_every_region_setting():
-    """It exists in ONE region and no pod elsewhere can attach it, so a
-    multi-region list must not send a pod where the attach would fail."""
-    pool = _pool({"data_centers": "EU-FR-1,EU-RO-1"}, _AcctDC(dcs=["EUR-NO-1"]))
-    pool._volume_dc = "EU-NL-1"
-    assert pool._data_centers() == ["EU-NL-1"]
-
-
-def test_candidates_are_expanded_across_regions_for_the_capacity_fallback():
-    """The existing retry walks `ranked` on a capacity miss; pairing each card
-    with each region makes it walk regions too, with no new control flow."""
-    src = (ROOT / "codai/api/runpod_worker.py").read_text()
-    blk = src.split("ranked = _rank_gpus(client, self.mcfg, self.account)")[1][:700]
-    assert 'dict(sel, _dc=dc) for sel in ranked for dc in dcs' in blk
-    assert "if len(dcs) > 1:" in blk
-
-
-def test_the_pod_is_created_in_and_records_the_region_it_got():
-    src = (ROOT / "codai/api/runpod_worker.py").read_text()
-    assert src.count('sel.get("_dc") or self._data_center()') >= 3   # create, log, handle

tests/test_runpod_scale_out.py

@@ -53,6 +53,11 @@ def _pool(max_pods=64, threshold=16, parallel=4, pods=(), booting=0, waiting=0):
     cfg.scale_up_inflight_per_pod = threshold
     cfg.scale_up_max_parallel_boots = parallel
     pool.mcfg = cfg
+    # Growth tries adoption before renting (a pod already running for this pool
+    # is cheaper than a new card). These tests are about the counting and the
+    # scale-up, so there is nothing to adopt unless a test says otherwise —
+    # left real, it would read this machine's own pod registry.
+    pool._adopt_shared_pod = lambda: False
     return pool
 
 
@@ -252,3 +257,22 @@ def test_growth_is_announced_with_the_numbers():
     that does not need to — which is how ten minutes of this went unnoticed."""
     src = (ROOT / "codai/api/runpod_worker.py").read_text()
     assert "growing to meet" in src and "booting" in src
+
+
+def test_growth_prefers_a_pod_already_running_over_renting():
+    """Live, 2026-10-13: a run adopted one surviving pod, then grew — and rented
+    a card beside a SECOND healthy pod it already owned, which billed for
+    nothing until the reaper's grace expired. Only the cold paths adopted."""
+    pool = _pool(pods=[_pod("a", inflight=16)], waiting=48)
+    provisioned, adopted = [], []
+    pool._provision_one = lambda: provisioned.append(1)
+    pool._adopt_shared_pod = lambda: (adopted.append(1), True)[1]
+    pool._pick = lambda healthy, affinity: healthy[0]
+    pool._remember_affinity = lambda a, p: None
+    pool.acquire(timeout=5)
+    for _ in range(200):
+        if pool._booting == 0 and adopted:
+            break
+        __import__("time").sleep(0.01)
+    assert adopted == [1]
+    assert provisioned == [], "rented a card beside a pod we already had"

tests/test_surya_thread_boundary.py

@@ -169,3 +169,58 @@ def test_the_batch_hook_is_reported_when_surya_has_no_such_entry_point(monkeypat
     w = _worker(mod, token_trace=True, max_tokens=3072)
     w._install_trace_hook()
     assert "no chat_completions_batch to hook" in capsys.readouterr().err
+
+
+# --- the report must name the real reason, not assert a config value --------
+
+def test_a_looping_page_is_not_reported_as_resumption_being_off():
+    """The message was a hardcoded string in a branch reached four ways.
+
+    A caller read "resumption is off (ocr.surya_max_continuations = 0)" exactly
+    as written and went looking for a setting that was never set; the real cause
+    was a generation that had already started repeating itself, with a budget of
+    4 in force. A diagnostic that names the wrong cause is worse than silence,
+    because it is acted on.
+    """
+    mod = _worker_module()
+    mod._trace_begin()
+    mod._state()["budget"] = 4
+    mod._no_resume("the generation was already repeating itself")
+    assert mod._state()["no_resume"] == "the generation was already repeating itself"
+    src = (ROOT / "codai/ocr/workers/ocr_worker.py").read_text()
+    # The old text claimed a value it never read. It must not come back.
+    assert "off (ocr.surya_max_continuations = 0)" not in src
+    assert 'why = _state().get("no_resume")' in src
+
+
+def test_the_budget_in_force_is_published():
+    """So a caller can tell a request that landed from one that was ignored."""
+    src = (ROOT / "codai/ocr/workers/ocr_worker.py").read_text()
+    assert 'info["max_continuations"] = int(_budget)' in src
+
+
+def test_a_new_page_does_not_inherit_the_previous_reason():
+    mod = _worker_module()
+    mod._trace_begin()
+    mod._no_resume("looping")
+    mod._trace_begin()
+    assert "no_resume" not in mod._state()
+    assert "budget" not in mod._state()
+
+
+def test_zero_budget_still_says_resumption_is_off(monkeypatch):
+    """That message was right in ONE of the four cases; it must survive there."""
+    mod = _worker_module()
+    oc = _fake_surya(monkeypatch, mod)
+    def _at_the_cap(*a, **k):
+        r = _Result()
+        r.token_count = 3072        # the reply filled its budget
+        return r
+
+    oc._generate_one = _at_the_cap
+    w = _worker(mod, token_trace=True, max_tokens=3072, max_continuations=0)
+    w._install_trace_hook()
+    mod._trace_begin()
+    mod._trace_opts({"max_tokens": 3072, "max_continuations": 0})
+    oc._generate_one(_Item(3072), max_tokens_default=3072)
+    assert "max_continuations = 0" in (mod._state().get("no_resume") or "")