runpod+ocr: fill a warm floor in waves, and put the OCR pipeline on the pod image (v0.3.69)

THE WARM FLOOR. A declared floor of 8 took ~40 minutes to exist. Measured on the
Digesta orchestrator, one pod per scaler pass at ~5 minutes each:

    15:15:27  warm floor 8, 2 pod(s) healthy — renting one
    15:21:12  warm floor 8, 3 pod(s) healthy — renting one
    15:32:41  warm floor 8, 4 pod(s) healthy — renting one

Two causes. First, `maintain()` rented exactly one pod per pass, gated on
`not self._booting`. A floor is a DECLARATION, not a prediction -- min_pods,
keep_warm or a client lease already said how many should be warm -- so the
deficit is filled at once, bounded by the model's existing
`scale_up_max_parallel_boots` (no new knob; it already means "how many pods may
be booting at once"). Second, `_provision_one` is "Blocking (minutes)" and this
called it INLINE from the scaler loop, so one warm boot stalled that loop --
including the registry heartbeat that lives in it and runs every 60s. A stalled
heartbeat is exactly what makes a sibling treat our booting pod as abandoned,
which 0.3.68 made load-bearing. Warm boots now run on their own threads.

Adoption loops too (a restart orphans the whole fleet; eleven pods at one per
tick took six minutes), but BOUNDED two ways: `_adopt_shared_pod()` returning
True is a report, not a guarantee -- it can succeed without the pool gaining a
healthy pod -- and the first `while ... and adopt()` form span forever the first
time that happened, taking the scaler thread and the heartbeat with it. At most
`deficit` attempts, and stop when an adoption does not raise the count. A pass
that adopted rents nothing: renting beside a fresh adoption is how one job pays
for two cards.

THE IMAGE. coderai-vllm-ocr existed to pin vLLM 0.20.1, the version surya
spawns, because a page of crisp Italian came back from 0.27.1 with one U+FFFD
per accented letter. That was the wrong conclusion from a real measurement:
surya-ocr-2's tokenizer is BYTE-level, and the fleet has been reading accented
Italian on 0.27.1 ever since. Two images serving the same weights on two vLLMs
is an invisible difference, and it cost three rounds of blaming the pod image.
So the pins are now IDENTICAL and the check READS the fleet's pin from disk
rather than restating a version -- bumping profiles/vllm cannot silently leave
this image behind.

What distinguishes the image instead is the OCR PIPELINE on the pod: the surya
client venv beside the model, plus pypdfium2 in the main venv, so a page is
rasterised and read on the card rather than crossing the network as a ~1 MB
base64 PNG (and four more times over for a page resumed four times).

CPU TORCH, and it is the whole image-size story. Measured in a resolved surya
venv: nvidia 3196 MB + torch 1175 MB + triton 896 MB = ~5.3 GB of CUDA stack,
for a venv whose only local torch workload is surya's small line detector. On
the pod the GPU is already committed to serving the model, so giving the
detector the card would take VRAM and SMs from the thing that matters; on a
GPU-less orchestrator it can never execute at all (cuda available: False today
while its surya venv carries cu130). ~280 MB instead of ~5.3 GB, and the venv
check asserts torch.version.cuda is None so a resolve cannot quietly add 5 GB
back.

THE DETECTOR rides that same CPU torch, which is why it costs ~30 MB rather
than a third torch. detect.py tries ultralytics in-process first (every existing
install is unchanged) and falls back to codai/ocr/workers/detect_worker.py in
whichever venv has it -- surya's first. A detector that cannot be reached looks
like a MISSING detector (the ordinary 503), never a broken install. No weights
are baked: detect_model_path defaults to "" and a released image must contain no
models, configs, keys or secrets.

Tests: 17 warm-floor (+4 for the spin), 14 detector. Four older tests encoded
the decisions this reverses -- two asserted the 0.20.1 pin (replaced by one
requiring the two pin files to be identical), one asserted the warm floor "must
still be single-flight", one counted a source string globally and could not tell
intent-held-spelling-changed from a regression (now per path). Suite at the 25
pre-existing failures.

NOT BUILT and NOT USABLE YET: nothing has resolved surya-ocr + pillow<11 +
torch+cpu + ultralytics together, and the orchestrator cannot drive a pod-side
pipeline until the dual mode lands.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.68"
+__version__ = "0.3.69"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/runpod_worker.py

@@ -172,6 +172,13 @@ class RunpodModelConfig:
     # start billing together — so this is the knob that says how fast an
     # operator is willing to spend to absorb a burst. Growth never exceeds
     # max_pods, and the cost caps still refuse each pod individually.
+    #
+    # Also bounds filling a WARM FLOOR (min_pods, keep_warm, or a client
+    # lease). That used to rent one pod per scaler pass and block the
+    # thread until it was ready, so a declared floor of 8 took ~40 minutes
+    # to exist. A floor is a declaration rather than a prediction, so the
+    # deficit is filled at once -- but still no more of them booting
+    # together than this, because it is the same question about money.
     scale_up_max_parallel_boots: int = 4
     # Scaling DOWN again when the pressure that justified the pods is gone.
     # Idle teardown alone is not enough: it only fires on a pod with NO work and
@@ -3803,7 +3810,7 @@ class RunpodPodPool:
             healthy_n = len([p for p in self.pods if p.healthy])
             if self._release_pending and healthy_n <= keep_min:
                 self._release_pending = False          # satisfied
-        if healthy_n < keep_min and not self._closed and not self._booting:
+        if healthy_n < keep_min and not self._closed:
             # A pod that is ALREADY RUNNING for this pool beats waiting for one
             # that is booting, and beats renting a second. Tried before the
             # sibling check on purpose: after a restart the pods this deployment
@@ -3811,33 +3818,88 @@ class RunpodPodPool:
             # deferring to a sibling's boot marker left them unowned -- the pool
             # empty, the account full. Adoption is the recovery path, so it must
             # not sit behind the thing it recovers from.
+            #
+            # In a LOOP: one adoption fills one slot, and a restart orphans the
+            # whole fleet. Adopting eleven pods one scaler tick at a time took
+            # six minutes where it should take one pass.
+            # Bounded TWO ways, because `_adopt_shared_pod` returning True is a
+            # report, not a guarantee: it can succeed without the pool gaining a
+            # healthy pod (a pod registered but not yet probed, a race with the
+            # reaper). `while ... and adopt()` spun forever the first time that
+            # happened and took the scaler thread -- and with it the registry
+            # heartbeat -- down with it.
+            adopted = False
             try:
-                if self._adopt_shared_pod():
-                    return
+                for _ in range(max(0, keep_min - healthy_n)):
+                    if healthy_n >= keep_min or not self._adopt_shared_pod():
+                        break
+                    adopted = True
+                    with self._cv:
+                        got = len([p for p in self.pods if p.healthy])
+                    if got <= healthy_n:
+                        print(f"[runpod] {self.model_key!r}: adoption reported "
+                              f"success without adding a pod — not asking again "
+                              f"this pass", flush=True)
+                        break
+                    healthy_n = got
             except Exception as exc:
                 print(f"[runpod] maintain: adoption failed: {exc}", flush=True)
+            # Adopted something? Then rent NOTHING this pass, even if the floor
+            # is still short. The next pass is 15s away and will count what we
+            # just took; renting beside a fresh adoption is how one job ends up
+            # paying for two cards -- and a pool whose adoption reports success
+            # without adding a pod must not be charged for the discrepancy.
+            if healthy_n >= keep_min or adopted:
+                return
             # One already on the way (from here, from ensure_ready, or from a
             # sibling engine sharing this pool) is one pod, not none: topping up
             # against a booting pod is how a warm floor of 1 became two.
             if sibling_is_provisioning(str(self.model_key)):
                 return
+            # Fill the WHOLE deficit, not one pod per tick. A pod needs ~5
+            # minutes to have weights resident, and this used to rent exactly one
+            # per pass and then BLOCK the scaler thread until it was ready, so a
+            # client lease of 8 took ~40 minutes to become real -- measured on
+            # 2026-10-14, 15:15 to 15:42 for four of them. Bounded by the model's
+            # own `scale_up_max_parallel_boots`, which already means "how many
+            # pods may be booting at once", and by max_pods.
             with self._cv:
-                if self._booting:
+                in_hand = len([p for p in self.pods if p.healthy]) + self._booting
+                parallel = max(1, int(getattr(self.mcfg,
+                                              "scale_up_max_parallel_boots", 4) or 4))
+                want = min(keep_min - in_hand,
+                           max(0, parallel - self._booting),
+                           max(0, int(self.mcfg.max_pods or 0) - in_hand))
+                if want <= 0:
                     return
-                self._booting += 1
+                self._booting += want
             # Say so before spending. A warm floor silently failing to be
             # re-established is indistinguishable, from outside, from a warm
             # floor that is fine — which is how a pod terminated out of band
             # went unnoticed until the next request paid a cold boot.
             print(f"[runpod] {self.model_key!r}: warm floor {keep_min}, "
-                  f"{healthy_n} pod(s) healthy — renting one", flush=True)
-            try:
-                self._provision_one()
-            except Exception as exc:
-                print(f"[runpod] maintain: warm provision failed: {exc}", flush=True)
-            finally:
-                with self._cv:
-                    self._booting = max(0, self._booting - 1)
+                  f"{healthy_n} pod(s) healthy — renting {want} "
+                  f"(up to {parallel} may boot at once)", flush=True)
+            for _ in range(want):
+                # OFF the scaler thread. _provision_one blocks for minutes, and
+                # this ran inline: one warm boot stalled the registry heartbeat
+                # (same loop, every 60s) and every other pool's maintain() for
+                # the whole boot. A stalled heartbeat is what makes a sibling
+                # treat our booting pod as abandoned.
+                threading.Thread(target=self._warm_provision_one, daemon=True,
+                                 name=f"runpod-warm-{self.model_key}").start()
+
+    def _warm_provision_one(self):
+        """One warm-floor boot, on its own thread. Always gives `_booting` back:
+        a leaked increment is a pool that can never grow again."""
+        try:
+            self._provision_one()
+        except Exception as exc:
+            print(f"[runpod] maintain: warm provision failed: {exc}", flush=True)
+        finally:
+            with self._cv:
+                self._booting = max(0, self._booting - 1)
+                self._cv.notify_all()
 
     def close(self):
         self._closed = True

codai/ocr/detect.py

@@ -29,6 +29,8 @@ Configurable via ``ocr.detect_mode``:
 The detector model is loaded once per weights path and reused across requests.
 """
 
+import json
+import os
 import threading
 from typing import Dict, List, Optional
 
@@ -69,27 +71,109 @@ _detector_cache: Dict[str, "YoloDetector"] = {}
 _cache_lock = threading.Lock()
 
 
+class _DetectWorker:
+    """The YOLO detector, driven in whichever venv actually has ultralytics.
+
+    One process per install, shared by every weights path (the worker caches
+    models by path), because loading YOLO twice costs memory for nothing.
+
+    Deliberately NOT a hard dependency: `get()` returns None when no such venv
+    exists, and the caller then raises the ordinary "ultralytics is not
+    installed" error. A detector that cannot be reached must look like a missing
+    detector, not like a broken install.
+    """
+
+    _inst = None
+    _lock = threading.Lock()
+
+    #: Where a baked venv carrying ultralytics may be. `surya` first: that is
+    #: where the pod image puts it, sharing the CPU torch already there.
+    _VENVS = ("/opt/coderai/venvs/surya", "/opt/coderai/venvs/detect",
+              "/opt/coderai/venvs/ocr-paddle")
+
+    def __init__(self, python: str):
+        import subprocess
+        worker = os.path.join(os.path.dirname(__file__), "workers",
+                              "detect_worker.py")
+        self._proc = subprocess.Popen([python, "-I", worker],
+                                      stdin=subprocess.PIPE,
+                                      stdout=subprocess.PIPE, stderr=None,
+                                      bufsize=0)
+        hello = self._reply()
+        if not hello.get("ready"):
+            self.kill()
+            raise OcrError(hello.get("error") or "detector worker not ready",
+                           status=503)
+        self._io = threading.Lock()
+
+    @classmethod
+    def get(cls):
+        """The shared worker, or None when no venv carrying ultralytics exists."""
+        with cls._lock:
+            if cls._inst is not None:
+                return cls._inst
+            for vdir in cls._VENVS:
+                py = os.path.join(vdir, "bin", "python")
+                if not os.path.isfile(py):
+                    continue
+                try:
+                    cls._inst = cls(py)
+                except Exception as exc:
+                    print(f"[ocr] detector venv {vdir} would not start ({exc})",
+                          flush=True)
+                    continue
+                print(f"[ocr] stamp/signature detector runs in {vdir}",
+                      flush=True)
+                return cls._inst
+            return None
+
+    def _reply(self) -> dict:
+        line = self._proc.stdout.readline()
+        if not line:
+            raise OcrError("detector worker exited", status=500)
+        return json.loads(line.decode("utf-8"))
+
+    def detect(self, image, model_path: str, conf: float) -> List[dict]:
+        import numpy as np
+        rgb = image.convert("RGB")
+        raw = np.array(rgb).tobytes()
+        hdr = {"cmd": "detect", "model": model_path, "conf": float(conf),
+               "w": rgb.width, "h": rgb.height, "nbytes": len(raw)}
+        # One page at a time down one pipe: the frame is header-then-bytes, and
+        # two callers interleaving would desynchronise it.
+        with self._io:
+            self._proc.stdin.write(json.dumps(hdr).encode("utf-8") + b"\n")
+            self._proc.stdin.write(raw)
+            self._proc.stdin.flush()
+            r = self._reply()
+        if not r.get("ok"):
+            raise OcrError(r.get("error") or "YOLO detection failed",
+                           status=int(r.get("status", 500)))
+        return list(r.get("boxes") or [])
+
+    def kill(self):
+        try:
+            self._proc.kill()
+        except Exception:
+            pass
+
+
 class YoloDetector:
     def __init__(self, model_path: str, conf: float):
         self.model_path = model_path
         self.conf = conf
         self._model = None
+        #: Set instead of `_model` when ultralytics is not importable in this
+        #: process but a venv carrying it is available (see _DetectWorker).
+        self._remote = None
         self._load_lock = threading.Lock()
 
     def ensure_loaded(self):
-        if self._model is not None:
+        if self._model is not None or self._remote is not None:
             return
         with self._load_lock:
-            if self._model is not None:
+            if self._model is not None or self._remote is not None:
                 return
-            try:
-                from ultralytics import YOLO
-            except Exception as e:
-                raise OcrError(
-                    "ultralytics is not installed (needed for detector mode). Install: "
-                    f"`pip install ultralytics`. Import error: {e}",
-                    status=503,
-                )
             import os
             if not self.model_path or not os.path.isfile(self.model_path):
                 raise OcrError(
@@ -97,6 +181,22 @@ class YoloDetector:
                     "to a YOLO .pt file trained on signatures/stamps.",
                     status=400,
                 )
+            try:
+                from ultralytics import YOLO
+            except Exception as in_process_error:
+                # Not importable HERE is not the same as not installed. On a
+                # light-core pod image ultralytics lives beside surya, sharing
+                # the CPU torch already in that venv -- which is what makes the
+                # detector cost ~30 MB instead of a third torch. Reach it.
+                self._remote = _DetectWorker.get()
+                if self._remote is None:
+                    raise OcrError(
+                        "ultralytics is not installed (needed for detector mode) and no "
+                        "venv carrying it was found. Install: `pip install ultralytics`, "
+                        f"or bake it beside surya. Import error: {in_process_error}",
+                        status=503,
+                    )
+                return
             try:
                 self._model = YOLO(self.model_path)
             except Exception as e:
@@ -104,6 +204,8 @@ class YoloDetector:
 
     def detect(self, image) -> List[dict]:
         self.ensure_loaded()
+        if self._remote is not None:
+            return self._remote.detect(image, self.model_path, self.conf)
         import numpy as np
 
         arr = np.array(image.convert("RGB"))

codai/ocr/workers/detect_worker.py

+#!/usr/bin/env python3
+"""Run the stamp/signature YOLO detector in a venv that has a torch.
+
+``codai/ocr/detect.py`` imports ultralytics IN-PROCESS, which works on any
+install whose main venv carries torch. Two places it does not:
+
+* the ``vllm-ocr`` pod image, whose core is LIGHT on purpose —
+  ``Dockerfile.capability`` fails the build if torch reaches the main venv, and
+  a third torch there would cost ~2.5 GB compressed for a workload that does
+  not even want the card (the GPU is serving the model);
+* any install that would rather not have torch in its main venv at all.
+
+So ultralytics is installed beside surya, sharing the CPU torch already in that
+venv (``profiles/vllm-ocr.venv-surya.txt``), and reached from here. That sharing
+is the whole reason the detector costs ~30 MB rather than another torch.
+
+Run as a SCRIPT with the OTHER venv's python, never imported: importing
+``codai.ocr`` costs 377 MB and 1.5 s via litellm, and the point is to stay small.
+
+Protocol, newline-delimited JSON on stdin/stdout:
+
+    -> {"cmd": "detect", "model": "/path/to.pt", "conf": 0.35,
+        "w": 850, "h": 1100, "nbytes": 2805000}   + <raw RGB bytes>
+    <- {"ok": true, "boxes": [{"label": "stamp", "conf": 0.91,
+                               "bbox": [x0, y0, x1, y1]}, ...]}
+
+    -> {"cmd": "shutdown"}
+    <- {"ok": true}
+
+Raw RGB rather than PNG: the parent already holds the decoded page, and a page
+on its way to a detector must not pay a compress/decompress round trip.
+
+A failure answers ``{"ok": false, "error": ..., "status": ...}`` and leaves the
+process usable: one bad page must not cost the detector its loaded weights.
+"""
+
+import json
+import sys
+
+
+def _reply(obj):
+    sys.stdout.buffer.write(json.dumps(obj).encode("utf-8") + b"\n")
+    sys.stdout.buffer.flush()
+
+
+def _read_exactly(n: int) -> bytes:
+    buf = bytearray()
+    while len(buf) < n:
+        chunk = sys.stdin.buffer.read(n - len(buf))
+        if not chunk:
+            raise EOFError(f"stdin closed after {len(buf)} of {n} bytes")
+        buf.extend(chunk)
+    return bytes(buf)
+
+
+def main() -> int:
+    try:
+        from ultralytics import YOLO
+        import numpy as np
+    except Exception as e:
+        _reply({"ready": False, "error": f"detector deps missing: {e}"})
+        return 1
+    _reply({"ready": True})
+
+    models = {}          # weights path -> loaded YOLO
+
+    while True:
+        line = sys.stdin.buffer.readline()
+        if not line:
+            break
+        try:
+            msg = json.loads(line.decode("utf-8"))
+        except Exception as e:
+            _reply({"ok": False, "error": f"bad request frame: {e}",
+                    "status": 500})
+            continue
+        cmd = msg.get("cmd")
+
+        if cmd == "shutdown":
+            _reply({"ok": True})
+            return 0
+
+        if cmd != "detect":
+            _reply({"ok": False, "error": f"unknown cmd {cmd!r}", "status": 500})
+            continue
+
+        try:
+            raw = _read_exactly(int(msg.get("nbytes") or 0))
+        except Exception as e:
+            # The frame is lost; the parent must replace us rather than guess
+            # where the next header starts.
+            _reply({"ok": False, "error": str(e), "status": 500})
+            return 1
+
+        path = str(msg.get("model") or "")
+        try:
+            model = models.get(path)
+            if model is None:
+                model = models[path] = YOLO(path)
+        except Exception as e:
+            _reply({"ok": False, "error": f"failed to load YOLO detector: {e}",
+                    "status": 500})
+            continue
+
+        try:
+            arr = np.frombuffer(raw, dtype=np.uint8).reshape(
+                int(msg["h"]), int(msg["w"]), 3)
+            results = model.predict(arr, conf=float(msg.get("conf") or 0.35),
+                                    verbose=False)
+        except Exception as e:
+            _reply({"ok": False, "error": f"YOLO detection failed: {e}",
+                    "status": 500})
+            continue
+
+        boxes = []
+        for res in results:
+            names = getattr(res, "names", {}) or {}
+            bs = getattr(res, "boxes", None)
+            if bs is None:
+                continue
+            for b in bs:
+                try:
+                    cls = int(b.cls[0])
+                    xy = [float(v) for v in b.xyxy[0]]
+                    boxes.append({"label": str(names.get(cls, cls)),
+                                  "conf": float(b.conf[0]), "bbox": xy})
+                except Exception:
+                    continue
+        _reply({"ok": True, "boxes": boxes})
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

packaging/runpod/profiles/ocr.venv-surya.txt

@@ -9,5 +9,16 @@
 # with surya_serve="vllm" the heavy model runs in vLLM on the GPU pod and this
 # side only renders, batches and post-processes. So the venv goes on the
 # ORCHESTRATOR image, never on the rented card.
+# CPU torch, explicitly. Measured in a resolved surya venv: nvidia wheels
+# 3196 MB + triton 896 MB + torch 1175 MB = ~5.3 GB of CUDA stack, against
+# ~250 MB for a CPU build. The `ocr` profile's own card runs docTR and (when
+# baked) surya's local line detector; a CUDA resolve here is ~5 GB of image for
+# a venv that, on a GPU-less orchestrator, can never execute any of it --
+# `torch.cuda.is_available()` is False on the Digesta orchestrator today while
+# its surya venv carries cu130.
+#
+# Set an explicit CUDA pin here if you WANT surya's line detection on the card.
+--extra-index-url https://download.pytorch.org/whl/cpu
+torch==2.14.1+cpu
 surya-ocr==0.22.1
 pillow<11

packaging/runpod/profiles/vllm-ocr.txt

@@ -35,3 +35,38 @@
 # NOT a drop-in replacement for `vllm`: 0.20.1 is seven minor releases behind,
 # and anything else that image serves would be pinned back with it. Rent this
 # one for the OCR model only.
+
+# ---------------------------------------------------------------------------
+# WHAT THIS IMAGE IS NOW (0.3.69)
+#
+# Not "a different vLLM" — that difference is gone, this profile's venv-vllm.txt
+# is byte-identical to profiles/vllm.venv-vllm.txt and its check enforces that.
+# It is "the OCR PIPELINE on the pod": the surya client venv sits beside the
+# model, so the orchestrator sends a ONE-PAGE PDF and the card rasterises and
+# reads it, instead of shipping a ~1 MB base64 PNG per page across the internet
+# and resuming cut-off pages over four more round trips of the same.
+#
+# The main venv therefore needs the rasteriser. It must NOT grow a torch: this
+# is a light-core image and Dockerfile.capability fails the build if torch
+# reaches the main venv.
+#
+# STAMP/SIGNATURE DETECTION STAYS ON THE ORCHESTRATOR, deliberately, and this
+# image is not missing anything by not carrying it:
+#
+#   * `detect.py` imports ultralytics IN-PROCESS, and ultralytics brings torch.
+#     Putting it here means either a third torch in the image (main + vllm venv
+#     + surya venv, ~+2.5 GB compressed, slower pulls, and a slower warm floor)
+#     or a fourth venv plus a detect worker to reach it.
+#   * There is no model to carry anyway: `ocr.detect_model_path` defaults to ""
+#     and the weights are operator-supplied. Baking them would also break the
+#     rule that a released image contains no models, configs, keys or secrets.
+#   * And the orchestrator does not need the pod's help: it SPLIT the page, so
+#     it still holds that one-page PDF and can render it locally when — and only
+#     when — detection is enabled. `detect_mode` is off by default, so most
+#     installs pay nothing, and a detector hunting stamps does not need OCR
+#     resolution: at 100 DPI an A4 page is 0.97 Mpixels against 6.04 at 250.
+#
+# So `detect_mode` works the same in both modes. What differs is only WHERE the
+# pixels come from, and in neither case do they cross the network twice.
+pypdfium2>=4.20.0             # rasterise the page ON the card; no system poppler
+jsonschema>=4.0.0             # structured-extraction schemas are validated here

packaging/runpod/profiles/vllm-ocr.venv-surya.check

+# Deep check for the surya+detector venv on the POD.
+import PIL
+from PIL import Image                                       # noqa
+
+assert int(PIL.__version__.split(".")[0]) < 11, f"pillow {PIL.__version__} — surya caps <11"
+
+# The predictors ocr_worker.load() actually constructs.
+from surya.detection import DetectionPredictor              # noqa
+from surya.recognition import RecognitionPredictor          # noqa
+
+import surya
+import torch
+print("surya", getattr(surya, "__version__", "ok"), "pillow", PIL.__version__)
+print("torch", torch.__version__)
+
+# CPU on purpose, and asserted: a CUDA torch resolving in here silently adds
+# ~5.3 GB (nvidia 3196 MB + triton 896 MB + torch 1175 MB measured) AND puts
+# surya's local line detector on the card vLLM is serving the model from.
+assert torch.version.cuda is None, (
+    "this venv must hold a CPU torch (got cuda %s): the GPU belongs to vLLM, "
+    "and a CUDA build costs ~5.3 GB of image for a workload that does not want "
+    "the card" % torch.version.cuda)
+
+# The detector runs here too, sharing that torch — which is the whole reason it
+# costs ~30 MB instead of a third torch.
+from ultralytics import YOLO                                # noqa
+import ultralytics
+print("ultralytics", getattr(ultralytics, "__version__", "ok"), "on cpu torch")

packaging/runpod/profiles/vllm-ocr.venv-surya.txt

+# The surya CLIENT and the stamp/signature detector, on the POD — beside the
+# model they drive. CPU torch, deliberately.
+#
+# profiles/ocr.venv-surya.txt says of the same venv: "So the venv goes on the
+# ORCHESTRATOR image, never on the rented card." That was right while the
+# orchestrator held the pipeline; this image exists to invert it, and the
+# measurements are why:
+#
+#   * every page crosses the network as a base64 PNG today — ~1.06 MB per A4
+#     page at 250 DPI, where the source PDF page is ~474 KB;
+#   * a page RESUMED four times makes four internet round trips, each carrying
+#     the image again, where on the pod they are loopback;
+#   * the orchestrator pays ~105 ms/page to rasterise and holds the bitmap five
+#     times over (window, pipe, worker, surya, request body) — the accounting
+#     that got it OOM-killed twice at ~91 GB on 2026-10-14;
+#   * and surya's LOCAL line detector (DetectionPredictor, constructed in
+#     ocr_worker.load and passed into every recognition call) is a torch model
+#     running per page on a box with no GPU at all.
+#
+# WHY CPU TORCH AND NOT CUDA. Measured in the orchestrator's surya venv: nvidia
+# wheels 3196 MB + triton 896 MB + torch 1175 MB = ~5.3 GB of GPU stack. On this
+# image the GPU is already fully committed to vLLM serving the model, and the
+# one local torch workload — line detection — is small. Giving it the card would
+# take VRAM and SMs from the thing that matters; giving it the pod's spare vCPUs
+# costs nothing anyone is waiting on. So: ~250 MB instead of ~5.3 GB, and no
+# contention with vLLM.
+#
+# The detector rides the same CPU torch, which is what makes it ~30 MB here
+# rather than a third torch in the image. It is NOT accompanied by weights:
+# ocr.detect_model_path defaults to "" and the model is operator-supplied, and a
+# released image must contain no models, configs, keys or secrets.
+--extra-index-url https://download.pytorch.org/whl/cpu
+torch==2.14.1+cpu
+surya-ocr==0.22.1
+pillow<11
+# detector mode (ocr.detect_mode=detector|both). Shares the torch above.
+ultralytics>=8.1.0

packaging/runpod/profiles/vllm-ocr.venv-vllm.check

-# Deep check for the pinned-0.20.1 vllm venv: the OpenAI server module must
-# import, and the version must be the one this image exists to pin — a silent
-# resolve to something newer would defeat the whole point of the image.
+# Deep check for the vllm venv: the OpenAI server module must import, and the
+# version must MATCH the plain `vllm` profile rather than being pinned away from
+# it. Two images serving the same weights on two different vLLMs is an invisible
+# difference, and this image used to be exactly that.
+import pathlib
 import vllm.entrypoints.openai.api_server                 # noqa
 import torch, vllm
 print("vllm", vllm.__version__, "torch", torch.__version__, "cuda", torch.version.cuda)
-assert vllm.__version__.startswith("0.20."), (
-    "this image exists to pin vLLM 0.20.x (what surya-ocr validates against); "
-    "got %s" % vllm.__version__)
+
+# The pin lives in profiles/vllm.venv-vllm.txt; read it rather than restate it,
+# so bumping the fleet cannot silently leave this image behind.
+_want = ""
+for _p in ("/tmp/venv-profiles/vllm.venv-vllm.txt",
+           "packaging/runpod/profiles/vllm.venv-vllm.txt"):
+    _f = pathlib.Path(_p)
+    if _f.is_file():
+        for _line in _f.read_text().splitlines():
+            if _line.strip().startswith("vllm=="):
+                _want = _line.strip().split("==", 1)[1].split("+")[0]
+        break
+if _want:
+    assert vllm.__version__.startswith(_want), (
+        "this image must serve the same vLLM as profiles/vllm (%s); got %s"
+        % (_want, vllm.__version__))
+    print("vllm matches the fleet profile:", _want)
+else:
+    print("could not read the fleet's vllm pin; version match not enforced here")
 
 # Surya-2 is Qwen3.5-VL, so the architecture has to be in this vLLM's registry or
-# the pod cannot serve the model at all. Checked through the registry's own
-# lookup where one is exposed and reported-not-enforced otherwise: the private
-# table has been renamed before, and failing a BUILD on the name of a private
-# symbol would be failing for the wrong reason.
+# the pod cannot serve the model at all. Reported-not-enforced where the registry
+# cannot be enumerated: the private table has been renamed before, and failing a
+# BUILD on the name of a private symbol would be failing for the wrong reason.
 _ARCH = "Qwen3_5ForConditionalGeneration"
 try:
     from vllm.model_executor.models import registry as _reg

packaging/runpod/profiles/vllm-ocr.venv-vllm.txt

-# vLLM 0.20.1 — the version `surya-ocr` 0.22.1 spawns and validates against
-# (surya/settings.py:96). See vllm-ocr.txt for why this image exists.
+# vLLM for Surya-2 OCR, on the SAME version the rest of the fleet runs.
 #
-# The CUDA trap is the same one the 0.27.1 profile documents: the PyPI wheel is
-# built against a CUDA the pod hosts do not have, so the +cu129 variant from
-# vLLM's own index is used, with torch's matching cu129 builds. CUDA 12.x
-# minor-version compatibility runs a 12.9 runtime on a 12.8 driver. Resolved
-# with uv (the .uv marker); pip cannot cope with the pin set.
+# This file used to pin v0.20.1, because that is the version `surya-ocr` itself
+# spawns (surya/settings.py:96 VLLM_DOCKER_IMAGE = "vllm/vllm-openai:v0.20.1")
+# and because a page of crisp synthetic Italian came back from 0.27.1 with one
+# U+FFFD per accented letter. The pin was the wrong conclusion from a real
+# measurement: surya-ocr-2's tokenizer is BYTE-level, so a U+FFFD per accent
+# does not show the model chose it — it shows bytes being split and decoded
+# separately somewhere downstream. The Digesta fleet has since been serving
+# surya-ocr-2 on 0.27.1 (`coderai-vllm`) and reading accented Italian.
 #
-# The torch triple is NOT a guess: it is what vLLM 0.20.1's own metadata pins
-# (torch==2.11.0, torchvision==0.26.0, torchaudio==2.11.0), and every one of
-# those exists as a +cu129 cp313 wheel on torch's index — checked before this
-# file was written, because a profile that cannot build is worse than none.
---extra-index-url https://wheels.vllm.ai/0.20.1/cu129
+# So the version difference is gone and the reason for this image changed: it is
+# no longer "a different vLLM" but "the OCR PIPELINE on the pod" — the surya
+# client venv beside the model (vllm-ocr.venv-surya.txt), so a page is
+# rasterised and read on the card instead of crossing the network as a bitmap.
+# Keep this list identical to profiles/vllm.venv-vllm.txt: two images serving
+# the same weights on two vLLMs is exactly the invisible difference that cost
+# the Digesta dev three rounds of blaming the pod image.
+--extra-index-url https://wheels.vllm.ai/0.27.1/cu129
 --extra-index-url https://download.pytorch.org/whl/cu129
-vllm==0.20.1+cu129
-torch==2.11.0+cu129
-torchvision==0.26.0+cu129
+vllm==0.27.1+cu129
+torch==2.13.0+cu129
+torchvision==0.28.0+cu129
 torchaudio==2.11.0+cu129
-# NO torchcodec pin here, deliberately. 0.27.1 needs one (>=0.14) and pinning it
-# to a cu129 build is what stops `import vllm.entrypoints.openai.api_server`
-# dying on libnvrtc.so.13 — but 0.20.1's PyPI metadata does not list torchcodec
-# at all (checked), and pinning a package nothing requires is its own hazard. If
-# a build ever resolves one from PyPI and the server import fails on an nvrtc
-# soname, add `torchcodec==0.11.1+cu129` (it exists for cp313 and matches torch
-# 2.11).
-#
-# transformers: 0.20.1 excludes 5.0-5.4 and 5.5.0, so uv lands on 5.5.1+. That
-# is required anyway — surya-ocr-2 is `qwen3_5`, which only transformers 5.x
-# knows.
+torchcodec==0.16.0+cu129

tests/test_ocr_accent_repair.py

@@ -250,19 +250,26 @@ def test_it_defaults_on():
 
 
 # ------------------------------------------- option 2: the pinned-vLLM image #
-def test_the_vllm_ocr_profile_exists_and_pins_the_version_surya_validates():
-    """Surya spawns `vllm/vllm-openai:v0.20.1` (surya/settings.py:96). Serving the
-    same weights on 0.27.1 is what loses the accents, so the second half of the
-    answer is an image on the version its authors test."""
+def test_the_vllm_ocr_profile_runs_the_SAME_vllm_as_the_fleet():
+    """This image was built to pin vLLM 0.20.1 — the version surya spawns
+    (surya/settings.py:96) — because a page of crisp synthetic Italian came back
+    from 0.27.1 with one U+FFFD per accented letter.
+
+    The pin was the wrong conclusion from a real measurement: surya-ocr-2's
+    tokenizer is BYTE-level, so a U+FFFD per accent does not show the model
+    choosing it, and the Digesta fleet has since been reading accented Italian on
+    0.27.1 (`coderai-vllm`). Two images serving the same weights on two vLLMs is
+    an invisible difference, and that difference cost three rounds of blaming the
+    pod image. So the pins must be IDENTICAL, and 0.3.69 repurposed this image
+    for what actually distinguishes it: the OCR pipeline on the pod."""
     import pathlib
     prof = pathlib.Path(__file__).resolve().parents[1] / "packaging/runpod/profiles"
-    reqs = (prof / "vllm-ocr.venv-vllm.txt").read_text()
-    assert "vllm==0.20.1+cu129" in reqs
-    # The torch triple is vLLM 0.20.1's own metadata, not a guess.
-    assert "torch==2.11.0+cu129" in reqs
-    assert "torchvision==0.26.0+cu129" in reqs
-    assert "torchaudio==2.11.0+cu129" in reqs
-    # uv, because pip cannot resolve this pin set (as for the 0.27.1 profile).
+    fleet = [l.strip() for l in (prof / "vllm.venv-vllm.txt").read_text().splitlines()
+             if l.strip() and not l.strip().startswith("#")]
+    ours = [l.strip() for l in (prof / "vllm-ocr.venv-vllm.txt").read_text().splitlines()
+            if l.strip() and not l.strip().startswith("#")]
+    assert ours == fleet, "the OCR image drifted from the fleet's vLLM again"
+    # uv, because pip cannot resolve this pin set.
     assert (prof / "vllm-ocr.venv-vllm.uv").exists()
     # The light core, like the profile it is derived from.
     assert (prof / "vllm-ocr.light").exists()
@@ -270,17 +277,31 @@ def test_the_vllm_ocr_profile_exists_and_pins_the_version_surya_validates():
     assert "gcc" in (prof / "vllm-ocr.apt").read_text()
 
 
-def test_the_build_check_refuses_a_silent_version_drift():
-    """The whole point of the image is the pin. If uv resolved something newer the
-    build has to fail rather than produce an image that looks right."""
+def test_the_build_check_refuses_a_silent_drift_from_the_fleet():
+    """The check no longer asserts a hardcoded version — it READS the fleet's pin
+    from disk and requires a match, so bumping profiles/vllm cannot silently
+    leave the OCR image behind on an older vLLM."""
     import pathlib
     prof = pathlib.Path(__file__).resolve().parents[1] / "packaging/runpod/profiles"
     check = (prof / "vllm-ocr.venv-vllm.check").read_text()
-    assert 'startswith("0.20.")' in check
+    assert "vllm.venv-vllm.txt" in check, "the check does not read the fleet pin"
+    assert "vllm.__version__.startswith(_want)" in check
     # ...but it must not fail the build on the NAME of a private vLLM symbol.
     assert "except ImportError" in check
 
 
+def test_the_pipeline_is_what_distinguishes_the_ocr_image():
+    """Since the vLLM versions match, the surya client venv IS the difference."""
+    import pathlib
+    prof = pathlib.Path(__file__).resolve().parents[1] / "packaging/runpod/profiles"
+    assert (prof / "vllm-ocr.venv-surya.txt").exists()
+    surya = (prof / "vllm-ocr.venv-surya.txt").read_text()
+    assert "surya-ocr==" in surya
+    assert "pillow<11" in surya
+    # and the rasteriser, in the main venv, because the pod renders its own page
+    assert "pypdfium2" in (prof / "vllm-ocr.txt").read_text()
+
+
 def test_the_profile_is_discoverable_by_the_builder():
     """Profiles are found by globbing *.txt, so the file's existence is the
     registration — but the .txt must not be mistaken for a venv spec."""

tests/test_ocr_detector_venv.py

+"""The stamp/signature detector, reachable from a venv that is not ours.
+
+`detect.py` imported ultralytics IN-PROCESS, which works wherever the main venv
+carries torch. The `vllm-ocr` pod image is light-core on purpose —
+Dockerfile.capability FAILS the build if torch reaches the main venv — so the
+detector could not live there without a third torch (~2.5 GB compressed) for a
+workload that does not even want the card: the GPU is serving the model.
+
+So ultralytics is installed beside surya, sharing the CPU torch already in that
+venv, and reached through a worker. Measured, in the orchestrator's surya venv:
+nvidia 3196 MB + triton 896 MB + torch 1175 MB = ~5.3 GB of GPU stack, for a
+venv whose only local torch workload is surya's small line detector. A CPU torch
+is ~250 MB, and the detector then costs ~30 MB on top instead of a torch.
+
+The rule these tests hold: a detector that cannot be reached must look like a
+MISSING detector (the ordinary 503), never like a broken install.
+"""
+
+import io
+import pathlib
+import sys
+
+import pytest
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from codai.ocr import detect as dt                           # noqa: E402
+from codai.ocr.base import OcrError                          # noqa: E402
+
+
+@pytest.fixture(autouse=True)
+def _no_shared_worker(monkeypatch):
+    monkeypatch.setattr(dt._DetectWorker, "_inst", None, raising=False)
+    yield
+    dt._DetectWorker._inst = None
+
+
+def _img(w=40, h=40):
+    Image = pytest.importorskip("PIL.Image")
+    return Image.new("RGB", (w, h), "white")
+
+
+# ------------------------------------------------- in-process stays first #
+def test_in_process_ultralytics_is_preferred(monkeypatch, tmp_path):
+    """Nothing changes on an install whose main venv has torch."""
+    pt = tmp_path / "w.pt"
+    pt.write_bytes(b"x")
+    used = {}
+
+    class _YOLO:
+        def __init__(self, path):
+            used["path"] = path
+
+        def predict(self, arr, conf=0.0, verbose=False):
+            return []
+    mod = type(sys)("ultralytics")
+    mod.YOLO = _YOLO
+    monkeypatch.setitem(sys.modules, "ultralytics", mod)
+    monkeypatch.setattr(dt._DetectWorker, "get",
+                        classmethod(lambda cls: pytest.fail("worker used")))
+    d = dt.YoloDetector(str(pt), 0.35)
+    assert d.detect(_img()) == []
+    assert used["path"] == str(pt)
+
+
+def test_missing_weights_is_a_400_before_anything_is_loaded(tmp_path):
+    """The weights are operator-supplied — ocr.detect_model_path defaults to ""
+    and a released image carries no models — so this is the common misconfig."""
+    d = dt.YoloDetector(str(tmp_path / "nope.pt"), 0.35)
+    with pytest.raises(OcrError) as e:
+        d.ensure_loaded()
+    assert e.value.status == 400
+
+
+def test_an_empty_weights_path_is_also_a_400():
+    d = dt.YoloDetector("", 0.35)
+    with pytest.raises(OcrError) as e:
+        d.ensure_loaded()
+    assert e.value.status == 400
+
+
+# ------------------------------------------------------- the venv fallback #
+def test_no_ultralytics_and_no_venv_is_the_ordinary_503(monkeypatch, tmp_path):
+    pt = tmp_path / "w.pt"
+    pt.write_bytes(b"x")
+    monkeypatch.setitem(sys.modules, "ultralytics", None)
+    monkeypatch.setattr(dt._DetectWorker, "get", classmethod(lambda cls: None))
+    d = dt.YoloDetector(str(pt), 0.35)
+    with pytest.raises(OcrError) as e:
+        d.ensure_loaded()
+    assert e.value.status == 503
+    assert "ultralytics is not installed" in str(e.value)
+
+
+def test_the_worker_is_used_when_ultralytics_is_elsewhere(monkeypatch, tmp_path):
+    pt = tmp_path / "w.pt"
+    pt.write_bytes(b"x")
+    monkeypatch.setitem(sys.modules, "ultralytics", None)
+    seen = {}
+
+    class _Fake:
+        def detect(self, image, model_path, conf):
+            seen["path"] = model_path
+            seen["conf"] = conf
+            seen["size"] = image.size
+            return [{"label": "stamp", "conf": 0.9, "bbox": [1, 2, 3, 4]}]
+    monkeypatch.setattr(dt._DetectWorker, "get", classmethod(lambda cls: _Fake()))
+    d = dt.YoloDetector(str(pt), 0.42)
+    got = d.detect(_img(50, 60))
+    assert got == [{"label": "stamp", "conf": 0.9, "bbox": [1, 2, 3, 4]}]
+    assert seen == {"path": str(pt), "conf": 0.42, "size": (50, 60)}
+
+
+def test_the_surya_venv_is_looked_at_first():
+    """That is where the pod image puts ultralytics, sharing the CPU torch
+    already there — the reason it costs ~30 MB and not another torch."""
+    assert dt._DetectWorker._VENVS[0] == "/opt/coderai/venvs/surya"
+
+
+def test_a_venv_that_will_not_start_does_not_raise(monkeypatch, tmp_path):
+    """A broken venv must degrade to 'no detector', not to a 500."""
+    vdir = tmp_path / "venvs" / "surya" / "bin"
+    vdir.mkdir(parents=True)
+    (vdir / "python").write_text("#!/bin/false\n")
+    monkeypatch.setattr(dt._DetectWorker, "_VENVS",
+                        (str(tmp_path / "venvs" / "surya"),))
+
+    def _boom(self, python):
+        raise OcrError("nope", status=503)
+    monkeypatch.setattr(dt._DetectWorker, "__init__", _boom)
+    assert dt._DetectWorker.get() is None
+
+
+def test_the_worker_is_shared_across_detectors(monkeypatch):
+    calls = []
+
+    class _Fake:
+        pass
+    monkeypatch.setattr(dt._DetectWorker, "_VENVS", ())
+    # With no venvs, get() returns None every time and never caches a None as
+    # if it were a worker -- so a venv appearing later is still found.
+    assert dt._DetectWorker.get() is None
+    assert dt._DetectWorker._inst is None
+
+
+# -------------------------------------------------------- the worker script #
+def test_the_worker_does_not_import_the_ocr_package():
+    """`import codai.ocr` costs 377 MB and 1.5 s via litellm; the worker exists
+    to stay small."""
+    import ast
+    src = (ROOT / "codai/ocr/workers/detect_worker.py").read_text()
+    names = set()
+    for node in ast.walk(ast.parse(src)):
+        if isinstance(node, ast.Import):
+            names.update(a.name.split(".")[0] for a in node.names)
+        elif isinstance(node, ast.ImportFrom) and node.module:
+            names.add(node.module.split(".")[0])
+    assert "codai" not in names, sorted(names)
+
+
+def test_the_worker_is_run_isolated():
+    src = (ROOT / "codai/ocr/detect.py").read_text()
+    assert '"-I"' in src, "the worker must not pick up modules from the cwd"
+
+
+def test_the_worker_keeps_serving_after_a_bad_page():
+    """One undecodable page must not cost the detector its loaded weights."""
+    src = (ROOT / "codai/ocr/workers/detect_worker.py").read_text()
+    body = src.split('if cmd != "detect"')[1]
+    assert body.count("continue") >= 3, (
+        "a failure path exits instead of answering and carrying on")
+
+
+# ------------------------------------------------- the image-size reasoning #
+def test_the_pod_venv_pins_a_cpu_torch():
+    """A CUDA torch resolving in there silently adds ~5.3 GB AND puts surya's
+    line detector on the card vLLM is serving from."""
+    txt = (ROOT / "packaging/runpod/profiles/vllm-ocr.venv-surya.txt").read_text()
+    assert "download.pytorch.org/whl/cpu" in txt
+    assert "+cpu" in txt
+    assert "ultralytics" in txt
+
+
+def test_the_pod_venv_check_enforces_cpu_torch():
+    chk = (ROOT / "packaging/runpod/profiles/vllm-ocr.venv-surya.check").read_text()
+    assert "torch.version.cuda is None" in chk
+    assert "from ultralytics import YOLO" in chk
+
+
+def test_the_pod_image_does_not_bake_detector_weights():
+    """ocr.detect_model_path defaults to "" and a released image must contain no
+    models, configs, keys or secrets."""
+    for f in ("vllm-ocr.txt", "vllm-ocr.venv-surya.txt"):
+        txt = (ROOT / "packaging/runpod/profiles" / f).read_text()
+        assert ".pt" not in txt.replace("detect_model_path", ""), f

tests/test_pool_growth_adopts.py

@@ -80,9 +80,25 @@ def test_every_provisioning_path_tries_adoption_first():
     by booting a pod THERE, and adopting a pod elsewhere would not do that.
     """
     src = (ROOT / "codai/api/runpod_worker.py").read_text()
-    calls = src.count("self._provision_one()")
-    guarded = src.count("if not self._adopt_shared_pod():")
-    adopt_first = src.count("if self._adopt_shared_pod():")
     # 5 call sites: ensure_ready, acquire, growth, warm floor, seed_region.
-    assert calls == 5
-    assert guarded + adopt_first == 4
+    assert src.count("self._provision_one()") == 5
+
+    # Per PATH rather than a global count of one spelling. The warm floor stopped
+    # matching `if not self._adopt_shared_pod():` in 0.3.69 when it became a
+    # BOUNDED loop (adoption can report success without adding a pod, and the
+    # `while` form span forever the first time it did) -- the intent held, the
+    # string did not, and a count cannot tell those apart.
+    def _body(start: str, end: str) -> str:
+        return src.split(start)[1].split(end)[0]
+
+    paths = {
+        "ensure_ready": _body("def ensure_ready", "\n    def "),
+        "acquire": _body("def acquire", "\n    def "),
+        "growth": _body("def _grow_in_background", "\n    def "),
+        "warm floor": _body("# Keep min_pods warm.", "def _warm_provision_one"),
+    }
+    for name, body in paths.items():
+        assert "_adopt_shared_pod()" in body, f"{name} does not try adoption"
+        if "_provision_one" in body:
+            assert body.index("_adopt_shared_pod()") < body.index("_provision_one"), (
+                f"{name} rents before it tries adoption")

tests/test_runpod_client_scale.py

@@ -337,9 +337,15 @@ def test_ensure_ready_marks_the_pool_as_provisioning():
 
 
 def test_the_warm_top_up_does_not_stack_on_a_booting_pod():
+    """A floor of 1 must not become 2 by topping up against a pod already
+    booting. 0.3.69 fills a floor in waves rather than one pod per pass, so
+    the old `not self._booting` guard is gone and the intent is arithmetic:
+    `_booting` is counted in `in_hand`, so the deficit a wave may request is
+    already reduced by whatever is on the way."""
     src = __import__("pathlib").Path(rw.__file__).read_text()
     blk = src.split("def maintain")[1].split("def close")[0]
-    assert "not self._booting" in blk
+    assert "in_hand = len([p for p in self.pods if p.healthy]) + self._booting" in blk
+    assert "want = min(keep_min - in_hand," in blk
     assert "sibling_is_provisioning" in blk
 
 

tests/test_runpod_scale_out.py

@@ -169,14 +169,23 @@ def test_the_counters_exist_without___init__():
 
 
 def test_the_single_flight_paths_ask_how_many_not_whether():
-    """ensure_ready (the FIRST pod) and the warm-floor top-up must still be
-    single-flight — they are not allowed to grow a fleet — so they check the
-    count directly rather than a boolean that scale-up also writes."""
+    """ensure_ready (the FIRST pod) must still be single-flight: one request may
+    not rent a fleet, so it checks the count directly rather than a boolean that
+    scale-up also writes.
+
+    The warm-floor top-up deliberately is NOT single-flight any more (0.3.69). A
+    floor is a DECLARATION — min_pods, keep_warm, or a client lease already said
+    how many pods should be warm — so filling it is not speculative. One pod per
+    pass, each taking ~5 minutes, made a declared floor of 8 take ~40 minutes to
+    exist. It is bounded instead by the model's own scale_up_max_parallel_boots,
+    and `_booting` is counted so a wave cannot overshoot the floor."""
     src = (ROOT / "codai/api/runpod_worker.py").read_text()
     ready = src.split("def ensure_ready")[1].split("\n    def ")[0]
     assert "if self._booting:" in ready and "self._booting += 1" in ready
     maint = src.split("def maintain")[1].split("\n    def ")[0]
-    assert "not self._booting" in maint
+    assert "scale_up_max_parallel_boots" in maint
+    assert "parallel - self._booting" in maint
+    assert "+ self._booting" in maint, "a wave must count what is already booting"
 
 
 def test_the_counter_returns_to_zero_after_a_scale_up():

tests/test_runpod_warm_floor_rerent.py

@@ -25,6 +25,7 @@ API.
 
 import pathlib
 import sys
+import time
 import threading
 
 ROOT = pathlib.Path(__file__).resolve().parents[1]
@@ -279,8 +280,18 @@ def test_the_floor_is_re_established_once_the_ghost_is_gone(monkeypatch):
     monkeypatch.setattr("builtins.print", lambda *a, **k: out.append(" ".join(map(str, a))))
     pool.maintain()
     monkeypatch.undo()
+    # Since 0.3.69 the floor is filled on BACKGROUND threads: _provision_one is
+    # "Blocking (minutes)" and calling it inline from the scaler loop stalled
+    # the registry heartbeat that shares that loop. So wait for the boot this
+    # pass started rather than reading `rented` the instant maintain() returns.
+    for _ in range(100):
+        if rented:
+            break
+        time.sleep(0.02)
     assert rented == ["rented"]
-    assert any("warm floor 1" in line and "renting one" in line for line in out), out
+    # And the wording carries the wave size now: a floor of 8 says "renting 4"
+    # when four may boot at once, which is the number an operator needs to see.
+    assert any("warm floor 1" in line and "renting 1" in line for line in out), out
 
 
 def test_a_sibling_already_renting_is_one_pod_not_none(monkeypatch):