surya read nothing because its lease never stayed home (v0.3.98)

THE SURYA BUG. `/v1/ocr/attach/<lease>/…` is an OCR engine attaching to THIS
install's gateway. The lease is minted here and lives in this process's memory
— and the remote gateway was FORWARDING those requests to the pod, which
correctly refused a token it never issued:

    404 {"error": {"message": "unknown OCR engine lease"}}

surya's client swallows that and returns an empty generation, so every page
came back unread with the GPU never asked to read one. Measured on a clean
2-page PDF that chandra (2212 chars) and olmOCR (1992) both read: surya
returned 2 characters, and the pod's vLLM logged a single 6-token warm-up for
the whole document. $88 of surya time in one day produced nothing.

The gateway behind that prefix does its OWN forwarding, with its own pool and
its own lease. That is the hop, and it has to start here. `/v1/ocr` itself
stays routable, so pinning the attachment does not strand the subsystem.

This also corrects an earlier diagnosis of mine: v0.3.87 treated this as a
lease VOIDED BY A RESTART and shipped a re-resolve for it. No restart was
involved — the lease was wrong from the first page, which is why surya has
been broken continuously rather than only after restarts. That work was not
wasted (the retry cap took four paid generations per page down to two, and the
502 plus `unread_pages` are what made this diagnosable at all) but it did not
address the cause.

Third instance today of one mistake: the gateway forwards anything under /v1/
that looks like a model request, including this install's own control plane.
/v1/runpod rented a pod to deliver a teardown; /v1/loras was staying local by
luck; now the OCR lease. All three are pinned. The structural answer — making
control-plane routes opt IN to forwarding — is not in this commit.

Also in here, from the fleet-hygiene pass:

  * accelerate added to every profile that carries transformers (audio, ocr,
    stt, tts, voice, tts-xtts.venv-TTS), with a test that fails if a new
    profile reintroduces the gap. Same latent fault that broke embeddings.
  * an embeddings pod is warmed on /v1/embeddings, not with a chat completion
    it refuses by design — that 404 was printed on every boot of a healthy pod
    and cost a wrong diagnosis.
  * the seed plan is announced once, not every fifteen seconds while a pod
    boots, so a boot log stays readable when it matters.
  * `pool:` is dropped from pod NAMES (ownership is the prefix, never the rest).
  * the reaper owns a pod by REGISTRY as well as by name, so renaming
    deployment_id no longer orphans pods that are already running and billing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.97"
+__version__ = "0.3.98"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/remote_gateway.py

@@ -155,8 +155,21 @@ _SKIP_PATHS = ("/v1/chat/completions", "/v1/completions", "/v1/models",
 #: would resolve nowhere. It carries no `model` field today, so it happens to
 #: stay local by accident; that is luck, not design, and the same shape of
 #: accident as /v1/runpod, which did not stay local.
+#: ``/v1/ocr/attach`` is an OCR engine's attachment to THIS install's gateway.
+#: The lease in the path was minted here and lives in this process's memory, so
+#: forwarding the request hands a pod a token it never issued — which it
+#: correctly refuses:
+#:
+#:     404 {"error": {"message": "unknown OCR engine lease"}}
+#:
+#: surya's client swallows that and returns an empty generation, so every page
+#: came back unread with the GPU never asked to read one. Measured: the pod's
+#: vLLM logged a single 6-token warm-up for a whole document, and $88 of surya
+#: time in a day produced nothing. The gateway behind this prefix does its own
+#: forwarding, with its own pool and its own lease — that is the hop, and it
+#: must start here.
 _LOCAL_PREFIXES = ("/v1/files/upload", "/v1/files/blob", "/v1/runpod",
-                   "/v1/loras")
+                   "/v1/loras", "/v1/ocr/attach")
 
 #: Orchestration endpoints: a sequence of calls to other endpoints, not a model.
 #: They are NEVER forwarded as a whole — the chain stays here and each step goes

codai/api/runpod_worker.py

@@ -1234,6 +1234,11 @@ def _ocr_external_server(engine: str):
     return ""
 
 
+#: Seed lists already announced, so a plan recomputed every scaler pass does
+#: not repeat itself. Bounded: this must never become a place memory grows.
+_PLAN_SAID: dict = {}
+
+
 def _plan(engine, image, args, mcfg, api_key, entry, served,
           seed_entries: list = None, capability: str = "") -> dict:  # noqa: D401
     """Finish a plan for a coderai pod: auth plus the models it must serve."""
@@ -1302,7 +1307,17 @@ def _plan(engine, image, args, mcfg, api_key, entry, served,
             seen.add(key)
             unique.append(s)
         env["CODERAI_SEED_MODELS"] = _json.dumps(unique)
-        print(f"[runpod] pod will be told about {len(unique)} model(s)", flush=True)
+        # Once per distinct seed list. A plan is recomputed on every scaler
+        # pass — roughly every fifteen seconds while a pod boots — and saying
+        # the same sentence each time buries the lines that matter in a boot
+        # log somebody is reading precisely because something went wrong.
+        _sig = (str(image), env["CODERAI_SEED_MODELS"])
+        if _PLAN_SAID.get(_sig) is None:
+            while len(_PLAN_SAID) > 64:
+                _PLAN_SAID.pop(next(iter(_PLAN_SAID)))
+            _PLAN_SAID[_sig] = True
+            print(f"[runpod] pod will be told about {len(unique)} model(s)",
+                  flush=True)
     # A pod rented for a subsystem that ships disabled has to arrive with it on.
     # An OCR pod booted, accepted the request and answered "OCR subsystem is
     # disabled (enable it in Settings → OCR)" — a settings screen nobody will
@@ -2899,8 +2914,14 @@ class RunpodPodPool:
             block = self._budget_blocks(sel["price"])
             if block:
                 raise RunpodError(f"RunPod budget cap hit — not provisioning ({block}).")
+            # `pool:` is dropped from the NAME only — it is legal on RunPod
+            # but makes a console listing hard to read, and ownership is
+            # decided by the `coderai-<deployment>-` prefix, never by the rest.
+            _key = str(self.model_key)
+            if _key.startswith("pool:"):
+                _key = _key[len("pool:"):]
             name = (f"coderai-{_deploy_tag(self.account)}-"
-                    + str(self.model_key)[:20].replace("/", "_").replace(" ", "_")
+                    + _key[:20].replace("/", "_").replace(" ", "_")
                     + "-" + uuid.uuid4().hex[:6])
             _dc = sel.get("_dc") or self._data_center()
             # Say so when this is not the card that was asked for. Renting a
@@ -3213,14 +3234,30 @@ class RunpodPodPool:
         if budget <= 0:
             return False
         t0 = time.time()
-        payload = {"model": self.served or str(self.model_key),
-                   "messages": [{"role": "user", "content": "ok"}],
-                   "max_tokens": 1, "temperature": 0.0}
+        name = self.served or str(self.model_key)
+        # The RIGHT endpoint for what this pod serves. Asking an embeddings pod
+        # for a chat completion is a 404 by design —
+        #   "Model 'BAAI/bge-m3' is a embedding model and cannot be used for
+        #    text generation. Use the embedding endpoint instead."
+        # — printed at WARNING volume on every boot of a pod that is perfectly
+        # healthy. It reads as a broken pod, and it cost a wrong diagnosis.
+        cap = ""
+        try:
+            cap = model_capability(self.entry or {})
+        except Exception:
+            cap = ""
+        if cap == "embeddings":
+            path, payload = "/v1/embeddings", {"model": name, "input": "ok"}
+        else:
+            path = "/v1/chat/completions"
+            payload = {"model": name,
+                       "messages": [{"role": "user", "content": "ok"}],
+                       "max_tokens": 1, "temperature": 0.0}
         headers = {"Content-Type": "application/json"}
         if self.api_key:
             headers["Authorization"] = f"Bearer {self.api_key}"
         try:
-            reply = pod_http.post(url.rstrip("/") + "/v1/chat/completions",
+            reply = pod_http.post(url.rstrip("/") + path,
                                   json=payload, headers=headers, timeout=budget)
         except Exception as exc:
             print(f"[runpod] pod {pod_id}: warm-up did not complete "
@@ -5523,6 +5560,17 @@ def reap_orphans() -> int:
         return 0
     from codai.api.runpod_client import RunpodClient, RunpodError
     prefix = f"coderai-{_deploy_tag(acct)}-"
+    # Pods this deployment recorded, whatever they are CALLED. Ownership was
+    # decided by the name alone, so renaming `deployment_id` instantly orphaned
+    # every pod already running under the old prefix: still billing, and no
+    # longer anything's to reap. The registry is the better authority — it is
+    # written when we create a pod — and the prefix stays as the fallback for a
+    # pod this process never recorded.
+    _ours = set()
+    try:
+        _ours = {str(k) for k in (_registry_update(lambda d: False) or {})}
+    except Exception:
+        _ours = set()
     try:
         client = RunpodClient(acct)
         pods = client.list_pods()
@@ -5564,7 +5612,7 @@ def reap_orphans() -> int:
     for p in pods:
         pid, name = p.get("id"), (p.get("name") or "")
         status = str(p.get("status") or "").upper()
-        if not pid or not name.startswith(prefix):
+        if not pid or not (name.startswith(prefix) or str(pid) in _ours):
             continue                      # not ours (or another deployment's)
         if pid in known:
             continue                      # tracked by a live pool / provisioning

packaging/runpod/profiles/audio.txt

 # /v1/audio/generate, /v1/audio/stems, /v1/audio/cleanup
 torchaudio
 transformers>=4.35.0
+# transformers refuses a `device_map`/`torch.device` load without accelerate, so
+# a pod boots healthy, answers /v1/models, and then fails EVERY request with
+#   Model '<name>' failed to load: Using a `device_map` ... requires `accelerate`
+# which reads as a model problem rather than an image problem. Measured on a
+# real A40 for the embeddings image: 303s of boot paid before anyone could find
+# out. Every profile that carries transformers carries this with it.
+accelerate>=0.26.0
 diffusers>=0.25.0
 demucs>=4.0.1
 deepfilternet>=0.5.6

packaging/runpod/profiles/ocr.txt

@@ -13,6 +13,13 @@
 # and can be baked in later with the <profile>.venv-<name>.txt mechanism if a
 # pod ever needs its layout quality.
 transformers>=4.35.0
+# transformers refuses a `device_map`/`torch.device` load without accelerate, so
+# a pod boots healthy, answers /v1/models, and then fails EVERY request with
+#   Model '<name>' failed to load: Using a `device_map` ... requires `accelerate`
+# which reads as a model problem rather than an image problem. Measured on a
+# real A40 for the embeddings image: 303s of boot paid before anyone could find
+# out. Every profile that carries transformers carries this with it.
+accelerate>=0.26.0
 pypdfium2>=4.20.0
 python-doctr>=0.8.0
 ultralytics>=8.1.0

packaging/runpod/profiles/stt.txt

 # /v1/audio/transcriptions + diarization/speaker endpoints
 torchaudio
 transformers>=4.35.0
+# transformers refuses a `device_map`/`torch.device` load without accelerate, so
+# a pod boots healthy, answers /v1/models, and then fails EVERY request with
+#   Model '<name>' failed to load: Using a `device_map` ... requires `accelerate`
+# which reads as a model problem rather than an image problem. Measured on a
+# real A40 for the embeddings image: 303s of boot paid before anyone could find
+# out. Every profile that carries transformers carries this with it.
+accelerate>=0.26.0
 faster-whisper>=0.10.0
 openai-whisper>=20231117
 vosk>=0.3.45

packaging/runpod/profiles/tts-xtts.venv-TTS.txt

@@ -3,6 +3,13 @@
 torch==2.5.1+cu124
 torchaudio==2.5.1+cu124
 transformers==4.46.3
+# transformers refuses a `device_map`/`torch.device` load without accelerate, so
+# a pod boots healthy, answers /v1/models, and then fails EVERY request with
+#   Model '<name>' failed to load: Using a `device_map` ... requires `accelerate`
+# which reads as a model problem rather than an image problem. Measured on a
+# real A40 for the embeddings image: 303s of boot paid before anyone could find
+# out. Every profile that carries transformers carries this with it.
+accelerate>=0.26.0
 coqui-tts>=0.24.0
 # coqui-tts imports click at module load but does not declare it, so the venv
 # installs cleanly and then fails its import check. Found by iterating the venv

packaging/runpod/profiles/tts.txt

 # /v1/audio/speech
 torchaudio
 transformers>=4.35.0
+# transformers refuses a `device_map`/`torch.device` load without accelerate, so
+# a pod boots healthy, answers /v1/models, and then fails EVERY request with
+#   Model '<name>' failed to load: Using a `device_map` ... requires `accelerate`
+# which reads as a model problem rather than an image problem. Measured on a
+# real A40 for the embeddings image: 303s of boot paid before anyone could find
+# out. Every profile that carries transformers carries this with it.
+accelerate>=0.26.0
 soundfile>=0.12.0
 scipy>=1.11.0
 edge-tts>=6.1.9

packaging/runpod/profiles/voice.txt

 # /v1/audio/clone, /v1/audio/voices, /v1/audio/convert, watermarking
 torchaudio
 transformers>=4.35.0
+# transformers refuses a `device_map`/`torch.device` load without accelerate, so
+# a pod boots healthy, answers /v1/models, and then fails EVERY request with
+#   Model '<name>' failed to load: Using a `device_map` ... requires `accelerate`
+# which reads as a model problem rather than an image problem. Measured on a
+# real A40 for the embeddings image: 303s of boot paid before anyone could find
+# out. Every profile that carries transformers carries this with it.
+accelerate>=0.26.0
 f5-tts>=1.1.0
 seed-vc>=0.4.0
 soundfile>=0.12.0