ocr: surya should ask who serves the model, not where the GPU is (v0.3.7)

OCR on the Digesta install answered 503: "vLLM isolated venv not found at
/cache/vllm_venv". That host has no GPU at all — renting A40 pods is its whole
job — so the venv it was told to build could never have run, while the Surya-2
checkpoint it needed was already a configured model (backend: runpod,
engine: coderai-vllm) nothing in the OCR path could ask for.

The model layer was never the problem; `ocr.surya_serve` was asking the wrong
question. It enumerated server PROCESSES — torch in a local venv, our own vLLM,
a URL typed in by hand — when the answer the operator had already written down
was a MODEL. olmOCR has had that bridge since it shipped ("model" mode, through
our own chat path); Surya never got one, because Surya2 drives its server
directly with guided-JSON decoding and logprobs that a proxy hop would drop.

So it gets the same bridge, in the only shape it can take: resolve the model
through whichever backend owns it and hand the worker the address. Three values,
because Surya refuses a server whose /v1/models disagrees with the checkpoint it
expects, and a pod URL is useless without its token — which only Surya's vLLM
client reads, so a bearer-locked endpoint is always attached to as "vllm".

The address is a lease, not a location: a min_pods=0 pod is reaped when OCR goes
quiet, and the page that finds it gone re-resolves once — renting a fresh one —
instead of every later page failing against a dead URL. A failure against a
server that still answers /health is a real error and is reported as one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.6"
+__version__ = "0.3.7"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -4139,6 +4139,7 @@ def build_settings_dict(c, gpu_cards):
             "surya_auto_build": c.ocr.surya_auto_build,
             "surya_serve": c.ocr.surya_serve,
             "surya_model": c.ocr.surya_model,
+            "surya_model_id": c.ocr.surya_model_id,
             "surya_server_url": c.ocr.surya_server_url,
             "build_retry_cooldown_s": c.ocr.build_retry_cooldown_s,
             "evict_drain_timeout_s": c.ocr.evict_drain_timeout_s,
@@ -4936,6 +4937,7 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
         if "surya_auto_build" in d: o.surya_auto_build = bool(d["surya_auto_build"])
         if "surya_serve" in d: o.surya_serve = (d.get("surya_serve") or "local").strip()
         if "surya_model" in d: o.surya_model = (d.get("surya_model") or "").strip()
+        if "surya_model_id" in d: o.surya_model_id = (d.get("surya_model_id") or "").strip()
         if "surya_server_url" in d: o.surya_server_url = (d.get("surya_server_url") or "").strip()
         if "build_retry_cooldown_s" in d: o.build_retry_cooldown_s = _f("build_retry_cooldown_s", o.build_retry_cooldown_s, 0.0)
         if "vlm_gpu_memory_utilization" in d: o.vlm_gpu_memory_utilization = _f("vlm_gpu_memory_utilization", o.vlm_gpu_memory_utilization, 0.0)

codai/admin/templates/settings.html

@@ -1189,6 +1189,20 @@
       <label style="display:flex;align-items:center;gap:.4rem;cursor:pointer"><input type="checkbox" id="s-ocr-surya-auto-build"><span>Auto-build on first use</span></label>
     </div>
     <div style="display:flex;gap:.5rem;align-items:center;flex-wrap:wrap"><button type="button" class="btn btn-sm" onclick="ocrBuildVenv('surya')">Build Surya venv now</button><span id="s-ocr-surya-build-status" class="form-hint"></span></div>
+    <p class="form-hint" style="margin:.5rem 0 .3rem"><b>Serving.</b> <i>Local</i> runs detection+recognition on torch in that venv (surya-ocr &le;0.17). The others serve the <b>Surya2</b> VLM from an OpenAI server Surya attaches to: <i>Model</i> uses a model from this install's catalogue — a RunPod pod rented on demand, a <code>service_url</code> elsewhere, or the local vLLM backend — and is the only one that works with <b>no GPU on this box</b>; <i>vLLM</i> serves the checkpoint from coderai's own isolated vLLM venv (needs CUDA here); <i>External server</i> attaches to a URL you already run.</p>
+    <div class="form-row" style="display:flex;gap:1rem;flex-wrap:wrap;align-items:center">
+      <div class="form-group" style="flex:1;min-width:150px"><label class="form-label">Serve</label>
+        <select id="s-ocr-surya-serve" class="form-input">
+          <option value="local">Local (torch, this box)</option>
+          <option value="model">Model (served by this install)</option>
+          <option value="vllm">vLLM (own venv, CUDA)</option>
+          <option value="llamacpp">External server (URL)</option>
+        </select>
+      </div>
+      <div class="form-group" style="flex:2;min-width:200px"><label class="form-label">Surya2 checkpoint</label><input type="text" id="s-ocr-surya-model" class="form-input" placeholder="datalab-to/surya-ocr-2"></div>
+      <div class="form-group" style="flex:2;min-width:200px"><label class="form-label">Model id (Model mode)</label><input type="text" id="s-ocr-surya-model-id" class="form-input" placeholder="datalab-to/surya-ocr-2"></div>
+      <div class="form-group" style="flex:2;min-width:200px"><label class="form-label">Server URL (External)</label><input type="text" id="s-ocr-surya-server-url" class="form-input" placeholder="http://127.0.0.1:8080/v1"></div>
+    </div>
 
     <div class="card-title" style="font-size:13px;margin-top:.4rem">olmOCR-2 <span class="muted" style="font-weight:400">(document VLM)</span></div>
     <p class="form-hint" style="margin:0 0 .4rem"><a href="https://huggingface.co/allenai/olmOCR-2-7B-1025" target="_blank" rel="noopener">allenai/olmOCR-2</a> (Apache-2.0) reads a whole page at once — equations as LaTeX, tables as HTML, reading order sorted — and is the one to reach for on old scans, maths and multi-column pages. It is a <b>vision model</b>, so it returns <b>no bounding boxes</b> and no layout regions: use paddle or Surya when you need those.</p>
@@ -2119,6 +2133,10 @@ async function loadSettings(){
     document.getElementById('s-ocr-surya-langs').value = ocr.surya_langs ?? 'it';
     document.getElementById('s-ocr-surya-venv').value = ocr.surya_venv ?? '';
     document.getElementById('s-ocr-surya-auto-build').checked = !!ocr.surya_auto_build;
+    document.getElementById('s-ocr-surya-serve').value = ocr.surya_serve || 'local';
+    document.getElementById('s-ocr-surya-model').value = ocr.surya_model ?? '';
+    document.getElementById('s-ocr-surya-model-id').value = ocr.surya_model_id ?? '';
+    document.getElementById('s-ocr-surya-server-url').value = ocr.surya_server_url ?? '';
     document.getElementById('s-ocr-olmocr-enabled').checked = !!ocr.olmocr_enabled;
     document.getElementById('s-ocr-olmocr-serve').value = ocr.olmocr_serve ?? 'model';
     document.getElementById('s-ocr-olmocr-instances').value = ocr.olmocr_instances ?? 1;
@@ -2367,6 +2385,10 @@ async function saveSettings(silent){
       surya_langs: document.getElementById('s-ocr-surya-langs').value.trim() || 'it',
       surya_venv: document.getElementById('s-ocr-surya-venv').value.trim(),
       surya_auto_build: document.getElementById('s-ocr-surya-auto-build').checked,
+      surya_serve: document.getElementById('s-ocr-surya-serve').value,
+      surya_model: document.getElementById('s-ocr-surya-model').value.trim(),
+      surya_model_id: document.getElementById('s-ocr-surya-model-id').value.trim(),
+      surya_server_url: document.getElementById('s-ocr-surya-server-url').value.trim(),
       olmocr_enabled: document.getElementById('s-ocr-olmocr-enabled').checked,
       olmocr_serve: document.getElementById('s-ocr-olmocr-serve').value || 'model',
       olmocr_instances: parseInt(document.getElementById('s-ocr-olmocr-instances').value) || 1,

codai/backends/runpod.py

@@ -157,6 +157,27 @@ class RunpodBackend(ModelBackend):
         self._last_pod = pod
         return url.rstrip("/") + "/v1", (lambda: self._pool.release(pod))
 
+    def openai_endpoint(self):
+        """``(base_url ending /v1, api_key)`` for a client that will speak to this
+        model over HTTP ITSELF, instead of going through us.
+
+        The OCR VLM engines work that way: Surya and olmOCR attach to an OpenAI server
+        and drive it with their own prompts, guided-decoding schemas and logprobs — all
+        things a proxy hop would quietly drop. So they get the address and the key.
+
+        Provisioning happens here (a cold pool rents and boots a pod before returning),
+        but the in-flight slot is released at once: a long-lived attachment must not
+        hold a scaling slot open, and the pod's own idle reaper decides when it goes.
+        The address is therefore a lease — the caller re-resolves when it stops
+        answering rather than treating it as permanent."""
+        base, release = self._acquire_endpoint()
+        try:
+            auth = (self._headers or {}).get("Authorization", "")
+            key = auth[7:].strip() if auth[:7].lower() == "bearer " else ""
+            return base, key
+        finally:
+            release()
+
     def _unreachable(self) -> None:
         """The pod refused the connection: drop it and let the caller retry.
 

codai/config.py

@@ -966,12 +966,19 @@ class OcrConfig:
     surya_venv: str = ""               # isolated venv dir; blank = ~/.coderai/surya_venv
     surya_auto_build: bool = False     # create the venv + pip install requirements-surya.txt on first use
     # Surya serving mode: "local" = classic det+recognition on torch in the isolated venv
-    # (surya-ocr <=0.17); "vllm"/"llamacpp" = the latest "Surya2" VLM served by an external
-    # OpenAI server that Surya attaches to (SURYA_INFERENCE_URL). "vllm" reuses coderai's
-    # vLLM backend to serve `surya_model` (continuous batching); "llamacpp" attaches to a
-    # llama.cpp server at `surya_server_url`.
-    surya_serve: str = "local"         # local | vllm | llamacpp
+    # (surya-ocr <=0.17); the rest serve the latest "Surya2" VLM from an OpenAI server that
+    # Surya attaches to (SURYA_INFERENCE_URL), which is where the work then happens:
+    #   "model"    — the VLM is a MODEL in models.json (`surya_model_id`), served however
+    #                that entry says: a RunPod pod this install rents on demand, another
+    #                server via service_url, or the local vLLM backend. The only served
+    #                mode that works with no GPU on this box — which is precisely the box
+    #                whose job is to rent one.
+    #   "vllm"     — coderai's OWN isolated vLLM venv serves `surya_model` here (CUDA).
+    #   "llamacpp" — attach to a llama.cpp/OpenAI server already running, at
+    #                `surya_server_url`. ("server" is accepted as a synonym.)
+    surya_serve: str = "local"         # local | model | vllm | llamacpp
     surya_model: str = "datalab-to/surya-ocr-2"   # HF checkpoint for the served (vllm) backend
+    surya_model_id: str = ""           # ("model" mode) the models.json id serving Surya2
     surya_server_url: str = ""         # external OpenAI server URL (llamacpp/manual); blank = auto
 
     # --- olmOCR-2 (AllenAI, Apache-2.0) --- a document VLM: whole-page transcription in
@@ -1606,6 +1613,7 @@ class ConfigManager:
                 "surya_auto_build": self.config.ocr.surya_auto_build,
                 "surya_serve": self.config.ocr.surya_serve,
                 "surya_model": self.config.ocr.surya_model,
+                "surya_model_id": self.config.ocr.surya_model_id,
                 "surya_server_url": self.config.ocr.surya_server_url,
                 "build_retry_cooldown_s": self.config.ocr.build_retry_cooldown_s,
                 "evict_drain_timeout_s": self.config.ocr.evict_drain_timeout_s,
@@ -1726,6 +1734,8 @@ class ConfigManager:
                                 ("CODERAI_OCR_OLMOCR_MODEL_ID", "olmocr_model_id"),
                                 ("CODERAI_OCR_OLMOCR_SERVER_URL", "olmocr_server_url"),
                                 ("CODERAI_OCR_SURYA_SERVE", "surya_serve"),
+                                ("CODERAI_OCR_SURYA_MODEL", "surya_model"),
+                                ("CODERAI_OCR_SURYA_MODEL_ID", "surya_model_id"),
                                 ("CODERAI_OCR_SURYA_SERVER_URL", "surya_server_url"),
                                 ("CODERAI_OCR_FALLBACK_ENGINES", "fallback_engines")):
                 _val = (_os.environ.get(_env) or "").strip()

codai/frontproxy/app.py

@@ -1180,8 +1180,8 @@ class FrontProxy:
             local.append(True)
         if getattr(o, "surya_enabled", False) and getattr(o, "surya_accept_license", False):
             mode = (getattr(o, "surya_serve", "local") or "local").strip().lower()
-            external = mode == "llamacpp" and bool(
-                (getattr(o, "surya_server_url", "") or "").strip())
+            external = mode == "model" or (mode in ("llamacpp", "server") and bool(
+                (getattr(o, "surya_server_url", "") or "").strip()))
             local.append(not external)
         if getattr(o, "olmocr_enabled", False):
             mode = (getattr(o, "olmocr_serve", "model") or "model").strip().lower()

codai/ocr/served_model.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+# GPLv3 - see the project LICENSE.
+
+"""An OpenAI endpoint for a model coderai already serves, wherever it is serving it.
+
+Surya-2 and olmOCR are VLMs: the OCR engine is a thin client and the actual work
+happens in a model server. Until now an install could point an OCR engine at coderai's
+OWN vLLM (``ocr.*_serve = "vllm"``, an isolated venv on this box) or at a URL typed in
+by hand (``"llamacpp"``/``"server"``). Both assume the model server is somewhere the
+operator can name.
+
+On a GPU-less orchestrator neither holds. The Digesta install is the case that found
+it: ``ru002848`` has no GPU at all — coderai's whole job there is to rent A40 pods on
+demand — and the OCR VLM is already a first-class model in ``models.json`` with
+``backend: runpod``. Asking for OCR built a local vLLM venv that could never run,
+failed with ``NO_VLLM_VENV``, and the document went unread, while the pod that would
+have served it in sixty seconds was never asked for.
+
+So this module answers one question — *where is model X being served, and with what
+key* — and lets whichever backend owns the model answer it: a RunPod pod pool
+provisions one and reports its proxy URL, a model pinned to another coderai reports
+its ``service_url``, a local vLLM model reports its own instance. The OCR engine then
+attaches over HTTP exactly as it would to a hand-typed URL, and the renting, the
+scaling, the budget ledger and the idle reaper stay where they belong.
+
+The endpoint is a lease, not an address: a scale-to-zero pod is reaped when OCR goes
+quiet, and the caller is expected to re-resolve rather than to trust it forever.
+"""
+
+import threading
+
+from codai.ocr.base import OcrError
+
+_lock = threading.Lock()
+_backends: dict = {}            # model id -> a loaded backend that knows its endpoint
+
+
+def _entry_backend(model_id: str):
+    """A loaded backend for ``model_id``, or an OcrError saying why there is none.
+
+    Only the backends that END UP somewhere addressable can answer: a model loaded
+    into this process has no URL, and inventing a loopback to our own ``/v1`` would
+    hide a misconfiguration behind an extra hop.
+    """
+    from codai.models import manager as mm
+
+    try:
+        from codai.backends import remote_openai
+        if remote_openai.remote_should_handle(model_id):
+            backend = remote_openai.RemoteOpenAIBackend()
+            backend.load_model(model_id)
+            return backend
+    except OcrError:
+        raise
+    except Exception as exc:
+        raise OcrError(f"model {model_id!r}: remote endpoint unusable: {exc}", status=503)
+
+    try:
+        if mm.runpod_should_handle(model_id):
+            from codai.backends.runpod import RunpodBackend
+            backend = RunpodBackend(mm.get_active_runpod_config())
+            backend.load_model(model_id)
+            return backend
+    except Exception as exc:
+        raise OcrError(f"model {model_id!r}: RunPod endpoint unusable: {exc}", status=503)
+
+    if mm.resolve_engine_backend(model_id) == "vllm":
+        try:
+            from codai.api import runpod_worker, vllm_worker
+            from codai.models.manager import get_active_vllm_config
+            vcfg = get_active_vllm_config()
+            path = (runpod_worker._model_path_for(model_id) or "").strip() or model_id
+            base = vllm_worker.ensure_service(vcfg, model_path=path, served_name=model_id)
+            return _Fixed(base.rstrip("/") + "/v1", "", model_id)
+        except Exception as exc:
+            raise OcrError(f"model {model_id!r}: local vLLM unusable: {exc}", status=503)
+
+    raise OcrError(
+        f"model {model_id!r} is not served over HTTP by this install, so an OCR engine "
+        f"cannot attach to it. Pin the model to a backend that has an endpoint "
+        f"(backend: \"runpod\", or a service_url pointing at a server that holds it), "
+        f"or pick a serve mode that fits this box.",
+        status=503,
+    )
+
+
+class _Fixed:
+    """A backend-shaped answer for the cases that resolve to one fixed URL."""
+
+    def __init__(self, url: str, key: str, served: str):
+        self._url, self._key, self._served = url, key, served
+
+    def openai_endpoint(self):
+        return self._url, self._key
+
+    @property
+    def served_name(self) -> str:
+        return self._served
+
+
+def _served_name_of(backend, model_id: str) -> str:
+    for attr in ("served_name", "_served_name"):
+        name = getattr(backend, attr, "")
+        if isinstance(name, str) and name.strip():
+            return name.strip()
+    return model_id
+
+
+def endpoint_for(model_id: str):
+    """``(base_url ending /v1, api_key, served_name)`` for ``model_id``.
+
+    Blocks while a cold pool rents and boots a pod — a first OCR request after an idle
+    night legitimately waits for a GPU to exist. Raises :class:`OcrError` (503) with a
+    message that says what to configure, because that error is what reaches the client.
+    """
+    model_id = (model_id or "").strip()
+    if not model_id:
+        raise OcrError("no model id configured for the served OCR engine", status=400)
+
+    with _lock:
+        backend = _backends.get(model_id)
+    if backend is None:
+        backend = _entry_backend(model_id)
+        with _lock:
+            _backends.setdefault(model_id, backend)
+            backend = _backends[model_id]
+
+    try:
+        base, key = backend.openai_endpoint()
+    except OcrError:
+        raise
+    except Exception as exc:
+        raise OcrError(f"model {model_id!r}: no endpoint available ({exc})", status=503)
+    if not base:
+        raise OcrError(f"model {model_id!r}: backend reported no endpoint", status=503)
+    return base, key, _served_name_of(backend, model_id)
+
+
+def forget(model_id: str = "") -> None:
+    """Drop cached backends so the next resolve starts over (config changed, or a
+    pool that no longer answers). A pool itself is global to the process, so this
+    costs nothing but a re-``load_model``."""
+    with _lock:
+        if model_id:
+            _backends.pop(model_id.strip(), None)
+        else:
+            _backends.clear()
+
+
+def health_url(base: str) -> str:
+    """The ``/health`` base an attached client probes: the OpenAI base without ``/v1``.
+
+    Surya's attach path refuses a URL whose ``/health`` does not answer 200, which is
+    also the cheapest way for us to find out that a leased pod has gone away.
+    """
+    base = (base or "").rstrip("/")
+    return base[:-len("/v1")].rstrip("/") if base.endswith("/v1") else base

codai/ocr/surya.py

@@ -24,6 +24,12 @@ class SuryaEngine(SubprocessOcrEngine):
     name = "surya"
     _worker_engine = "surya"
 
+    # Where the recognition VLM is, what the server calls it, and the token that
+    # opens it. Empty in the local mode, and resolved by load() in the served ones.
+    _server_url = ""
+    _server_key = ""
+    _served_name = ""
+
     def _venv_dir(self) -> str:
         return _resolve_venv_dir(self.cfg.surya_venv, "surya_venv", ("surya",))
 
@@ -36,19 +42,39 @@ class SuryaEngine(SubprocessOcrEngine):
     def _worker_opts(self) -> dict:
         return {"langs": self.cfg.surya_langs or "it"}
 
+    #: Modes where the recognition VLM lives in a model server we only talk to.
+    SERVED_MODES = ("model", "vllm", "llamacpp", "server")
+
     def _serve_mode(self) -> str:
-        return (getattr(self.cfg, "surya_serve", "local") or "local").strip().lower()
+        mode = (getattr(self.cfg, "surya_serve", "local") or "local").strip().lower()
+        return "llamacpp" if mode == "server" else mode
 
     def _worker_env(self) -> dict:
         """For the served ("Surya2") modes, tell the worker to attach to an external
-        OpenAI server instead of running detection+recognition locally."""
+        OpenAI server instead of running detection+recognition locally.
+
+        Surya reads these from its own settings, so the handshake is three values and
+        no patching: which client to use, where it is, and what the server calls the
+        model — it refuses to attach to a server whose /v1/models disagrees with the
+        checkpoint it expects. The key matters as much: a pod URL is reachable by
+        anyone who learns it, so ours are launched bearer-locked, and Surya's vLLM
+        client takes that token from VLLM_API_KEY (its llama.cpp client hardcodes
+        "EMPTY", which is why a locked endpoint is always attached to as "vllm")."""
         mode = self._serve_mode()
-        if mode not in ("vllm", "llamacpp"):
+        if mode not in self.SERVED_MODES:
             return {}
         url = self._server_url
         if not url:
             return {}
-        return {"SURYA_INFERENCE_BACKEND": mode, "SURYA_INFERENCE_URL": url}
+        env = {
+            "SURYA_INFERENCE_BACKEND": "llamacpp" if mode == "llamacpp" else "vllm",
+            "SURYA_INFERENCE_URL": url,
+        }
+        if self._served_name:
+            env["SURYA_MODEL_CHECKPOINT"] = self._served_name
+        if self._server_key:
+            env["VLLM_API_KEY"] = self._server_key
+        return env
 
     def prelaunch_vram_gb(self) -> float:
         """In the served modes Surya rides a vLLM instance that grabs
@@ -83,7 +109,7 @@ class SuryaEngine(SubprocessOcrEngine):
         nil, and the instance's real cost is reported once through
         :meth:`prelaunch_vram_gb` / the vLLM stop estimate. Counting a whole GB per
         instance here asked to free 24 GB on a 24-instance pool."""
-        return 0.05 if self._serve_mode() in ("vllm", "llamacpp") else 1.0
+        return 0.05 if self._serve_mode() in self.SERVED_MODES else 1.0
 
     def load(self) -> None:
         if not self.cfg.surya_accept_license:
@@ -92,8 +118,28 @@ class SuryaEngine(SubprocessOcrEngine):
                 status=400,
             )
         self._server_url = ""
+        self._server_key = ""
+        self._served_name = ""
         mode = self._serve_mode()
-        if mode == "vllm":
+        if mode == "model":
+            # The recognition VLM is a MODEL this install already serves — on the
+            # Digesta orchestrator a RunPod A40 pod it rents on demand. Ask whichever
+            # backend owns it where it ended up, and attach there. This is the only
+            # served mode that works on a box with no GPU of its own, which is also
+            # the box whose entire job is to rent one.
+            from codai.ocr.served_model import endpoint_for
+            model_id = ((getattr(self.cfg, "surya_model_id", "") or "").strip()
+                        or (getattr(self.cfg, "surya_model", "") or "").strip())
+            if not model_id:
+                raise OcrError(
+                    "Surya 'model' mode needs ocr.surya_model_id — the id of the "
+                    "Surya-2 checkpoint in models.json (it must be served over HTTP: "
+                    "backend \"runpod\", a service_url, or the local vLLM backend).",
+                    status=400)
+            self._server_url, self._server_key, self._served_name = endpoint_for(model_id)
+            print(f"[ocr] surya attaches to {self._server_url} "
+                  f"(served as {self._served_name!r})", flush=True)
+        elif mode == "vllm":
             # Serve the Surya2 VLM checkpoint via coderai's vLLM backend (continuous
             # batching) and attach Surya to it. Requires vllm enabled/auto-buildable.
             from codai.api import vllm_worker
@@ -106,6 +152,7 @@ class SuryaEngine(SubprocessOcrEngine):
                                               gpu_memory_utilization=self._vlm_gmu(),
                                               **vlm_serve_limits(self.cfg))
             self._server_url = base.rstrip("/") + "/v1"
+            self._served_name = model
         elif mode == "llamacpp":
             url = (getattr(self.cfg, "surya_server_url", "") or "").strip()
             if not url:
@@ -114,3 +161,38 @@ class SuryaEngine(SubprocessOcrEngine):
                     "OpenAI endpoint).", status=400)
             self._server_url = url
         super().load()
+
+    # ------------------------------------------------------------------ leases
+
+    def _endpoint_alive(self) -> bool:
+        """Whether the server this worker was pinned to still answers /health."""
+        import requests
+        from codai.ocr.served_model import health_url
+        try:
+            r = requests.get(health_url(self._server_url) + "/health", timeout=5)
+            return r.status_code == 200
+        except Exception:
+            return False
+
+    def recognize_image(self, image):
+        """One page, re-resolving the endpoint if the lease expired underneath us.
+
+        In 'model' mode the address is a rented pod, and a pod that has been idle for
+        its timeout is reaped — correctly, it costs money. The worker is then pinned to
+        a URL nothing answers, and would fail every page from then on. So a failure is
+        checked against the endpoint itself: if it is still healthy the error is real
+        and is reported, and if it is gone the worker is restarted against a freshly
+        provisioned one. Exactly once — a second failure is an outage, not a lease."""
+        try:
+            return super().recognize_image(image)
+        except OcrError as first:
+            if self._serve_mode() != "model" or self._endpoint_alive():
+                raise
+            print(f"[ocr] surya endpoint {self._server_url} is gone "
+                  f"({first}) — re-resolving", flush=True)
+            from codai.ocr import served_model
+            served_model.forget()
+            self.cleanup()
+            self.load()
+            self._loaded = True
+            return super().recognize_image(image)

docs/ocr.md

@@ -63,15 +63,46 @@ matches the chosen serving mode:
 
 | `surya_serve` | What it is | surya‑ocr version | How it runs |
 |---|---|---|---|
-| `local` (default) | classic detection + recognition on torch | **≤ 0.17.1** (`requirements-surya.txt` pins 0.17.1) | in the isolated venv, GPU |
-| `vllm` | the latest **"Surya2"** VLM (Qwen3.5‑VL, `datalab-to/surya-ocr-2`) | **≥ 0.20** (e.g. 0.22.1) | served by coderai's **vLLM backend**; Surya attaches via `SURYA_INFERENCE_URL` |
-| `llamacpp` | same Surya2 VLM | ≥ 0.20 | attaches to a `llama-server` at `ocr.surya_server_url` |
-
-Surya2 (≥0.20) is a VLM that CANNOT run standalone in the venv — it requires an external
-OpenAI server (vLLM/llama‑server). `vllm` mode is recommended: coderai serves
-`ocr.surya_model` through the vLLM backend (continuous batching) and points Surya at it.
-So: install **surya‑ocr 0.17.1** for `local`, or **surya‑ocr>=0.20** for `vllm`/`llamacpp` —
-one per venv.
+| `local` (default) | classic detection + recognition on torch | **≤ 0.17.1** | in the isolated venv, on a GPU of this box |
+| `model` | the **"Surya2"** VLM as a MODEL this install serves (`ocr.surya_model_id` → `models.json`) | **≥ 0.20** (pinned 0.22.1) | wherever that entry says: a **RunPod pod rented on demand**, another host via `service_url`, or the local vLLM backend. **The only served mode that needs no GPU here.** |
+| `vllm` | the same Surya2 VLM, served from coderai's own isolated vLLM venv | ≥ 0.20 | this box, CUDA required |
+| `llamacpp` (alias `server`) | the same Surya2 VLM | ≥ 0.20 | attaches to a server you already run, at `ocr.surya_server_url` |
+
+Surya2 (≥0.20) is a VLM that CANNOT run standalone in the venv — it needs an OpenAI
+server to attach to. In the served modes the venv is therefore light: it renders the page
+and speaks HTTP, and every GPU second is spent on the other side.
+
+### `model` mode: Surya on whatever serves the model
+
+Surya is a *client* — layout, reading order, guided-JSON decoding and logprobs are its
+own, which is why it drives a model server directly instead of going back through
+coderai's `/v1` (one proxy hop would drop the guided decoding and the logprobs with it).
+What it needs from coderai is therefore an address, and `model` mode is where it asks for
+one in the vocabulary the rest of the server already speaks:
+
+```jsonc
+// models.json — the VLM is an ordinary model entry, served like any other
+"datalab-to/surya-ocr-2": { "backend": "runpod", "engine": "coderai-vllm", … }
+```
+```jsonc
+// config.json
+"ocr": { "surya_serve": "model", "surya_model_id": "datalab-to/surya-ocr-2" }
+```
+
+coderai resolves that id through the backend that owns it — provisioning a pod if the
+pool is cold — and hands the worker `SURYA_INFERENCE_URL`, `SURYA_MODEL_CHECKPOINT` (the
+served name; Surya refuses a server whose `/v1/models` disagrees) and `VLLM_API_KEY` (pods
+are bearer-locked). Renting, autoscaling, the spend ledger and the idle reaper stay with
+the model layer; the OCR side holds only a lease. When a scale-to-zero pod is reaped
+between documents, the next page finds the endpoint unhealthy, re-resolves once — which
+rents a fresh pod — and is read rather than lost.
+
+This is the mode for a **GPU-less orchestrator**: an install whose job is to rent cards
+has no card of its own, so `vllm` mode there can only ever fail (`NO_VLLM_VENV`), and a
+URL pasted into `llamacpp` mode stops existing fifteen minutes after the last page.
+
+So: install **surya‑ocr 0.17.1** for `local`, or **surya‑ocr>=0.20** for
+`model`/`vllm`/`llamacpp` — one per venv.
 Or set `paddle_auto_build` / `surya_auto_build` to have coderai create the venv and install
 on first use. In the OCI image these venvs are **baked in** at `/opt/coderai/paddle_venv` /
 `/opt/coderai/surya_venv` (same as the `lipsync_venv` / `parler-venv`), so the image is

requirements-surya.txt

@@ -21,7 +21,13 @@
 # det+recognition locally on torch (GPU), so it's the classic pin. See docs/ocr.md.
 surya-ocr==0.22.1
 pypdfium2>=4.20.0
-# NOTE: 0.17.1 is for ocr.surya_serve="local" (classic det+recognition on torch, runs in
-# THIS venv). For ocr.surya_serve="vllm"/"llamacpp" (the latest "Surya2" VLM, Qwen3.5-VL
-# `datalab-to/surya-ocr-2`), install surya-ocr>=0.20 instead (e.g. 0.22.1) — it needs an
-# external vLLM/llama-server backend, not local torch. One version per venv; see docs/ocr.md.
+# NOTE: ONE VERSION PER VENV, and the mode decides which.
+#   ocr.surya_serve="local"  -> surya-ocr==0.17.1, the last release that runs
+#                               det+recognition locally on torch (needs a GPU here).
+#   ocr.surya_serve="model" / "vllm" / "llamacpp"  -> surya-ocr>=0.20 (pinned 0.22.1
+#                               above): the "Surya2" VLM (`datalab-to/surya-ocr-2`),
+#                               which this venv only TALKS to over HTTP. That makes the
+#                               venv light — PIL, httpx and an OpenAI client do the work
+#                               — and is what lets a GPU-less install serve Surya at all
+#                               ("model" mode: the VLM is a models.json entry, e.g. a
+#                               RunPod pod rented on demand). See docs/ocr.md.

tests/test_ocr_served_model.py

+"""Surya attached to the model layer instead of to a GPU this box may not have.
+
+On 2026-10-11 OCR on the Digesta install (`ru002848`) answered HTTP 503: "OCR engine
+'surya' failed to load: vLLM isolated venv not found at /cache/vllm_venv". The host has
+no GPU at all — renting A40 pods on RunPod is that install's entire job — so the venv it
+was asked to build could never have run, and the Surya-2 checkpoint it needed was already
+a configured model (`backend: runpod`, `engine: coderai-vllm`) that the OCR path had no
+way to ask for.
+
+The OCR config was asking the wrong question. `surya_serve` enumerated server PROCESSES
+(a local torch venv, our own vLLM, a URL typed in by hand) when the answer the operator
+already wrote down was a MODEL. `model` mode asks the question the rest of the server
+answers for everything else: where is this model served, and with what key.
+
+What these tests hold:
+
+* the resolver delegates — it does not invent a second path to a GPU, and a model nothing
+  serves over HTTP is a clear 503, not a silent loopback;
+* an endpoint is a LEASE: a reaped pod costs one re-resolve, not every page after it;
+* a pod's bearer token reaches Surya, through the one setting Surya's vLLM client reads;
+* the served modes claim no local VRAM, because they hold none.
+"""
+
+import pathlib
+import sys
+import types
+
+import pytest
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+SURYA_SRC = (ROOT / "codai/ocr/surya.py").read_text()
+SERVED_SRC = (ROOT / "codai/ocr/served_model.py").read_text()
+CONFIG_SRC = (ROOT / "codai/config.py").read_text()
+SETTINGS_HTML = (ROOT / "codai/admin/templates/settings.html").read_text()
+ROUTES_SRC = (ROOT / "codai/admin/routes.py").read_text()
+
+from codai.ocr.base import OcrError                       # noqa: E402
+from codai.ocr import served_model                        # noqa: E402
+from codai.ocr.surya import SuryaEngine                   # noqa: E402
+
+
+class Cfg:
+    """Just the OCR fields these paths read."""
+
+    def __init__(self, **kw):
+        self.surya_accept_license = True
+        self.surya_serve = "model"
+        self.surya_model = "datalab-to/surya-ocr-2"
+        self.surya_model_id = ""
+        self.surya_server_url = ""
+        self.surya_venv = ""
+        self.surya_langs = "it"
+        self.surya_auto_build = False
+        self.vlm_gpu_memory_utilization = 0.0
+        self.__dict__.update(kw)
+
+
+class FakeBackend:
+    """A backend that knows where its model is being served."""
+
+    def __init__(self, url="https://pod-8000.proxy.runpod.net/v1", key="podtoken",
+                 served="datalab-to/surya-ocr-2"):
+        self.url, self.key, self._served_name = url, key, served
+        self.calls = 0
+
+    def openai_endpoint(self):
+        self.calls += 1
+        return self.url, self.key
+
+
+@pytest.fixture(autouse=True)
+def _clean_cache():
+    served_model.forget()
+    yield
+    served_model.forget()
+
+
+# ------------------------------------------------------------------ the resolver
+def test_the_endpoint_comes_from_whichever_backend_owns_the_model(monkeypatch):
+    """The resolver's whole job is delegation: it must not know what a pod is."""
+    backend = FakeBackend()
+    monkeypatch.setattr(served_model, "_entry_backend", lambda mid: backend)
+    base, key, served = served_model.endpoint_for("datalab-to/surya-ocr-2")
+    assert base == "https://pod-8000.proxy.runpod.net/v1"
+    assert key == "podtoken"
+    assert served == "datalab-to/surya-ocr-2"
+
+
+def test_the_backend_is_resolved_once_not_per_page(monkeypatch):
+    """Rebuilding a pod pool per page would re-read config, re-register the pool and
+    log a rental decision for every page of a 400-page file."""
+    built = []
+
+    def _build(mid):
+        built.append(mid)
+        return FakeBackend()
+
+    monkeypatch.setattr(served_model, "_entry_backend", _build)
+    for _ in range(5):
+        served_model.endpoint_for("m")
+    assert built == ["m"]
+
+
+def test_a_model_nobody_serves_over_http_says_so(monkeypatch):
+    """The alternative is a loopback to our own /v1, which would hide the
+    misconfiguration behind an extra hop and drop Surya's guided decoding."""
+    mod = types.SimpleNamespace(
+        runpod_should_handle=lambda m: False,
+        resolve_engine_backend=lambda m: None,
+        get_active_runpod_config=lambda: None,
+    )
+    monkeypatch.setitem(sys.modules, "codai.models.manager", mod)
+    with pytest.raises(OcrError) as caught:
+        served_model.endpoint_for("local-only-model")
+    assert caught.value.status == 503
+    assert "not served over HTTP" in str(caught.value)
+
+
+def test_an_empty_model_id_is_a_configuration_error_not_a_lookup(monkeypatch):
+    with pytest.raises(OcrError) as caught:
+        served_model.endpoint_for("   ")
+    assert caught.value.status == 400
+
+
+def test_forget_lets_a_config_change_take_effect(monkeypatch):
+    built = []
+    monkeypatch.setattr(served_model, "_entry_backend",
+                        lambda mid: built.append(mid) or FakeBackend())
+    served_model.endpoint_for("m")
+    served_model.forget("m")
+    served_model.endpoint_for("m")
+    assert built == ["m", "m"]
+
+
+def test_the_health_base_drops_the_openai_suffix():
+    """Surya probes /health on the base WITHOUT /v1, and refuses to attach when it
+    does not answer 200 — so getting this wrong fails every served mode."""
+    assert served_model.health_url("http://h:8000/v1") == "http://h:8000"
+    assert served_model.health_url("http://h:8000/v1/") == "http://h:8000"
+    assert served_model.health_url("http://h:8000") == "http://h:8000"
+
+
+def test_the_resolver_delegates_rather_than_reimplementing_a_backend():
+    """A copy of the pod-rental logic here would drift from the real one."""
+    for forbidden in ("runpod_client", "graphql", "_provision", "gpu_types"):
+        assert forbidden not in SERVED_SRC
+
+
+# ------------------------------------------------------------------- the engine
+def test_model_mode_hands_surya_the_url_the_name_and_the_key(monkeypatch):
+    """Three values, because Surya refuses a server whose /v1/models disagrees with
+    the checkpoint it expects, and a pod URL is useless without its token."""
+    eng = SuryaEngine(Cfg(surya_model_id="datalab-to/surya-ocr-2"))
+    eng._server_url = "https://pod-8000.proxy.runpod.net/v1"
+    eng._server_key = "podtoken"
+    eng._served_name = "datalab-to/surya-ocr-2"
+    env = eng._worker_env()
+    assert env["SURYA_INFERENCE_URL"] == "https://pod-8000.proxy.runpod.net/v1"
+    assert env["SURYA_MODEL_CHECKPOINT"] == "datalab-to/surya-ocr-2"
+    assert env["VLLM_API_KEY"] == "podtoken"
+
+
+def test_a_locked_endpoint_is_attached_to_as_vllm():
+    """Surya's llama.cpp client hardcodes api_key="EMPTY"; only its vLLM client reads
+    VLLM_API_KEY. Attaching to a bearer-locked pod as "llamacpp" would 401 on every
+    page, so the backend name is not simply the mode name."""
+    eng = SuryaEngine(Cfg(surya_model_id="m"))
+    eng._server_url, eng._server_key = "http://pod/v1", "tok"
+    assert eng._worker_env()["SURYA_INFERENCE_BACKEND"] == "vllm"
+
+
+def test_the_external_server_mode_still_attaches_as_llamacpp():
+    eng = SuryaEngine(Cfg(surya_serve="llamacpp", surya_server_url="http://h/v1"))
+    eng._server_url = "http://h/v1"
+    assert eng._worker_env()["SURYA_INFERENCE_BACKEND"] == "llamacpp"
+
+
+def test_server_is_accepted_as_a_synonym_of_llamacpp():
+    """olmOCR calls this mode "server"; one subsystem, one vocabulary."""
+    assert SuryaEngine(Cfg(surya_serve="server"))._serve_mode() == "llamacpp"
+
+
+def test_model_mode_needs_a_model_id_and_says_which_setting(monkeypatch):
+    eng = SuryaEngine(Cfg(surya_model_id="", surya_model=""))
+    with pytest.raises(OcrError) as caught:
+        eng.load()
+    assert caught.value.status == 400
+    assert "surya_model_id" in str(caught.value)
+
+
+def test_the_checkpoint_setting_is_the_fallback_model_id(monkeypatch):
+    """surya_model already names the Surya2 checkpoint; an install that set only that
+    one should not have to repeat it."""
+    seen = {}
+
+    def _endpoint(mid):
+        seen["id"] = mid
+        return "http://pod/v1", "k", mid
+
+    monkeypatch.setattr(served_model, "endpoint_for", _endpoint)
+    eng = SuryaEngine(Cfg(surya_model_id="", surya_model="datalab-to/surya-ocr-2"))
+    monkeypatch.setattr(SuryaEngine.__mro__[1], "load", lambda self: None)
+    eng.load()
+    assert seen["id"] == "datalab-to/surya-ocr-2"
+
+
+def test_the_served_modes_hold_no_local_vram():
+    """A 24-instance pool once asked to free 24 GB for workers that hold nothing. The
+    pod's VRAM is the pod's, and it is accounted where it is spent."""
+    for mode in ("model", "vllm", "llamacpp", "server"):
+        assert SuryaEngine(Cfg(surya_serve=mode)).vram_gb() < 0.1
+    assert SuryaEngine(Cfg(surya_serve="local")).vram_gb() >= 1.0
+
+
+def test_model_mode_reserves_no_vram_on_this_box():
+    """vllm mode must clear room on the local card first; model mode has no card to
+    clear, and reserving against one would block OCR on a host with no GPU."""
+    assert SuryaEngine(Cfg(surya_serve="model")).prelaunch_vram_gb() == 0.0
+
+
+# -------------------------------------------------------------------- the lease
+def _engine_with_fake_worker(monkeypatch, alive, fail_times=1):
+    eng = SuryaEngine(Cfg(surya_model_id="m"))
+    eng._server_url = "http://pod/v1"
+    state = {"fails": fail_times, "loads": 0, "pages": 0}
+
+    def _recognize(self, image):
+        state["pages"] += 1
+        if state["fails"] > 0:
+            state["fails"] -= 1
+            raise OcrError("surya: worker closed the pipe", status=500)
+        return "page"
+
+    monkeypatch.setattr(SuryaEngine.__mro__[1], "recognize_image", _recognize)
+    monkeypatch.setattr(SuryaEngine, "load", lambda self: state.__setitem__("loads", state["loads"] + 1))
+    monkeypatch.setattr(SuryaEngine, "cleanup", lambda self: None)
+    monkeypatch.setattr(SuryaEngine, "_endpoint_alive", lambda self: alive)
+    return eng, state
+
+
+def test_a_reaped_pod_costs_one_re_resolve_not_the_document(monkeypatch):
+    """min_pods=0 means the pod goes away when OCR goes quiet. Without this the
+    worker stays pinned to a dead URL and every later page fails."""
+    eng, state = _engine_with_fake_worker(monkeypatch, alive=False)
+    assert eng.recognize_image(object()) == "page"
+    assert state["loads"] == 1
+
+
+def test_a_real_error_is_reported_not_retried_against_a_healthy_server(monkeypatch):
+    """Restarting the worker on every failure would double the cost of a page that
+    was always going to fail, and hide the reason."""
+    eng, state = _engine_with_fake_worker(monkeypatch, alive=True)
+    with pytest.raises(OcrError):
+        eng.recognize_image(object())
+    assert state["loads"] == 0
+
+
+def test_the_retry_happens_once(monkeypatch):
+    """A second failure is an outage, not an expired lease."""
+    eng, state = _engine_with_fake_worker(monkeypatch, alive=False, fail_times=2)
+    with pytest.raises(OcrError):
+        eng.recognize_image(object())
+    assert state["loads"] == 1
+
+
+def test_only_model_mode_re_resolves(monkeypatch):
+    """A URL the operator typed is theirs; coderai does not go looking for another."""
+    eng, state = _engine_with_fake_worker(monkeypatch, alive=False)
+    eng.cfg.surya_serve = "llamacpp"
+    with pytest.raises(OcrError):
+        eng.recognize_image(object())
+    assert state["loads"] == 0
+
+
+# ------------------------------------------------------------ what it is wired to
+def test_the_mode_is_configurable_from_the_admin_ui():
+    """A serving decision that only exists in a hand-edited config.json is not
+    configured — the standing rule is that a model is orchestrated from the UI too."""
+    assert 's-ocr-surya-serve' in SETTINGS_HTML
+    assert 's-ocr-surya-model-id' in SETTINGS_HTML
+    assert '<option value="model">' in SETTINGS_HTML
+    assert "surya_model_id: document.getElementById('s-ocr-surya-model-id')" in SETTINGS_HTML
+    assert "ocr.surya_model_id" in SETTINGS_HTML
+
+
+def test_the_admin_api_round_trips_the_new_field():
+    """Saved by the GUI, and not dropped by the whitelist on the way back in."""
+    assert '"surya_model_id": c.ocr.surya_model_id,' in ROUTES_SRC
+    assert 'if "surya_model_id" in d:' in ROUTES_SRC
+    assert '"surya_model_id": self.config.ocr.surya_model_id,' in CONFIG_SRC
+
+
+def test_a_pod_can_be_told_the_mode_through_the_environment():
+    """A rented pod gets its OCR settings from the renting install, never its own
+    config file."""
+    assert '("CODERAI_OCR_SURYA_MODEL_ID", "surya_model_id")' in CONFIG_SRC
+
+
+def test_the_incident_is_recorded_where_the_mode_is_defined():
+    """-GPU-less is the case that found this; the next reader deserves the why."""
+    assert "no GPU" in SERVED_SRC
+    assert "NO_VLLM_VENV" in SERVED_SRC