ocr: one address for the engine, every server behind it (v0.3.9)

An OCR pipeline is handed one server address and drives it itself, which is
what keeps its page images, its strict json_schema and its logprobs intact.
One address cannot spread load, and three things followed from that:

  * a pool grows on in-flight requests counted as they pass through it. An
    attached engine never passed through, so the pool saw no load however hard
    OCR ran, never grew, and the engine had one pod's URL anyway;
  * several cluster nodes serving the model resolved to whichever answered
    first, for ever, however busy it became;
  * a local model configured to burst to RunPod never did: that trigger lives
    on the proxy path an attached engine bypasses.

So the engine gets a lease on an address of ours, and codai/ocr/dispatch.py
chooses per request: a pool that is the model's configured home takes every
page (picking the least-loaded pod, renting when they are all at the
threshold); otherwise this machine and the nodes holding the model compete on
load — including the load a node reports for its whole install, so a node busy
with someone else's work is left alone — and the overflow past
ocr.serve_spill_inflight rents instead of queueing. An explicit `backend` pin
still wins over discovery: that was a decision, and load-aware picking must
not quietly repeal it.

The hop is allowed to exist only because it decides nothing: body bytes in,
body bytes out. The losses it is there to avoid came from PARSING a request
and re-emitting it through the backend abstraction.

Two gates stood in the way and neither could be satisfied the usual way: the
engine has no API token, and surya's /health and /v1/models probes send no
headers at all (bare httpx GETs), while refusing to attach if /health is not
200. The lease is therefore read from the URL, and the engine-only internal
gate exempts that path alone — tested to be no wider than that.

olmOCR's model mode asks the same dispatcher, and still takes the in-process
route when the answer is local, where the model manager's loading, eviction
and queueing belong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.8"
+__version__ = "0.3.9"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -4140,6 +4140,8 @@ def build_settings_dict(c, gpu_cards):
             "surya_serve": c.ocr.surya_serve,
             "surya_model": c.ocr.surya_model,
             "surya_model_id": c.ocr.surya_model_id,
+            "serve_fanout": c.ocr.serve_fanout,
+            "serve_spill_inflight": c.ocr.serve_spill_inflight,
             "surya_server_url": c.ocr.surya_server_url,
             "build_retry_cooldown_s": c.ocr.build_retry_cooldown_s,
             "evict_drain_timeout_s": c.ocr.evict_drain_timeout_s,
@@ -4938,6 +4940,12 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
         if "surya_serve" in d: o.surya_serve = (d.get("surya_serve") or "local").strip()
         if "surya_model" in d: o.surya_model = (d.get("surya_model") or "").strip()
         if "surya_model_id" in d: o.surya_model_id = (d.get("surya_model_id") or "").strip()
+        if "serve_fanout" in d: o.serve_fanout = bool(d.get("serve_fanout"))
+        if "serve_spill_inflight" in d:
+            try:
+                o.serve_spill_inflight = max(0, int(d.get("serve_spill_inflight") or 0))
+            except (TypeError, ValueError):
+                pass
         if "surya_server_url" in d: o.surya_server_url = (d.get("surya_server_url") or "").strip()
         if "build_retry_cooldown_s" in d: o.build_retry_cooldown_s = _f("build_retry_cooldown_s", o.build_retry_cooldown_s, 0.0)
         if "vlm_gpu_memory_utilization" in d: o.vlm_gpu_memory_utilization = _f("vlm_gpu_memory_utilization", o.vlm_gpu_memory_utilization, 0.0)

codai/admin/templates/settings.html

@@ -1204,6 +1204,13 @@
       <div class="form-group" style="flex:2;min-width:200px"><label class="form-label">Server URL (External)</label><input type="text" id="s-ocr-surya-server-url" class="form-input" placeholder="http://127.0.0.1:8080/v1"></div>
     </div>
 
+    <div class="card-title" style="font-size:13px;margin-top:.4rem">Scaling in <b>Model</b> mode <span class="muted" style="font-weight:400">(both VLM engines)</span></div>
+    <p class="form-hint" style="margin:0 0 .4rem">In <b>Model</b> mode the OCR engine drives the model server itself, which is what keeps its page images, strict JSON schema and <b>confidence scores</b> intact. <b>Spread the load</b> gives it one address here instead, forwarded per request — unchanged, byte for byte — to the least busy of: this machine's server, the cluster nodes that hold the model, and the RunPod pool. Without it the engine pins to a single pod, and the pool never sees the load so it never grows.</p>
+    <div class="form-row" style="display:flex;gap:1rem;flex-wrap:wrap;align-items:center">
+      <label style="display:flex;align-items:center;gap:.4rem;cursor:pointer"><input type="checkbox" id="s-ocr-serve-fanout"><span>Spread the load across servers</span></label>
+      <div class="form-group" style="flex:1;min-width:200px"><label class="form-label">Rent when busier than <span class="muted">(requests in flight; 0 = never on load)</span></label><input type="number" id="s-ocr-serve-spill" class="form-input" min="0" placeholder="4"></div>
+    </div>
+
     <div class="card-title" style="font-size:13px;margin-top:.4rem">olmOCR-2 <span class="muted" style="font-weight:400">(document VLM)</span></div>
     <p class="form-hint" style="margin:0 0 .4rem"><a href="https://huggingface.co/allenai/olmOCR-2-7B-1025" target="_blank" rel="noopener">allenai/olmOCR-2</a> (Apache-2.0) reads a whole page at once — equations as LaTeX, tables as HTML, reading order sorted — and is the one to reach for on old scans, maths and multi-column pages. It is a <b>vision model</b>, so it returns <b>no bounding boxes</b> and no layout regions: use paddle or Surya when you need those.</p>
     <div class="form-row" style="display:flex;gap:1rem;flex-wrap:wrap;align-items:center">
@@ -2137,6 +2144,8 @@ async function loadSettings(){
     document.getElementById('s-ocr-surya-model').value = ocr.surya_model ?? '';
     document.getElementById('s-ocr-surya-model-id').value = ocr.surya_model_id ?? '';
     document.getElementById('s-ocr-surya-server-url').value = ocr.surya_server_url ?? '';
+    document.getElementById('s-ocr-serve-fanout').checked = ocr.serve_fanout !== false;
+    document.getElementById('s-ocr-serve-spill').value = ocr.serve_spill_inflight ?? 4;
     document.getElementById('s-ocr-olmocr-enabled').checked = !!ocr.olmocr_enabled;
     document.getElementById('s-ocr-olmocr-serve').value = ocr.olmocr_serve ?? 'model';
     document.getElementById('s-ocr-olmocr-instances').value = ocr.olmocr_instances ?? 1;
@@ -2388,6 +2397,8 @@ async function saveSettings(silent){
       surya_serve: document.getElementById('s-ocr-surya-serve').value,
       surya_model: document.getElementById('s-ocr-surya-model').value.trim(),
       surya_model_id: document.getElementById('s-ocr-surya-model-id').value.trim(),
+      serve_fanout: document.getElementById('s-ocr-serve-fanout').checked,
+      serve_spill_inflight: parseInt(document.getElementById('s-ocr-serve-spill').value || '4', 10),
       surya_server_url: document.getElementById('s-ocr-surya-server-url').value.trim(),
       olmocr_enabled: document.getElementById('s-ocr-olmocr-enabled').checked,
       olmocr_serve: document.getElementById('s-ocr-olmocr-serve').value || 'model',

codai/api/app.py

@@ -256,6 +256,17 @@ class _InternalAuthMiddleware:
 
     async def __call__(self, scope, receive, send):
         if self._token and scope.get("type") == "http":
+            # The OCR fan-out gateway is the one exception, and it has to be: the
+            # client is an OCR engine running as a subprocess of THIS process, and
+            # it speaks plain OpenAI — surya's client sends a bearer token and
+            # nothing else, and its /health and /v1/models probes send no headers
+            # at all, so it cannot present the internal token however much we
+            # would like it to. The path is not open: every route under it demands
+            # a live lease (codai.ocr.gateway), a secret minted per attachment and
+            # held only in memory, and BearerAuthMiddleware checks it too.
+            if str(scope.get("path", "")).startswith("/v1/ocr/engine/"):
+                await self._app(scope, receive, send)
+                return
             headers = dict(scope.get("headers", []))
             got = headers.get(b"x-coderai-internal", b"").decode("latin-1")
             if got != self._token:
@@ -725,6 +736,11 @@ app.include_router(loras_router, tags=["LoRAs"])
 app.include_router(environments_router, tags=["Environments"])
 app.include_router(spatial_router, tags=["Spatial / 3D"])
 app.include_router(ocr_router, tags=["OCR"])
+# The address an attached OCR engine drives: one stable URL of ours, forwarded
+# per request to the least-busy server that holds the model. Not in the schema —
+# it is machine-to-machine, authenticated by a lease rather than an API token.
+from codai.ocr.gateway import router as ocr_gateway_router      # noqa: E402
+app.include_router(ocr_gateway_router, include_in_schema=False)
 app.include_router(rerank_router, tags=["Rerank"])
 from codai.api.models_transfer import router as models_transfer_router
 app.include_router(models_transfer_router, tags=["Models"])

codai/api/ratelimit.py

@@ -107,6 +107,24 @@ class BearerAuthMiddleware(BaseHTTPMiddleware):
         if _env_token and auth_header.lower().startswith("bearer "):
             if hmac.compare_digest(auth_header[7:].strip(), _env_token):
                 return await call_next(request)
+        # An OCR engine attached to the fan-out gateway. It is a subprocess of
+        # this install with no account and no API token; the credential it was
+        # given is a lease, minted per attachment and held only in memory, and it
+        # authorises nothing but its own gateway path.
+        #
+        # The lease is taken from the PATH, not the header, because the engine's
+        # probes carry no header: surya checks /health and /v1/models with a bare
+        # httpx GET (see surya/inference/backends/spawn.py) and refuses to attach
+        # at all if /health is not 200. A bearer is still accepted and still has
+        # to be a valid lease — it is what its chat requests send.
+        if path.startswith("/v1/ocr/engine/"):
+            try:
+                from codai.ocr.gateway import token_is_valid
+                lease_tok = path[len("/v1/ocr/engine/"):].split("/", 1)[0]
+                if token_is_valid(lease_tok):
+                    return await call_next(request)
+            except Exception:
+                pass
         # The shared cluster token (Settings → Cluster, or CODERAI_CLUSTER_TOKEN
         # on a capability image): what a head that discovered this install
         # presents on every request it forwards here.

codai/config.py

@@ -981,6 +981,22 @@ class OcrConfig:
     surya_model_id: str = ""           # ("model" mode) the models.json id serving Surya2
     surya_server_url: str = ""         # external OpenAI server URL (llamacpp/manual); blank = auto
 
+    # --- how a served engine reaches its model ("model" mode, both engines) --- #
+    # An OCR pipeline drives ONE server address itself, which is what keeps its
+    # images, json_schema and logprobs intact. One address cannot spread load,
+    # though: an attached engine pins to the pod it was given, and a pool that
+    # counts its own in-flight requests never sees OCR traffic at all, so it never
+    # scales. With fan-out on, the engine is handed an address of OURS that
+    # forwards each request — byte for byte, deciding nothing — to the least busy
+    # of the local server, the cluster nodes holding the model, and the pod pool.
+    # Turn it off to go back to one direct address (one pod, no growth).
+    serve_fanout: bool = True
+    # In-flight requests at which a FIXED server (local, or a cluster node) is
+    # considered busy enough to rent instead: the local-plus-spillover policy, in
+    # the one place an attached OCR engine can act on it. 0 = never rent on load
+    # alone (a pool is then used only when there is nothing else).
+    serve_spill_inflight: int = 4
+
     # --- olmOCR-2 (AllenAI, Apache-2.0) --- a document VLM: whole-page transcription in
     # reading order, equations as LaTeX, tables as HTML, no bounding boxes. Driven over an
     # OpenAI chat endpoint with olmOCR's own prompt; see codai/ocr/olmocr.py.
@@ -1615,6 +1631,8 @@ class ConfigManager:
                 "surya_model": self.config.ocr.surya_model,
                 "surya_model_id": self.config.ocr.surya_model_id,
                 "surya_server_url": self.config.ocr.surya_server_url,
+                "serve_fanout": self.config.ocr.serve_fanout,
+                "serve_spill_inflight": self.config.ocr.serve_spill_inflight,
                 "build_retry_cooldown_s": self.config.ocr.build_retry_cooldown_s,
                 "evict_drain_timeout_s": self.config.ocr.evict_drain_timeout_s,
                 "vlm_gpu_memory_utilization": self.config.ocr.vlm_gpu_memory_utilization,

codai/main.py

@@ -1529,6 +1529,13 @@ def main():
         print(f"API docs: http://{bind_host}:{bind_port}/docs")
         print(f"Admin UI: http://{bind_host}:{bind_port}/admin")
 
+    # Where this process can be reached on loopback. An OCR engine runs as a
+    # subprocess of THIS process and is handed a URL back into it (the fan-out
+    # gateway), so it has to know the port it actually bound — which for an engine
+    # is an internal port, not the configured public one.
+    os.environ["CODERAI_SELF_BASE"] = (
+        f"http{'s' if bind_https else ''}://127.0.0.1:{bind_port}")
+
     if model_manager.backend is not None:
         actual_backend = model_manager.backend_type
         if hasattr(model_manager.backend, 'force_cuda') and model_manager.backend.force_cuda:

codai/ocr/gateway.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+# GPLv3 - see the project LICENSE.
+
+"""One stable OpenAI address for an OCR engine, fanned out per request.
+
+An OCR pipeline (surya-2, olmOCR) is given ONE server address and drives it
+itself. That is deliberate: the engine sends multipart images, a strict
+``json_schema`` and ``logprobs: true``, and every one of those was quietly lost
+the last time OCR went through coderai's backend abstraction, which parses a
+request and re-emits it.
+
+This module keeps the engine's single address while letting the work spread, by
+being a hop that *decides nothing about the request*: the body goes upstream as
+the bytes that arrived, and the reply comes back as the bytes that were sent. The
+only decision is WHERE, taken per request by :mod:`codai.ocr.dispatch`, which can
+pick the local server, the least-busy cluster node, or a pod from the pool —
+counting the request against it so the pool grows under real load.
+
+The three routes are exactly what surya probes on attach (see
+``surya/inference/backends/spawn.py``):
+
+* ``GET  <root>/health``                 — 200, or it refuses to attach at all;
+* ``GET  <root>/v1/models``              — ``data[0].id`` must EQUAL the engine's
+  configured checkpoint, or it raises a model-mismatch error;
+* ``POST <root>/v1/chat/completions``    — the work.
+
+The lease in the path is a secret generated per engine attachment, and it is also
+the bearer token the engine is given, so the route is not reachable by anything
+that was not handed both.
+"""
+
+import os
+import secrets
+import threading
+from typing import Optional
+
+from fastapi import APIRouter, Request
+from fastapi.responses import JSONResponse, Response
+
+router = APIRouter()
+
+#: Path the engine attaches under. The OpenAI base it is given is
+#: ``<self>/v1/ocr/engine/<lease>/v1`` so that stripping ``/v1`` — which is how
+#: surya derives the health URL — lands on ``<self>/v1/ocr/engine/<lease>``.
+PREFIX = "/v1/ocr/engine"
+
+#: How long one page may take upstream. A dense scan on a cold pod legitimately
+#: takes minutes; a timeout here would look to the engine like a failed page.
+UPSTREAM_TIMEOUT_S = 3600.0
+
+_lock = threading.Lock()
+_leases: dict = {}              # lease token -> {"model_id", "served_name"}
+
+
+def issue(model_id: str, served_name: str) -> str:
+    """Mint a lease for one engine attachment and return its token."""
+    token = secrets.token_urlsafe(24)
+    with _lock:
+        _leases[token] = {"model_id": model_id, "served_name": served_name}
+    return token
+
+
+def revoke(model_id: str = "") -> None:
+    with _lock:
+        if not model_id:
+            _leases.clear()
+            return
+        for tok in [t for t, v in _leases.items() if v["model_id"] == model_id]:
+            _leases.pop(tok, None)
+
+
+def lease(token: str) -> Optional[dict]:
+    with _lock:
+        got = _leases.get((token or "").strip())
+        return dict(got) if got else None
+
+
+def token_is_valid(token: str) -> bool:
+    """Whether a bearer token is a live OCR lease.
+
+    The auth middleware asks this: an attached engine has no account and no API
+    token, and the lease is the credential it was given.
+    """
+    return lease(token) is not None
+
+
+def self_base() -> str:
+    """This install's own base URL, for an address we hand to a local engine.
+
+    Set at bind time (``CODERAI_SELF_BASE``). The engine runs as a subprocess on
+    this machine, so loopback is both correct and the only thing that needs no
+    certificate.
+    """
+    base = (os.environ.get("CODERAI_SELF_BASE") or "").strip().rstrip("/")
+    if base:
+        return base
+    port = (os.environ.get("CODERAI_PORT") or "").strip()
+    return f"http://127.0.0.1:{port or '8000'}"
+
+
+def engine_url(token: str) -> str:
+    """The OpenAI base an engine is given for this lease (ends ``/v1``)."""
+    return f"{self_base()}{PREFIX}/{token}/v1"
+
+
+# --------------------------------------------------------------------- routes #
+def _unknown_lease() -> JSONResponse:
+    return JSONResponse(status_code=404,
+                        content={"error": {"message": "unknown OCR engine lease",
+                                           "type": "invalid_request_error"}})
+
+
+@router.get(PREFIX + "/{token}/health", include_in_schema=False)
+async def health(token: str):
+    """200 while something can serve the model — without renting anything.
+
+    Surya refuses to attach to a URL whose ``/health`` is not 200, so this must
+    answer before any pod exists; a scale-to-zero pool is healthy precisely
+    because it can rent on demand.
+    """
+    got = lease(token)
+    if got is None:
+        return _unknown_lease()
+    from codai.ocr import dispatch
+    if not dispatch.any_candidate(got["model_id"]):
+        return JSONResponse(status_code=503,
+                            content={"error": {"message": "no server for "
+                                                          + got["model_id"]}})
+    return Response(status_code=200, content=b"OK", media_type="text/plain")
+
+
+@router.get(PREFIX + "/{token}/v1/models", include_in_schema=False)
+async def models(token: str):
+    """The id the engine expects, which is also what we will send upstream.
+
+    Surya compares ``data[0].id`` with its configured checkpoint and refuses on a
+    mismatch. Answering from the lease — rather than relaying whatever one
+    upstream happens to call it — is also what makes several upstreams
+    interchangeable to the engine.
+    """
+    got = lease(token)
+    if got is None:
+        return _unknown_lease()
+    return JSONResponse(content={
+        "object": "list",
+        "data": [{"id": got["served_name"], "object": "model",
+                  "owned_by": "coderai"}],
+    })
+
+
+@router.post(PREFIX + "/{token}/v1/chat/completions", include_in_schema=False)
+async def chat_completions(token: str, request: Request):
+    """Forward the request verbatim to whichever server is least busy.
+
+    Verbatim is the whole point. The body is not parsed, so the multipart image
+    stays an image, the ``json_schema`` stays strict, and ``logprobs`` is asked
+    for and answered — the three things a request-editing proxy dropped. The one
+    field that is rewritten is ``model``, and only when an upstream serves it
+    under another name.
+    """
+    got = lease(token)
+    if got is None:
+        return _unknown_lease()
+
+    import httpx
+
+    from codai.admin.routes import config_manager
+    from codai.ocr import dispatch
+    body = await request.body()
+    ocfg = getattr(getattr(config_manager, "config", None), "ocr", None)
+    spill_after = int(getattr(ocfg, "serve_spill_inflight", 0) or 0)
+
+    try:
+        chosen = dispatch.choose(got["model_id"], spill_after=spill_after)
+    except Exception as exc:
+        status = int(getattr(exc, "status", 503) or 503)
+        return JSONResponse(status_code=status,
+                            content={"error": {"message": str(exc),
+                                               "type": "server_error"}})
+
+    try:
+        headers = {"Content-Type": request.headers.get("content-type",
+                                                       "application/json")}
+        if chosen.api_key:
+            headers["Authorization"] = f"Bearer {chosen.api_key}"
+        url = chosen.base.rstrip("/") + "/chat/completions"
+        async with httpx.AsyncClient(timeout=UPSTREAM_TIMEOUT_S) as client:
+            reply = await client.post(url, content=body, headers=headers)
+        # The reply goes back as it came: a re-serialised response is how
+        # logprobs got lost in the first place.
+        media = reply.headers.get("content-type", "application/json")
+        return Response(status_code=reply.status_code, content=reply.content,
+                        media_type=media)
+    except Exception as exc:
+        return JSONResponse(status_code=502,
+                            content={"error": {
+                                "message": f"OCR upstream {chosen.kind} failed: {exc}",
+                                "type": "server_error"}})
+    finally:
+        chosen.release()

codai/ocr/olmocr.py

@@ -337,9 +337,36 @@ class OlmOcrEngine(OcrEngine):
         temperature = float(getattr(self.cfg, "olmocr_temperature", 0.1) or 0.0)
 
         if self._mode() == "model":
-            return self._ask_local(messages, max_tokens, temperature)
+            return self._ask_model(messages, max_tokens, temperature)
         return self._ask_http(messages, max_tokens, temperature)
 
+    def _ask_model(self, messages, max_tokens, temperature) -> str:
+        """Model mode: ask the dispatcher which server should take this page.
+
+        The in-process path below is the right one for a model this machine
+        serves — it goes through the model manager, so loading, VRAM eviction and
+        queueing all apply. It is the WRONG one for a model served elsewhere: it
+        cannot see which cluster node is least busy (nodes live in the front's
+        registry, not here), and it bypasses the front, where burst-to-RunPod
+        lives. So the choice is made per page, and only a local answer takes the
+        in-process route.
+        """
+        model_id = (getattr(self.cfg, "olmocr_model_id", "") or "").strip()
+        try:
+            from codai.ocr import dispatch
+            spill = int(getattr(self.cfg, "serve_spill_inflight", 0) or 0)
+            lease = dispatch.choose(model_id, spill_after=spill)
+        except Exception:
+            return self._ask_local(messages, max_tokens, temperature)
+        try:
+            if lease.kind == "local":
+                return self._ask_local(messages, max_tokens, temperature)
+            return self._ask_http(messages, max_tokens, temperature,
+                                  base=lease.base, api_key=lease.api_key,
+                                  model_name=model_id)
+        finally:
+            lease.release()
+
     def _ask_local(self, messages, max_tokens, temperature) -> str:
         """Through coderai's own in-process chat handler (model mode)."""
         import asyncio
@@ -365,22 +392,27 @@ class OlmOcrEngine(OcrEngine):
             raise OcrError(f"olmOCR model '{model_id}' returned no text", status=500)
         return content
 
-    def _ask_http(self, messages, max_tokens, temperature) -> str:
-        """Against an OpenAI-compatible server (vllm/server modes)."""
+    def _ask_http(self, messages, max_tokens, temperature, base: str = "",
+                  api_key: str = "", model_name: str = "") -> str:
+        """Against an OpenAI-compatible server (vllm/server modes).
+
+        ``base``/``api_key``/``model_name`` override the configured server for one
+        page, which is how model mode reaches the upstream the dispatcher chose.
+        """
         import requests
 
         payload = {
-            "model": self._model_name(),
+            "model": model_name or self._model_name(),
             "messages": messages,
             "max_tokens": max_tokens,
             "temperature": temperature,
             "stream": False,
         }
         headers = {"Content-Type": "application/json"}
-        key = (getattr(self.cfg, "olmocr_api_key", "") or "").strip()
+        key = api_key or (getattr(self.cfg, "olmocr_api_key", "") or "").strip()
         if key:
             headers["Authorization"] = f"Bearer {key}"
-        url = self._base.rstrip("/") + "/chat/completions"
+        url = (base or self._base).rstrip("/") + "/chat/completions"
         try:
             r = requests.post(url, json=payload, headers=headers,
                               timeout=float(getattr(self.cfg, "olmocr_timeout", 300.0) or 300.0))

codai/ocr/served_model.py

@@ -26,6 +26,15 @@ scaling, the budget ledger and the idle reaper stay where they belong.
 
 The endpoint is a lease, not an address: a scale-to-zero pod is reaped when OCR goes
 quiet, and the caller is expected to re-resolve rather than to trust it forever.
+
+One address, however, cannot spread load: an engine handed a pod's URL keeps using
+that pod however busy it gets, and the pool — which grows on in-flight requests it
+counts itself — never learns there is load at all. So unless the install turns it
+off (``ocr.serve_fanout``), what the engine is handed is a lease on OUR OpenAI
+address (:mod:`codai.ocr.gateway`), which forwards each request verbatim to
+whichever server is least busy (:mod:`codai.ocr.dispatch`). The direct address
+remains the behaviour with fan-out disabled, and is still what the gateway resolves
+to when there is exactly one place to go.
 """
 
 import threading
@@ -162,6 +171,30 @@ def _served_name_of(backend, model_id: str) -> str:
     return model_id
 
 
+def _fanout_enabled() -> bool:
+    try:
+        from codai.admin.routes import config_manager
+        ocfg = getattr(getattr(config_manager, "config", None), "ocr", None)
+        return bool(getattr(ocfg, "serve_fanout", True))
+    except Exception:
+        return True
+
+
+def _fanout_endpoint(model_id: str):
+    """``(gateway base, lease token, served name)``, or None if fan-out cannot be used.
+
+    The served name is resolved the same way as for a direct attachment, because the
+    engine checks it against its own checkpoint before sending a single page.
+    """
+    from codai.ocr import dispatch, gateway
+    cands = dispatch.candidates(model_id)
+    if not cands:
+        return None
+    served = model_id
+    token = gateway.issue(model_id, served)
+    return gateway.engine_url(token), token, served
+
+
 def endpoint_for(model_id: str):
     """``(base_url ending /v1, api_key, served_name)`` for ``model_id``.
 
@@ -173,6 +206,14 @@ def endpoint_for(model_id: str):
     if not model_id:
         raise OcrError("no model id configured for the served OCR engine", status=400)
 
+    # One address of ours, many servers behind it. Discovery happens now (so a
+    # misconfiguration is still a clear error at load time) but nothing is rented:
+    # a pod is picked, and paid for, per request.
+    if _fanout_enabled():
+        fan = _fanout_endpoint(model_id)
+        if fan is not None:
+            return fan
+
     with _lock:
         backend = _backends.get(model_id)
     if backend is None:

codai/ocr/surya.py

@@ -190,8 +190,12 @@ class SuryaEngine(SubprocessOcrEngine):
                 raise
             print(f"[ocr] surya endpoint {self._server_url} is gone "
                   f"({first}) — re-resolving", flush=True)
-            from codai.ocr import served_model
+            from codai.ocr import dispatch, served_model
             served_model.forget()
+            # The candidate set is discovered, so a vanished pod or node must be
+            # re-discovered too; otherwise the re-resolve hands back the same
+            # dead address it just rejected.
+            dispatch.forget()
             self.cleanup()
             self.load()
             self._loaded = True

docs/ocr.md

@@ -293,6 +293,42 @@ orchestrator: surya attaches straight to the endpoint the pool hands it. A pod r
 only the code baked into its image, so an install whose pods predate this needs its
 capability image rebuilt before the numbers appear.
 
+### Scaling in `model` mode
+
+An OCR pipeline drives one server address itself — that is what keeps the page
+images, the strict schema and the confidence scores intact. One address cannot
+spread load, though, and three things used to follow from that:
+
+* a RunPod pool grows when its least-loaded pod reaches
+  `scale_up_inflight_per_pod` **in-flight** requests, and in-flight is counted
+  when a request passes through the pool. An attached engine never did, so the
+  pool saw no load however hard OCR ran — and the engine had one pod's URL anyway;
+* several cluster nodes serving the model resolved to whichever answered first,
+  for ever, however busy it became;
+* a local model set to burst to RunPod never did: that trigger lives on the proxy
+  path an attached engine bypasses.
+
+With **`ocr.serve_fanout`** on (the default) the engine is instead handed one
+address of coderai's own, and each request is forwarded — **verbatim**, body bytes
+in and body bytes out, so nothing is decided about it — to whichever server is
+least busy:
+
+| the model is… | a page goes to |
+|---|---|
+| `backend: runpod` | the pool, every request: it picks the least-loaded pod and rents another when they are all at the threshold |
+| served on this machine | the local server, until it is busier than `ocr.serve_spill_inflight`, then a pod (the burst `runpod_spillover` describes) |
+| held by cluster nodes | the least busy of them, counting what each node reports for its **whole** install, so a node loaded with other work is left alone |
+| several of these at once | its configured home first — an explicit `backend` is a decision, not a hint — then the rest as overflow |
+
+`ocr.serve_spill_inflight: 0` means "never rent because of load": a pool is then
+used only when there is nothing else. Turning `serve_fanout` off restores the
+single direct address (one pod, no growth), which is one hop fewer and loses
+nothing on an install with exactly one server.
+
+The address is a lease: a secret minted per attachment, carried in the URL (the
+engine's `/health` and `/v1/models` probes send no headers, so it cannot be a
+bearer alone) and valid for nothing but that one gateway path.
+
 ```bash
 curl -F file=@sentenza.pdf -F structured=true -F detect=both -F schema=italian_sentenza \
      http://localhost:8776/v1/ocr

tests/test_ocr_served_model.py

@@ -393,45 +393,76 @@ class _Node:
         return {"Authorization": f"Bearer {self.api_key}"} if self.api_key else {}
 
 
-def test_a_model_held_by_a_cluster_node_resolves_to_that_node(monkeypatch):
+def test_a_model_held_by_a_cluster_node_is_found(monkeypatch):
     """A node is a whole coderai registered as a remote ENGINE, so no backend in the
     model layer knows it exists. Without asking the nodes, a node-hosted VLM answered
-    "not served over HTTP" — on an install that was serving it perfectly well."""
+    "not served over HTTP" — on an install that was serving it perfectly well.
+
+    Which node serves a given page is now decided per request by load
+    (codai.ocr.dispatch), so this asserts the discovery: the node that holds the
+    model is a candidate, carrying the shared cluster token."""
+    from codai.ocr import dispatch
+    dispatch.forget()
     _models_layer(monkeypatch)
     nodes = [_Node("alpha", "https://a.lan:8443"), _Node("beta", "https://b.lan:8443")]
     _cluster(monkeypatch, nodes, {"alpha": ["qwen"], "beta": ["datalab-to/surya-ocr-2"]})
-    base, key, served = served_model.endpoint_for("datalab-to/surya-ocr-2")
-    assert base == "https://b.lan:8443/v1"
-    assert key == "clustertok"          # the shared token fills a blank per-node key
-    assert served == "datalab-to/surya-ocr-2"
+    cands = dispatch._node_candidates("datalab-to/surya-ocr-2")
+    assert [c.base for c in cands] == ["https://b.lan:8443/v1"]
+    assert cands[0].api_key == "clustertok"   # shared token fills a blank node key
+
+
+def test_every_node_that_holds_it_is_a_candidate_not_just_the_first(monkeypatch):
+    """The first node that answered used to win for ever, however busy it got."""
+    from codai.ocr import dispatch
+    dispatch.forget()
+    _models_layer(monkeypatch)
+    nodes = [_Node("alpha", "https://a.lan:8443"), _Node("beta", "https://b.lan:8443")]
+    _cluster(monkeypatch, nodes, {"alpha": ["m"], "beta": ["m"]})
+    assert len(dispatch._node_candidates("m")) == 2
 
 
 def test_a_node_that_does_not_answer_is_skipped_not_fatal(monkeypatch):
+    from codai.ocr import dispatch
+    dispatch.forget()
     _models_layer(monkeypatch)
     nodes = [_Node("down", "https://d.lan:8443"), _Node("up", "https://u.lan:8443")]
     _cluster(monkeypatch, nodes, {"down": None, "up": ["m"]})
-    base, _, _ = served_model.endpoint_for("m")
-    assert base == "https://u.lan:8443/v1"
+    assert [c.base for c in dispatch._node_candidates("m")] == ["https://u.lan:8443/v1"]
 
 
 def test_the_nodes_are_not_asked_when_clustering_is_off(monkeypatch):
     """Probing every configured node on an install that does not cluster would add a
     timeout to the first page of every document."""
+    from codai.ocr import dispatch
+    dispatch.forget()
     _models_layer(monkeypatch)
     _cluster(monkeypatch, [_Node("alpha", "https://a.lan:8443")],
              {"alpha": ["m"]}, enabled=False)
-    with pytest.raises(OcrError):
-        served_model.endpoint_for("m")
+    assert dispatch._node_candidates("m") == []
 
 
 def test_runpod_wins_over_a_node_that_also_lists_it(monkeypatch):
     """An explicit backend pin on the entry is the operator's decision and is
-    authoritative; node discovery is the fallback, not an override."""
-    backend = FakeBackend(url="https://pod/v1", key="podtok")
-    _models_layer(monkeypatch, runpod=True, engine="runpod")
-    import codai.backends.runpod as rp
-    backend.load_model = lambda mid: None
-    monkeypatch.setattr(rp, "RunpodBackend", lambda cfg: backend)
-    _cluster(monkeypatch, [_Node("alpha", "https://a.lan:8443")], {"alpha": ["m"]})
-    base, key, _ = served_model.endpoint_for("m")
-    assert (base, key) == ("https://pod/v1", "podtok")
+    authoritative; node discovery is the fallback, not an override.
+
+    Load-aware picking must not quietly repeal that: a pool that is the model's
+    configured home is marked primary and is taken first, even though a node is
+    the cheaper hop and would otherwise win on load."""
+    from codai.ocr import dispatch
+    dispatch.forget()
+    pool_taken = []
+
+    node = dispatch.Candidate(key="node:alpha", kind="node",
+                              base="https://a.lan:8443/v1", priority=1)
+
+    def _acquire():
+        pool_taken.append(1)
+        return "https://pod/v1", "podtok", lambda: None
+    pod = dispatch.Candidate(key="pod:m", kind="pod", base="", priority=2,
+                             acquire=_acquire, primary=True)
+    monkeypatch.setattr(dispatch, "candidates", lambda m: [node, pod])
+    monkeypatch.setattr(dispatch, "_node_reported_load", lambda c: 0)
+    lease = dispatch.choose("m")
+    assert (lease.base, lease.api_key) == ("https://pod/v1", "podtok")
+    assert pool_taken == [1]
+    lease.release()