api: remote gateway — serve any capability from another coderai

Images, video, embeddings, rerank, OCR, TTS, STT, voice cloning, audio
generation, stems, 3D and pipelines all load their model in-process, so
none of them had a service to point elsewhere. Writing thirteen services
would be the long way round; another coderai already exposes all of them,
so it can be the service.

RemoteGatewayMiddleware sits in front of the routes and, when a request is
addressed to something configured as remote, replays it verbatim to that
endpoint and streams the answer back — JSON and multipart alike, since the
body is forwarded as raw bytes. Routing is per model (`service_url` on the
models.json entry) first, then per capability (`remotes.endpoints` in
config.json, or CODERAI_REMOTE_<CAP>_URL).

Placement matters and is deliberate: it is registered first, so it ends up
innermost — auth, rate limiting and request logging all run before it, and
a request still has to be authorised here before being forwarded. Chat and
completions are excluded: those go through RemoteOpenAIBackend, which keeps
usage accounting, tool parsing and the manager's bookkeeping.

When nothing is configured remote it returns on its first branch and never
buffers a body (cached 5s so it doesn't walk the model list per request).
Bodies over `max_body_mb` are served locally rather than held in RAM. A
non-forwarded request replays its buffered body downstream untouched —
tested, including multipart uploads that still parse locally.

docs/remote-execution.md documents all four remotization mechanisms
together.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY

codai/admin/routes.py

@@ -3527,6 +3527,12 @@ def build_settings_dict(c, gpu_cards):
             "allow_ffmpeg": c.enhance.allow_ffmpeg,
             "allow_rife_ncnn": c.enhance.allow_rife_ncnn,
         },
+        "remotes": {
+            "enabled": c.remotes.enabled,
+            "api_key": c.remotes.api_key,
+            "max_body_mb": c.remotes.max_body_mb,
+            "endpoints": dict(c.remotes.endpoints or {}),
+        },
         "ds4": {
             "enabled": c.ds4.enabled,
             "service_url": c.ds4.service_url,
@@ -3964,6 +3970,23 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
         except Exception:
             pass
 
+    if "remotes" in data:
+        d = data["remotes"] if isinstance(data["remotes"], dict) else {}
+        if "enabled" in d:
+            c.remotes.enabled = bool(d["enabled"])
+        if "api_key" in d:
+            c.remotes.api_key = (d.get("api_key") or "").strip()
+        if "max_body_mb" in d:
+            try:
+                c.remotes.max_body_mb = max(1, int(d["max_body_mb"]))
+            except (TypeError, ValueError):
+                pass
+        if "endpoints" in d:
+            eps = d.get("endpoints")
+            c.remotes.endpoints = ({str(k).strip().lower(): str(v).strip()
+                                    for k, v in eps.items() if str(v).strip()}
+                                   if isinstance(eps, dict) else {})
+
     if "ds4" in data:
         d = data["ds4"]
         if "service_url" in d:

codai/api/app.py

@@ -149,6 +149,13 @@ from codai.admin.routes import router as admin_router
 # Import and add middleware
 from codai.api.log import log_requests
 from codai.api.ratelimit import RateLimitMiddleware, BearerAuthMiddleware
+# Remote gateway: /v1 requests addressed to a model or capability configured as
+# remote are replayed to that endpoint instead of being served here. Registered
+# FIRST so it ends up innermost — auth, rate limiting and request logging all run
+# before it. It no-ops without touching the body when nothing is configured remote.
+from codai.api.remote_gateway import RemoteGatewayMiddleware
+app.add_middleware(RemoteGatewayMiddleware)
+
 app.middleware("http")(log_requests)
 app.add_middleware(RateLimitMiddleware)
 app.add_middleware(BearerAuthMiddleware)

codai/config.py

@@ -356,6 +356,32 @@ class CompactionConfig:
     estimate_safety: float = 1.15        # inflate the chars/4 prompt estimate by this factor for fit/trim (it undercounts code/JSON)
 
 
+@dataclass
+class RemotesConfig:
+    """Serve whole capabilities from another coderai instead of locally.
+
+    Every subsystem that loads its model in-process — images, video, embeddings,
+    rerank, OCR, TTS, STT, voice cloning, audio generation, stems, 3D, pipelines
+    — has no service of its own to redirect. But another coderai exposes the same
+    API, so it can BE the service: map a capability to a base URL here and
+    :class:`~codai.api.remote_gateway.RemoteGatewayMiddleware` replays matching
+    /v1 requests there and streams the answer back.
+
+    A model's own ``service_url`` (models.json) wins over this map. Chat and
+    completions are not routed here — they go through RemoteOpenAIBackend.
+
+    Example::
+
+        "remotes": {"enabled": true,
+                    "endpoints": {"images": "http://gpu-box:8000",
+                                  "video": "https://pod-xyz-8000.proxy.runpod.net"}}
+    """
+    enabled: bool = True
+    api_key: str = ""                        # bearer token for the remote(s)
+    max_body_mb: int = 512                   # bigger requests are served locally
+    endpoints: dict = field(default_factory=dict)   # capability -> base URL
+
+
 @dataclass
 class Ds4Config:
     """DeepSeek V4 via ds4 (antirez/DwarfStar) external-worker configuration.
@@ -707,6 +733,7 @@ class Config:
     thermal: ThermalConfig = field(default_factory=ThermalConfig)
     jobs: JobsConfig = field(default_factory=JobsConfig)
     enhance: EnhanceConfig = field(default_factory=EnhanceConfig)
+    remotes: RemotesConfig = field(default_factory=RemotesConfig)
     ds4: Ds4Config = field(default_factory=Ds4Config)
     colibri: ColibriConfig = field(default_factory=ColibriConfig)
     k3: K3Config = field(default_factory=K3Config)
@@ -898,6 +925,7 @@ class ConfigManager:
                 thermal=_dc(ThermalConfig, config_data.get("thermal", {})),
                 jobs=_dc(JobsConfig, config_data.get("jobs", {})),
                 enhance=_dc(EnhanceConfig, config_data.get("enhance", {})),
+                remotes=_dc(RemotesConfig, config_data.get("remotes", {})),
                 ds4=_dc(Ds4Config, config_data.get("ds4", {})),
                 colibri=_dc(ColibriConfig, config_data.get("colibri", {})),
                 k3=_dc(K3Config, config_data.get("k3", {})),
@@ -1068,6 +1096,12 @@ class ConfigManager:
                 "allow_ffmpeg": self.config.enhance.allow_ffmpeg,
                 "allow_rife_ncnn": self.config.enhance.allow_rife_ncnn,
             },
+            "remotes": {
+                "enabled": self.config.remotes.enabled,
+                "api_key": self.config.remotes.api_key,
+                "max_body_mb": self.config.remotes.max_body_mb,
+                "endpoints": dict(self.config.remotes.endpoints or {}),
+            },
             "ds4": {
                 "enabled": self.config.ds4.enabled,
                 "service_url": self.config.ds4.service_url,

docs/README.md

@@ -16,6 +16,7 @@ reference; these are the deep dives.
 | [kimi-k3.md](kimi-k3.md) | Kimi-K3 on CPU via kimi-k3-in-c |
 | [ktransformers.md](ktransformers.md) | ktransformers / SGLang CPU+GPU heterogeneous MoE |
 | [runpod.md](runpod.md) | Renting remote GPUs: pods, serverless, budgets, the reaper, spillover |
+| [remote-execution.md](remote-execution.md) | Serving any model from another machine: worker `service_url`, the mux service, remote text models, the capability gateway |
 
 ## Subsystems
 

docs/remote-execution.md

+# Running anything somewhere else
+
+Every model coderai serves can now be served from another machine instead: a
+second coderai, a bare `llama-server`/vLLM/SGLang, a rented RunPod pod, or a
+hosted OpenAI-compatible API. Nothing is downloaded locally, no local VRAM is
+used, and clients see no difference.
+
+There are four mechanisms, because the subsystems are shaped differently. Pick
+by what you are moving.
+
+| What you're moving | Mechanism | Where you configure it |
+|---|---|---|
+| An isolated-venv worker (vLLM, ds4, kt, h3, pyannote, MeloTTS, parler, Canary, CrisperWhisper) | `service_url` on the worker | engine config block, or `CODERAI_<NAME>_SERVICE_URL` |
+| colibri / k3 (mux engines) | `service_url` + `tools/mux_service.py` on the far side | `colibri.service_url` / `k3.service_url` |
+| Any text model, GGUF included | `service_url` on the model entry | models.json |
+| Images, video, embeddings, rerank, OCR, TTS, STT, voice, audio, stems, 3D, pipelines | remote gateway | `remotes.endpoints` in config.json |
+
+---
+
+## 1. Workers with a service
+
+Each isolated-venv worker's `ensure_service()` returns a base URL, so pointing
+it at a URL is enough — it health-checks the far side and returns it instead of
+building a venv, downloading weights, or spawning anything:
+
+```json
+{ "vllm": { "enabled": true, "service_url": "https://pod-abc-8000.proxy.runpod.net" } }
+```
+
+Env equivalents: `CODERAI_VLLM_SERVICE_URL`, `CODERAI_DS4_SERVICE_URL`,
+`CODERAI_KT_SERVICE_URL`, `CODERAI_H3_SERVICE_URL`,
+`CODERAI_PYANNOTE_SERVICE_URL`, `CODERAI_MELOTTS_SERVICE_URL`,
+`CODERAI_PARLER_SERVICE_URL`, `CODERAI_CANARY_SERVICE_URL`,
+`CODERAI_CRISPERWHISPER_SERVICE_URL`.
+
+A wrong or dead URL fails immediately with
+`configured service_url … is not answering its health check`, rather than as a
+confusing timeout later. vLLM additionally verifies the remote actually serves
+the requested model — the OCR path asks for surya-2 by name, and a generic
+remote vLLM would otherwise be proxied silently.
+
+## 2. colibri and k3
+
+These two drive their C engine over a stdin/stdout mux protocol, so they have no
+HTTP surface to redirect. `tools/mux_service.py` is that surface: run it on the
+machine that holds the container, and point coderai at it.
+
+On the remote host (needs the model container and the built engine):
+
+```bash
+python tools/mux_service.py --kind colibri --port 8092 \
+    --model-dir /models/GLM-5.2-int4 --ctx 100000
+# k3: --kind k3 --model-dir /models/Kimi-K3
+```
+
+In coderai:
+
+```json
+{ "colibri": { "enabled": true, "service_url": "http://engine-box:8092" } }
+```
+
+`codai/api/mux_remote.RemoteMuxEngine` presents the same surface as a local
+`MuxEngine`, so streaming, stats, pause/resume and mid-turn cancellation all
+behave identically — cancellation travels as a dropped connection.
+
+## 3. Text models, including GGUF
+
+GGUF models load in-process through llama-cpp-python; there is no service to
+redirect. So point the *model* at a URL instead — add `service_url` to its
+models.json entry and `codai/backends/remote_openai.py` proxies it:
+
+```json
+{
+  "path": "/AI/guffcache/gemma-4-31B-Q4_0.gguf",
+  "service_url": "http://gpu-box:8080",
+  "served_model": "gemma-4-31b",
+  "api_key": ""
+}
+```
+
+`service_url` may include `/v1` or not. `served_model` is the name the remote
+knows it by (defaults to the model's own name); `api_key` adds a bearer token.
+The manager checks this before every engine route, and such a model is never
+downloaded, cached, or charged local VRAM.
+
+The far side can be anything OpenAI-compatible: another coderai, `llama-server`,
+vLLM, SGLang, TGI, or a hosted API.
+
+### On RunPod
+
+A RunPod pod picks its server from the model:
+
+```json
+{ "runpod": { "mode": "pods", "engine": "auto",
+              "hf_gguf": "bartowski/gemma-4-31B-GGUF:Q4_0", "ctx": 32768 } }
+```
+
+* `engine: auto` — an HF repo goes to vLLM (`vllm/vllm-openai`), a GGUF goes to
+  llama.cpp (`ghcr.io/ggml-org/llama.cpp:server-cuda`).
+* `engine: vllm | llamacpp` — force one.
+
+GGUF does **not** simply go to vLLM. vLLM's GGUF loader is experimental: single
+file only (sharded GGUFs must be merged first), a limited architecture list, and
+it still wants the original repo for the tokenizer. llama.cpp's own server image
+serves the whole GGUF catalogue as-is, and exposes the same `/v1` surface, so
+the readiness probe and the proxy are unchanged.
+
+`hf_gguf` is required for a llama.cpp pod and takes llama.cpp's own `-hf` form
+(`user/repo:Q4_K_M`). A local `/AI/…/foo.gguf` path means nothing on a rented
+machine, and coderai does not upload multi-GB weights.
+
+## 4. Everything else — the remote gateway
+
+Images, video, embeddings, rerank, OCR, TTS, STT, voice cloning, audio
+generation, stems, 3D and pipelines all load their model in-process. Rather than
+write a service for each, note that *another coderai already exposes all of
+them* — so it can be the service.
+
+`codai/api/remote_gateway.py` sits in front of the routes, and when a request is
+addressed to something configured as remote it replays it verbatim to that
+endpoint and streams the answer back:
+
+```json
+{
+  "remotes": {
+    "enabled": true,
+    "api_key": "",
+    "max_body_mb": 512,
+    "endpoints": {
+      "images": "http://gpu-box:8000",
+      "video": "https://pod-xyz-8000.proxy.runpod.net",
+      "embeddings": "http://cpu-box:8000"
+    }
+  }
+}
+```
+
+Capability keys: `images`, `video`, `embeddings`, `rerank`, `ocr`, `tts`, `stt`,
+`voice`, `speaker`, `audio_gen`, `stems`, `audio_clean`, `spatial`, `faceswap`,
+`loras`, `characters`, `environments`, `pipelines`. Env equivalents:
+`CODERAI_REMOTE_IMAGES_URL` and friends.
+
+Precedence: a model's own `service_url` wins over the capability map.
+
+Notes:
+
+* `/v1/chat/completions` and `/v1/completions` are deliberately **not** gatewayed
+  — they go through `RemoteOpenAIBackend`, which keeps usage accounting, tool
+  parsing and the model manager's bookkeeping intact.
+* Auth and rate limiting run *before* the gateway, so a request still has to be
+  authorised here before it is forwarded.
+* When nothing is configured as remote the middleware returns on its first
+  branch and never buffers a body.
+* Requests larger than `max_body_mb` are served locally rather than buffered in
+  RAM; raise it if you forward large uploads.
+
+## Deployment shapes this enables
+
+* **One small coderai, many remote GPUs** — the local instance holds the API,
+  auth, catalogue and UI; each capability points at a box or pod that owns one
+  model family.
+* **Per-engine pod images** — vLLM's and llama.cpp's own images serve LLMs with
+  no coderai on the far side; the coderai image is only needed where a coderai
+  subsystem runs (the mux service, or a gateway target).
+* **Burst to RunPod** — pin a model to the `runpod` backend and it provisions,
+  scales and reaps pods against the configured budget (see `docs/runpod.md`).