host backend: a capability image on a machine you already have

The capability images are ordinary containers; nothing about them needs RunPod.
The same image can run on a box you own, a server rented by the month, a second
GPU in the next room — and coderai should use it the way it uses a pod: per
model, with an endpoint and a token, and nothing else.

  always on:  {"backend": "host", "host": {"url": ..., "api_key": ...}}
  on demand:  add start_cmd (+ optional stop_cmd, boot_timeout_s, idle_timeout_s)

start_cmd is whatever starts the thing — docker run, ssh box docker start, a
systemd unit. coderai runs it, waits for /healthz, uses the host, and runs
stop_cmd after idle_timeout_s with nothing in flight. It only ever stops a host
it started itself; one that was already up is never touched.

Deliberately absent: GPU search, price ranking, budgets, spot, capacity
fallback, the orphan reaper. Those exist because a cloud rents an anonymous
card by the second and can lose it. A host is a named machine you are
responsible for — the machinery would be wrong, not just unnecessary.

Verified for real: a capability container started by command from a cold
state, healthy in 6s, used, and stopped by the idle timer.

Also fixed on the way: a plain service_url carried no token, so a hand-run
capability image behind one answered 401 to everything. service_token travels
now. Neither host nor service_* is seeded to a pod: they say where a model runs
from HERE, and would make a pod try to forward to itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY

codai/api/host_worker.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+"""The ``host`` backend: a capability image on a machine you already have.
+
+The capability images are ordinary containers. Nothing about them needs RunPod
+— a pod is just a machine that pulls one and runs it. So the same image can run
+on a box you own, a server you rent by the month, a second GPU in the next room,
+and coderai should be able to use it exactly as it uses a pod: per model, with
+an endpoint and a token, and nothing else.
+
+Two shapes, chosen by whether ``start_cmd`` is set:
+
+  always on — the container is already running; coderai just talks to it.
+      {"backend": "host", "host": {"url": "http://gpubox:8000", "api_key": "..."}}
+
+  on demand — coderai runs a command to start it, waits for /healthz, uses it,
+      and runs another command to stop it after an idle timeout. The commands
+      are whatever starts the thing: docker run, ssh box docker start, a systemd
+      unit, a script.
+      {"backend": "host", "host": {
+          "url": "http://gpubox:8000", "api_key": "...",
+          "start_cmd": "ssh gpubox docker start coderai-images",
+          "stop_cmd":  "ssh gpubox docker stop coderai-images",
+          "boot_timeout_s": 120, "idle_timeout_s": 600}}
+
+What this deliberately does NOT have: GPU search, price ranking, budgets, spot
+bids, capacity fallback, a reaper for orphaned machines. Those exist because a
+cloud rents you an anonymous card by the second and can lose it. A host is a
+named machine you are responsible for; the machinery would be wrong, not just
+unnecessary.
+
+The pool wears the same acquire()/release() shape as RunpodPodPool so the
+gateway and the test run treat both identically.
+"""
+
+import shlex
+import subprocess
+import threading
+import time
+from dataclasses import dataclass, field
+
+
+@dataclass
+class HostConfig:
+    """Parsed per-model ``host`` block."""
+    url: str = ""
+    api_key: str = ""
+    start_cmd: str = ""                  # blank = always on
+    stop_cmd: str = ""
+    health_path: str = "/healthz"
+    boot_timeout_s: int = 300            # after start_cmd, until health answers
+    idle_timeout_s: int = 600            # after the last request, run stop_cmd
+    env: dict = field(default_factory=dict)
+
+
+def parse_host(block: dict) -> HostConfig:
+    b = block or {}
+    cfg = HostConfig()
+    cfg.url = str(b.get("url") or "").strip().rstrip("/")
+    cfg.api_key = str(b.get("api_key") or "").strip()
+    cfg.start_cmd = str(b.get("start_cmd") or "").strip()
+    cfg.stop_cmd = str(b.get("stop_cmd") or "").strip()
+    cfg.health_path = str(b.get("health_path") or "/healthz").strip() or "/healthz"
+    try:
+        cfg.boot_timeout_s = max(5, int(b.get("boot_timeout_s", cfg.boot_timeout_s)))
+    except (TypeError, ValueError):
+        pass
+    try:
+        cfg.idle_timeout_s = max(0, int(b.get("idle_timeout_s", cfg.idle_timeout_s)))
+    except (TypeError, ValueError):
+        pass
+    if isinstance(b.get("env"), dict):
+        cfg.env = {str(k): str(v) for k, v in b["env"].items()}
+    return cfg
+
+
+def _health_ok(url: str, path: str, api_key: str, timeout: float = 5.0) -> bool:
+    import requests
+    headers = {"Authorization": f"Bearer {api_key}"} if api_key else {}
+    try:
+        r = requests.get(url.rstrip("/") + path, headers=headers, timeout=timeout)
+        return r.status_code == 200
+    except Exception:
+        return False
+
+
+class HostError(RuntimeError):
+    pass
+
+
+class HostPool:
+    """One host, used by one model. acquire()/release() like a pod pool."""
+
+    def __init__(self, model_key: str, cfg: HostConfig):
+        self.model_key = model_key
+        self.cfg = cfg
+        self.api_key = cfg.api_key
+        self._lock = threading.RLock()
+        self._inflight = 0
+        self._last_used = 0.0
+        self._started_by_us = False
+        self._stopper = None
+
+    # -- the shape the gateway expects ------------------------------------ #
+    def acquire(self, timeout: float = 1200.0, affinity: str = ""):
+        with self._lock:
+            if not _health_ok(self.cfg.url, self.cfg.health_path, self.api_key):
+                if not self.cfg.start_cmd:
+                    raise HostError(
+                        f"host for {self.model_key!r} at {self.cfg.url} is not "
+                        f"answering {self.cfg.health_path}, and it has no start_cmd "
+                        f"— it is configured as always-on, so it should be up.")
+                self._start()
+            self._inflight += 1
+            self._last_used = time.time()
+            return self, self.cfg.url
+
+    def release(self, handle) -> None:
+        with self._lock:
+            self._inflight = max(0, self._inflight - 1)
+            self._last_used = time.time()
+            if self.cfg.stop_cmd and self._started_by_us and self.cfg.idle_timeout_s:
+                self._arm_stopper()
+
+    # -- lifecycle ---------------------------------------------------------- #
+    def _start(self) -> None:
+        print(f"[host] starting {self.model_key!r}: {self.cfg.start_cmd}", flush=True)
+        self._run(self.cfg.start_cmd)
+        deadline = time.time() + self.cfg.boot_timeout_s
+        t0 = time.time()
+        while time.time() < deadline:
+            if _health_ok(self.cfg.url, self.cfg.health_path, self.api_key):
+                self._started_by_us = True
+                print(f"[host] {self.model_key!r} up after {time.time() - t0:.0f}s "
+                      f"at {self.cfg.url}", flush=True)
+                return
+            time.sleep(3)
+        raise HostError(
+            f"host for {self.model_key!r} did not answer {self.cfg.health_path} "
+            f"within {self.cfg.boot_timeout_s}s of running start_cmd. Check the "
+            f"command, the URL, and that the token matches CODERAI_API_TOKEN.")
+
+    def _arm_stopper(self) -> None:
+        if self._stopper is not None:
+            self._stopper.cancel()
+
+        def _maybe_stop():
+            with self._lock:
+                idle = time.time() - self._last_used
+                if self._inflight > 0 or idle < self.cfg.idle_timeout_s:
+                    self._arm_stopper()          # busy again, or not idle enough yet
+                    return
+                print(f"[host] {self.model_key!r} idle {idle:.0f}s — "
+                      f"{self.cfg.stop_cmd}", flush=True)
+                try:
+                    self._run(self.cfg.stop_cmd)
+                finally:
+                    self._started_by_us = False
+                    self._stopper = None
+
+        self._stopper = threading.Timer(self.cfg.idle_timeout_s, _maybe_stop)
+        self._stopper.daemon = True
+        self._stopper.start()
+
+    def _run(self, cmd: str) -> None:
+        import os
+        env = {**os.environ, **self.cfg.env}
+        try:
+            r = subprocess.run(shlex.split(cmd), env=env, capture_output=True,
+                               text=True, timeout=self.cfg.boot_timeout_s)
+        except subprocess.TimeoutExpired:
+            raise HostError(f"command timed out after {self.cfg.boot_timeout_s}s: {cmd}")
+        if r.returncode != 0:
+            raise HostError(f"command failed ({r.returncode}): {cmd}\n"
+                            f"{(r.stderr or r.stdout or '').strip()[:400]}")
+
+    def stop(self) -> None:
+        """Explicit stop — the cleanup path, not the idle one."""
+        with self._lock:
+            if self._stopper is not None:
+                self._stopper.cancel()
+                self._stopper = None
+            if self.cfg.stop_cmd and self._started_by_us:
+                try:
+                    self._run(self.cfg.stop_cmd)
+                finally:
+                    self._started_by_us = False
+
+
+_pools: dict = {}
+_pools_lock = threading.Lock()
+
+
+def get_host_pool(model_key: str, block: dict) -> HostPool:
+    with _pools_lock:
+        pool = _pools.get(model_key)
+        if pool is None:
+            pool = HostPool(model_key, parse_host(block))
+            _pools[model_key] = pool
+        return pool

codai/api/model_test.py

@@ -447,6 +447,9 @@ def _no_remote_reason(model: str, entry: dict, capability: str) -> str:
         return ""
     if isinstance(entry.get("runpod"), dict) and entry["runpod"]:
         return ""
+    if str(entry.get("backend") or "").lower() == "host" and \
+            str((entry.get("host") or {}).get("url") or "").strip():
+        return ""
     try:
         from codai.api.remote_gateway import capability_endpoints, capability_pods
         if capability and (capability in capability_endpoints()
@@ -572,6 +575,10 @@ def _placement_summary(model: str, entry: dict, force: str) -> dict:
         return {"where": "remote", "target": str(detail)}
     if kind == "pod":
         return {"where": "runpod", "target": "its own pod"}
+    if kind == "host":
+        _entry, hblock = detail
+        how = "started on demand" if str(hblock.get("start_cmd") or "").strip() else "always on"
+        return {"where": "remote", "target": f"host {hblock.get('url')} ({how})"}
     backend = str((entry or {}).get("backend") or "").lower()
     if backend == "runpod":
         return {"where": "runpod", "target": "RunPod backend"}

codai/api/remote_gateway.py

@@ -210,6 +210,12 @@ def model_placement(model: str) -> tuple:
     block = entry.get("runpod") if isinstance(entry.get("runpod"), dict) else None
     if backend == "runpod" and block is not None:
         return "pod", (entry, block)
+    # A capability image on a machine you already have: an endpoint and a
+    # token, optionally a command to start it. Same acquire/release shape as a
+    # pod, none of the renting machinery.
+    hblock = entry.get("host") if isinstance(entry.get("host"), dict) else None
+    if backend == "host" and hblock is not None and str(hblock.get("url") or "").strip():
+        return "host", (entry, hblock)
     return "", None
 
 
@@ -358,7 +364,24 @@ def resolve_target(path: str, method: str, query: str, body: bytes,
     if where == "local":
         return None                              # pinned here, whatever the capability says
     if where == "url":
-        return Target(url=detail, reason=f"model {model!r}")
+        target = Target(url=detail, reason=f"model {model!r}")
+        # A capability image locks itself with CODERAI_API_TOKEN, so a
+        # hand-run one behind a plain service_url answers 401 to everything
+        # unless the entry carries its token. `service_token` is that.
+        try:
+            from codai.models.manager import _model_entry_for
+            target.api_key = str((_model_entry_for(model) or {}).get("service_token")
+                                 or "").strip()
+        except Exception:
+            pass
+        return target
+    if where == "host":
+        entry, hblock = detail
+        from codai.api.host_worker import get_host_pool
+        target = Target(pool=get_host_pool(model, hblock),
+                        reason=f"model {model!r} on a host you run")
+        target.api_key = str(hblock.get("api_key") or "").strip()
+        return target
     if where == "pod":
         entry, block = detail
         from codai.api.runpod_worker import (get_model_pod_pool, parse_model_runpod,

codai/api/runpod_worker.py

@@ -854,7 +854,11 @@ def seed_model_env(entry: dict, served: str = "", source: str = "") -> str:
             "lora_path", "lora_model_dir", "lora_scale", "loras",
             # Which MusicGen implementation to use. Describes the model, not
             # this deployment's placement, so it travels.
-            "audio_backend")
+            "audio_backend",
+            # NOT `host`, `service_url` or `service_token`: those say where a
+            # model runs from HERE. On the far side they would make a pod try
+            # to forward to itself, or to a machine it cannot reach.
+            )
     out = {k: entry[k] for k in keep if k in entry and entry[k] is not None}
     out["path"] = path
     # Adapters the pod cannot resolve by name are named by CONTENT id instead.

tests/test_remote_execution.py

@@ -1771,3 +1771,87 @@ def test_xtts_falls_through_to_its_own_venv_when_not_importable(monkeypatch):
         assert "/opt/coderai/venvs/TTS" in str(e)
     else:
         raise AssertionError("expected MissingEngineError")
+
+
+# --- the host backend: a capability image on a machine you already have ---- #
+
+def test_a_model_can_be_placed_on_a_host_you_run(monkeypatch):
+    """The capability images are ordinary containers. A box you own, a server
+    rented by the month, a second GPU down the hall — coderai uses it per model
+    with an endpoint and a token, and none of RunPod's renting machinery."""
+    import codai.models.manager as mgr
+    from codai.api.remote_gateway import model_placement
+
+    entry = {"path": "org/m", "model_type": "image_models", "backend": "host",
+             "host": {"url": "http://gpubox:8000", "api_key": "k"}}
+    monkeypatch.setattr(mgr, "_model_entry_for", lambda n: entry)
+    kind, detail = model_placement("m")
+    assert kind == "host" and detail[1]["url"] == "http://gpubox:8000"
+
+    # No url, no placement — silently routing to nowhere is worse than local.
+    entry["host"] = {"api_key": "k"}
+    assert model_placement("m") == ("", None)
+
+
+def test_an_always_on_host_that_is_down_says_so_instead_of_starting_nothing(monkeypatch):
+    from codai.api import host_worker as hw
+
+    monkeypatch.setattr(hw, "_health_ok", lambda *a, **k: False)
+    pool = hw.HostPool("m", hw.parse_host({"url": "http://gpubox:8000"}))
+    try:
+        pool.acquire()
+    except hw.HostError as e:
+        assert "always-on" in str(e) and "no start_cmd" in str(e)
+    else:
+        raise AssertionError("expected HostError")
+
+
+def test_an_on_demand_host_is_started_used_and_stopped_after_idle(monkeypatch):
+    """start_cmd runs, health is awaited, and after idle_timeout_s with nothing
+    in flight stop_cmd runs. A request arriving during the idle window keeps it
+    alive. Verified for real against a capability container as well; this is
+    the fast version."""
+    import time
+    from codai.api import host_worker as hw
+
+    ran = []
+    up = {"v": False}
+    def _run(self, cmd):
+        ran.append(cmd)
+        up["v"] = cmd.startswith("start")
+    monkeypatch.setattr(hw.HostPool, "_run", _run)
+    monkeypatch.setattr(hw, "_health_ok", lambda *a, **k: up["v"])
+
+    pool = hw.HostPool("m", hw.parse_host({
+        "url": "http://gpubox:8000", "start_cmd": "start it", "stop_cmd": "stop it",
+        "boot_timeout_s": 5, "idle_timeout_s": 1}))
+    h, url = pool.acquire()
+    assert ran == ["start it"] and url == "http://gpubox:8000" and pool._started_by_us
+    pool.release(h)
+    time.sleep(1.6)
+    assert ran == ["start it", "stop it"] and not pool._started_by_us
+
+    # A host we did NOT start is never stopped by us, even with a stop_cmd.
+    ran.clear(); up["v"] = True
+    pool2 = hw.HostPool("m2", hw.parse_host({
+        "url": "http://gpubox:8000", "start_cmd": "start it", "stop_cmd": "stop it",
+        "idle_timeout_s": 1}))
+    h2, _ = pool2.acquire(); pool2.release(h2)
+    time.sleep(1.6)
+    assert ran == [], "stopped a host we never started"
+
+
+def test_a_plain_service_url_carries_its_token(monkeypatch):
+    """A capability image locks itself with CODERAI_API_TOKEN, so a hand-run one
+    behind a bare service_url answered 401 to everything: the URL routed, the
+    token never travelled."""
+    import codai.models.manager as mgr
+    from codai.api import remote_gateway as rg
+
+    entry = {"path": "org/m", "model_type": "image_models",
+             "service_url": "http://gpubox:8000", "service_token": "tok-123"}
+    monkeypatch.setattr(mgr, "_model_entry_for", lambda n: entry)
+    monkeypatch.setattr(rg, "_model_remote", lambda m: "http://gpubox:8000")
+    t = rg.resolve_target("/v1/images/generations", "POST", "",
+                          b'{"model":"m"}', "application/json")
+    assert t is not None and t.url == "http://gpubox:8000" and t.api_key == "tok-123"