runpod: shared capability pools + trimmed per-capability images

Two things that both come down to not renting more GPU than the work
needs.

Shared pools: a pod is a whole GPU, and one coderai can serve images,
video and TTS from it. Capabilities naming the same `pool` now share one
set of pods and one budget; without it, three capabilities rented three
cards to do what one can. The pool is configured once on whichever
capability carries the settings — a bare `{"pool": "media"}` reference
will not reset it to defaults.

Trimmed images: packaging/runpod/ builds a coderai carrying ONE
capability's dependencies, since the flattened base is ~26.6 GB and every
cold pod pays that pull. Profiles under packaging/runpod/profiles/ —
core plus images/video/tts/stt/voice/audio/embeddings/ocr/faceswap — work
because the routers import lazily, so a pod only needs what its own
endpoints touch.

The Dockerfile fails the BUILD if a profile is missing something the app
imports, and build_capability_image.sh then boots the container and waits
for /healthz. Both checks exist because the alternative is finding out as
a boot timeout on a rented GPU, with nothing in the logs saying why.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY

codai/api/runpod_worker.py

@@ -740,6 +740,11 @@ def get_capability_pool(capability: str, block: dict):
 
     Defaults to `engine: coderai`, since a capability is served by a whole
     coderai on the far side rather than by an LLM server.
+
+    A block may name a shared ``pool``: capabilities naming the same pool share
+    one set of pods and one budget. That matters — a pod is a whole GPU, and one
+    coderai can serve images, video and TTS from it; without this, three
+    capabilities would rent three cards to do what one can.
     """
     from codai.models.manager import get_active_runpod_config
     acct = get_active_runpod_config()
@@ -750,8 +755,18 @@ def get_capability_pool(capability: str, block: dict):
             "capability at a URL instead.")
     b = dict(block or {})
     b.setdefault("engine", "coderai")
+    shared = str(b.get("pool") or "").strip().lower()
     mcfg = parse_model_runpod(b)
-    return get_pod_pool(f"capability:{capability}", acct, mcfg, capability)
+    key = f"capability:{shared or capability}"
+    # A shared pool is configured once (on whichever capability carries the
+    # settings) and referenced by the others, so don't let a bare `{"pool": …}`
+    # reference overwrite the real config with defaults.
+    if shared:
+        with _pools_lock:
+            existing = _pools.get(key)
+        if existing is not None and len(b) <= 2:
+            return existing
+    return get_pod_pool(key, acct, mcfg, shared or capability)
 
 
 def _all_pools_hourly_rate() -> float:

docs/remote-execution.md

@@ -252,6 +252,51 @@ larger `container_disk_gb`. And a full coderai carries every subsystem; if a pod
 only ever renders video, a trimmed image boots far faster and costs less per
 cold start.
 
+### Trimmed capability images
+
+A pod that only renders video has no use for the STT, OCR, voice-cloning or LLM
+stacks, and every cold pod pays for what it pulls. `packaging/runpod/` builds a
+coderai carrying one capability's dependencies:
+
+```bash
+./packaging/runpod/build_capability_image.sh video ghcr.io/<you>/coderai-video:latest
+PUSH=1 ./packaging/runpod/build_capability_image.sh video ghcr.io/<you>/coderai-video:latest
+```
+
+Profiles live in `packaging/runpod/profiles/` — `core.txt` (what any pod needs to
+boot) plus one file per capability: images, video, tts, stt, voice, audio,
+embeddings, ocr, faceswap. Add or trim entries there; the routers import lazily,
+so a pod only needs the libraries its own endpoints touch.
+
+The build fails if the profile is missing something the app imports, and the
+script then boots the container and waits for `/healthz` — because a pod that
+builds but never answers is a failure you would otherwise discover as a boot
+timeout on a rented GPU.
+
+Point the capability at it:
+
+```json
+"pods": {"video": {"image": "ghcr.io/<you>/coderai-video:latest"}}
+```
+
+### Sharing one pod between capabilities
+
+A pod is a whole GPU, and one coderai can serve several capabilities from it.
+Name a shared `pool` and they use the same pods and the same budget:
+
+```json
+"pods": {
+  "images": {"pool": "media", "image": "ghcr.io/<you>/coderai-media:latest",
+             "max_pods": 2, "max_hourly_usd": 0.6},
+  "video":  {"pool": "media"},
+  "tts":    {"pool": "media"}
+}
+```
+
+Configure the pool once, on whichever capability carries the settings; the others
+just reference it. Without `pool`, each capability rents its own card — three
+capabilities, three GPUs, to do what one can.
+
 **Private registry:** RunPod stores the credentials for you — *Settings →
 Container Registry Credentials* — and gives each set an id. Put that id in
 `runpod.registry_auth_id` (account-wide) or in a pod block's `registry_auth_id`

packaging/runpod/Dockerfile.capability

+# CoderAI capability pod — a trimmed coderai that serves ONE capability.
+#
+# The full coderai:base is ~26.6 GB flattened, and a rented pod pays that pull
+# before it can answer a single request. A pod that only renders video has no
+# use for the STT, OCR, voice-cloning or LLM stacks, so this image installs just
+# the profile's dependencies on top of a common core.
+#
+# It is a normal coderai: the remote gateway talks to it over the same /v1 API,
+# and `engine: coderai` pods are ready as soon as /healthz answers.
+#
+#   docker build -f packaging/runpod/Dockerfile.capability \
+#       --build-arg PROFILE=video -t ghcr.io/<you>/coderai-video:latest .
+#
+# Or use packaging/runpod/build_capability_image.sh, which also smoke-tests the
+# result before you push it.
+
+ARG BASE=python:3.12-slim
+FROM ${BASE}
+
+ARG PROFILE=images
+ARG TORCH_INDEX=https://download.pytorch.org/whl/cu124
+ENV PROFILE=${PROFILE} \
+    PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1 \
+    HF_HOME=/cache/huggingface \
+    CODERAI_HOME=/config
+
+# ffmpeg for every audio/video path; git because a few ML packages still want it
+# at install time; libgl/libglib for opencv.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        ffmpeg git curl ca-certificates libgl1 libglib2.0-0 \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /opt/coderai/app
+
+# Dependencies first so a code change doesn't re-resolve the ML stack.
+COPY packaging/runpod/profiles/ /tmp/profiles/
+RUN python -m pip install --upgrade pip \
+    && python -m pip install -r /tmp/profiles/core.txt \
+    && python -m pip install --extra-index-url ${TORCH_INDEX} -r /tmp/profiles/${PROFILE}.txt \
+    && rm -rf /tmp/profiles
+
+COPY . /opt/coderai/app
+
+# Fail the BUILD, not the pod, if the profile is missing something the app needs
+# to import. A pod that cannot import coderai just never answers /healthz, and
+# you find out by watching a boot timeout.
+RUN python -c "import codai.api.app; print('coderai imports OK for profile ${PROFILE}')"
+
+EXPOSE 8000
+# /healthz is what a `coderai` pod is probed on, and it answers before any model
+# is loaded — which is correct: the pod is ready to be asked.
+HEALTHCHECK --interval=30s --timeout=5s --start-period=120s \
+    CMD curl -fsS http://127.0.0.1:8000/healthz || exit 1
+
+CMD ["python", "-m", "uvicorn", "codai.api.app:app", "--host", "0.0.0.0", "--port", "8000"]

packaging/runpod/build_capability_image.sh

+#!/usr/bin/env bash
+# Build (and optionally push) a trimmed coderai image for one capability.
+#
+#   ./packaging/runpod/build_capability_image.sh video
+#   ./packaging/runpod/build_capability_image.sh video ghcr.io/me/coderai-video:0.1
+#   PUSH=1 ./packaging/runpod/build_capability_image.sh video ghcr.io/me/coderai-video:0.1
+#
+# The image is what a RunPod capability pod runs (`engine: coderai`). Push it to
+# a registry RunPod can pull from and put the tag in the pod block's `image`;
+# for a private registry also set `registry_auth_id`. See
+# docs/remote-execution.md.
+set -euo pipefail
+
+cd "$(dirname "$0")/../.."
+PROFILE="${1:-}"
+TAG="${2:-coderai-${PROFILE}:latest}"
+PROFILES_DIR="packaging/runpod/profiles"
+
+if [[ -z "$PROFILE" ]]; then
+    echo "usage: $0 <profile> [tag]" >&2
+    echo "profiles: $(ls "$PROFILES_DIR" | grep -v '^core.txt$' | sed 's/\.txt$//' | tr '\n' ' ')" >&2
+    exit 2
+fi
+if [[ ! -f "$PROFILES_DIR/$PROFILE.txt" ]]; then
+    echo "no such profile: $PROFILE (see $PROFILES_DIR/)" >&2
+    exit 2
+fi
+
+echo "==> building $TAG from profile '$PROFILE'"
+docker build -f packaging/runpod/Dockerfile.capability \
+    --build-arg "PROFILE=$PROFILE" \
+    -t "$TAG" .
+
+# The Dockerfile already proves the app imports. This proves it SERVES: a pod is
+# probed on /healthz, so a container that boots but never answers is the failure
+# mode worth catching here rather than on a rented GPU.
+echo "==> smoke-testing $TAG"
+cid=$(docker run -d -P "$TAG")
+trap 'docker rm -f "$cid" >/dev/null 2>&1 || true' EXIT
+port=$(docker port "$cid" 8000/tcp | head -1 | sed 's/.*://')
+for _ in $(seq 1 60); do
+    if curl -fsS "http://127.0.0.1:${port}/healthz" >/dev/null 2>&1; then
+        echo "    /healthz OK"
+        ok=1
+        break
+    fi
+    sleep 2
+done
+if [[ "${ok:-0}" != "1" ]]; then
+    echo "!!! $TAG never answered /healthz — logs follow" >&2
+    docker logs --tail 60 "$cid" >&2
+    exit 1
+fi
+
+size=$(docker image inspect "$TAG" --format '{{.Size}}')
+echo "==> $TAG built and healthy ($(( size / 1000000 )) MB)"
+
+if [[ "${PUSH:-0}" == "1" ]]; then
+    echo "==> pushing $TAG"
+    docker push "$TAG"
+fi

packaging/runpod/profiles/audio.txt

+# /v1/audio/generate, /v1/audio/stems, /v1/audio/cleanup
+torch>=2.5.0
+torchaudio
+transformers>=4.35.0
+diffusers>=0.25.0
+demucs>=4.0.1
+deepfilternet>=0.5.6
+soundfile>=0.12.0
+scipy>=1.11.0

packaging/runpod/profiles/core.txt

+# Core — what every capability pod needs to boot coderai and answer /healthz.
+# The routers import lazily, so a pod only needs the libraries its own capability
+# actually touches; this file is the floor under all of them.
+fastapi>=0.104.0
+uvicorn[standard]>=0.24.0
+pydantic>=2.5.0
+python-multipart>=0.0.6
+requests>=2.31.0
+psutil>=5.9.0
+setproctitle>=1.1
+Pillow>=10.0.0
+numpy>=1.24.0
+safetensors>=0.4.0
+huggingface_hub>=0.25.0

packaging/runpod/profiles/embeddings.txt

+# /v1/embeddings and /v1/rerank
+torch>=2.5.0
+transformers>=4.35.0
+sentence-transformers>=2.7.0
+geoclip>=1.2.0
+geopy>=2.4
+geographiclib>=2.0

packaging/runpod/profiles/faceswap.txt

+# /v1/images/faceswap
+torch>=2.5.0
+insightface>=0.7.3
+onnxruntime-gpu>=1.20.0
+opencv-python-headless>=4.8.0

packaging/runpod/profiles/images.txt

+# /v1/images/* — generation, edits, inpaint, outfit, upscale, deblur, segment, depth
+torch>=2.5.0
+torchvision
+transformers>=4.35.0
+accelerate>=0.24.0
+diffusers>=0.25.0
+ftfy>=6.0
+timm>=0.9.0
+realesrgan>=0.3.0
+basicsr>=1.4.2

packaging/runpod/profiles/ocr.txt

+# /v1/ocr — see also requirements-ocr.txt / requirements-surya.txt for the
+# engine-specific extras a given OCR backend needs.
+torch>=2.5.0
+transformers>=4.35.0
+pypdfium2>=4.20.0

packaging/runpod/profiles/stt.txt

+# /v1/audio/transcriptions + diarization/speaker endpoints
+torch>=2.5.0
+torchaudio
+transformers>=4.35.0
+faster-whisper>=0.10.0
+openai-whisper>=20231117
+vosk>=0.3.45
+soundfile>=0.12.0
+scipy>=1.11.0

packaging/runpod/profiles/tts.txt

+# /v1/audio/speech
+torch>=2.5.0
+torchaudio
+transformers>=4.35.0
+soundfile>=0.12.0
+scipy>=1.11.0
+edge-tts>=6.1.9
+kokoro-tts>=0.9.0

packaging/runpod/profiles/video.txt

+# /v1/video/* — generation, interpolation, upscale. The heaviest profile.
+torch>=2.5.0
+torchvision
+torchaudio
+transformers>=4.35.0
+accelerate>=0.24.0
+diffusers>=0.25.0
+ftfy>=6.0
+imageio[ffmpeg]>=2.33.0
+timm>=0.9.0
+realesrgan>=0.3.0
+basicsr>=1.4.2

packaging/runpod/profiles/voice.txt

+# /v1/audio/clone, /v1/audio/voices, /v1/audio/convert, watermarking
+torch>=2.5.0
+torchaudio
+transformers>=4.35.0
+f5-tts>=1.1.0
+seed-vc>=0.4.0
+soundfile>=0.12.0
+scipy>=1.11.0
+audioseal>=0.1.4

tests/test_remote_execution.py

@@ -540,6 +540,34 @@ def test_generated_file_urls_are_rewritten_and_followed_back(gateway_app, monkey
         shutdown()
 
 
+def test_capabilities_can_share_one_pod_pool(monkeypatch):
+    """A pod is a whole GPU. Three capabilities naming one pool must rent one
+    card between them, not three."""
+    import codai.api.runpod_worker as rw
+
+    class _Acct:
+        enabled = True
+
+    monkeypatch.setattr("codai.models.manager.get_active_runpod_config", lambda: _Acct())
+    monkeypatch.setattr(rw, "_pools", {})
+    monkeypatch.setattr(rw, "_ensure_scaler", lambda: None)
+
+    settings = {"pool": "media", "image": "reg/coderai:base", "max_pods": 3,
+                "max_hourly_usd": 0.6}
+    images = rw.get_capability_pool("images", settings)
+    video = rw.get_capability_pool("video", {"pool": "media"})
+    tts = rw.get_capability_pool("tts", {"pool": "media"})
+
+    assert images is video is tts
+    assert images.model_key == "capability:media"
+    # A bare reference must not reset the pool it points at to defaults.
+    assert images.mcfg.max_pods == 3 and images.mcfg.image == "reg/coderai:base"
+
+    # Without `pool`, every capability still gets its own.
+    ocr = rw.get_capability_pool("ocr", {"image": "reg/coderai:base"})
+    assert ocr is not images and ocr.model_key == "capability:ocr"
+
+
 def test_unreachable_remote_is_a_clean_502(gateway_app):
     client, gw, _ = gateway_app
     gw.capability_endpoints = lambda: {"images": "http://127.0.0.1:1"}