runpod: an engines pod image — ds4, colibri, k3 compiled for every pod GPU

The local ds4-server is an sm_86 build against CUDA 13 and the local
colibri has no GPU backend: neither could be copied into a pod. The new
engines image compiles both, plus kimi-k3-in-c, in a CUDA 12.8 stage for
sm_80/86/89/90/120a (ds4's Blackwell kernels need the arch-specific
target; plain sm_120 fails in ptxas) on a portable x86-64-v3 baseline,
and fails the build if a binary cannot find its libraries.

A RunPod-only model names the engine on its runpod block (backend:
runpod takes the pin slot); a local model pinned to the engine and
bursting carries the pin. Either way the pod arrives with the engine
enabled and this install's settings forwarded (CODERAI_<ENG>_CONFIG),
and volume_path points it at weights already on the network volume —
the only sane way to run 150 GB+ models. The model page warns loudly
the moment one of these engines is picked. ktransformers is refused
with a reason: it has no pod image.

Also: the batch build/publish scripts no longer list core-light and
venv profiles as images.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY

.gitignore

@@ -55,3 +55,6 @@ tools/coderai_media/
 .oci-*.log
 venv_build.log
 .build.pid
+
+# staged engine sources for the engines pod image (build_capability_image.sh)
+packaging/runpod/engines-src/

README.md

@@ -97,9 +97,19 @@ Plus:
   each carrying its own torch on a torch-free core so it stays under RunPod's image size
   limit. Fourteen have completed a real inference on a rented GPU; `faceswap` is the one
   that needs real faces to test
+- **The native MoE engines on rented GPUs**: an `engines` image compiles ds4, colibri and
+  kimi-k3-in-c for every pod GPU generation (Ampere → Blackwell) rather than copying this
+  machine's builds; a model names its engine on its runpod block and the pod arrives with
+  it enabled and configured. ⚠ Their weights are 100 GB–1.5 TB: put them on a network
+  volume (`volume_path`) or every cold pod downloads them — the UI warns, loudly
+- **Multi-GPU pods**: `gpu_count` rents one machine with N cards, priced and VRAM-checked as
+  a whole; vLLM and llama.cpp are told to shard across them
 - **Or a machine you already have**: `"backend": "host"` points a model at a capability
   image running anywhere — always-on, or started by a command and stopped after idle —
-  with none of the renting machinery
+  with none of the renting machinery. Both are one choice on the model page: *Runs on →
+  this machine / RunPod only / a machine of yours*
+- **Visible while it runs**: every request sent to a pod or host is a row on the Tasks page
+  naming the machine it landed on, and `/admin/runpod` lists the live pods
 
 Full guide: [`docs/runpod.md`](docs/runpod.md) ·
 remote placement of every model type, the capability images and the `host` backend:

codai/admin/routes.py

@@ -2795,7 +2795,7 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                   "registry_auth_id", "api_key", "pool",
                   # where the POD gets the weights
                   "source", "hf_repo", "model_url", "quantization",
-                  "network_volume_id", "volume_mount_path",
+                  "network_volume_id", "volume_mount_path", "volume_path",
                   "venv_name", "slim_image"):
             v = src.get(k)
             if isinstance(v, str) and v.strip():

codai/admin/templates/models.html

@@ -786,9 +786,15 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
                 <option value="vllm">vLLM (HF repo)</option>
                 <option value="llamacpp">llama.cpp (GGUF)</option>
                 <option value="coderai">coderai (a capability image)</option>
+                <option value="ds4">ds4 — DeepSeek-V4 (engines image)</option>
+                <option value="colibri">colibri — GLM-5.2 / DeepSeek-V4 / Kimi-K3 (engines image)</option>
+                <option value="k3">k3 — Kimi-K3 (engines image)</option>
                 <option value="custom">Custom image</option>
               </select>
               <span class="form-hint" id="cfg-rp-engine-hint" style="font-size:11px"></span>
+              <div id="cfg-rp-engine-warn" style="display:none;margin-top:.4rem;padding:.5rem .6rem;border:1px solid #c9a227;border-radius:6px;background:rgba(201,162,39,.12);font-size:12px;line-height:1.4">
+                <b>⚠ Frontier-size weights.</b> Models on this engine are 100 GB and up — DeepSeek-V4 alone is ~154 GB, Kimi-K3 is measured in terabytes. Without a network volume <b>every cold pod downloads all of it</b> before answering, typically one to two hours of rental per boot. Set a network volume and, better, the "weights already on the volume" path so nothing is downloaded at all. The pod's disk is sized from <i>Weights (GB)</i> when no volume is set.
+              </div>
             </div>
             <div class="form-row" style="margin:0">
               <label class="form-label">GGUF to serve <span class="muted">(llama.cpp pods)</span></label>
@@ -822,6 +828,11 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
               <input type="text" id="cfg-rp-volume" class="form-input" placeholder="blank = the account default, or none">
               <span class="form-hint" style="font-size:11px">Weights, uploads and adapters land here and outlive the pod. Secure Cloud only; pods are pinned to the volume's region.</span>
             </div>
+            <div class="form-row" style="margin:0">
+              <label class="form-label">Weights already on the volume <span class="muted">(path)</span></label>
+              <input type="text" id="cfg-rp-volume-path" class="form-input" placeholder="models/DeepSeek-V4-Q4.gguf, or a container dir">
+              <span class="form-hint" style="font-size:11px">Relative to the volume mount. The pod loads this instead of downloading — the only sane way to run a 150 GB+ engine model (ds4, colibri, k3).</span>
+            </div>
             <div class="form-row" style="margin:0;align-self:start">
               <label style="display:flex;align-items:center;gap:.3rem;font-size:12px;margin-top:1.4rem"><input type="checkbox" id="cfg-rp-venv-volume" style="width:auto"> Keep dependencies on the volume</label>
               <span class="form-hint" style="font-size:11px">Boots a small image and runs from a venv on the volume — the first pod builds it, later pods skip the 7 GB pull. Needs a volume; imports are slower off network storage.</span>
@@ -4290,6 +4301,7 @@ function _populateRunpod(s){
   set('cfg-rp-hf-gguf', rp.hf_gguf || '');
   set('cfg-rp-pool', rp.pool || '');
   set('cfg-rp-volume', rp.network_volume_id || '');
+  set('cfg-rp-volume-path', rp.volume_path || '');
   chk('cfg-rp-venv-volume', rp.venv_on_volume);
   set('cfg-rp-source', rp.source || 'auto');
   set('cfg-rp-hf-repo', rp.hf_repo || '');
@@ -4383,9 +4395,14 @@ function _updateRunpodEngineHint(){
     vllm: 'Serves an HF repo. vLLM\'s GGUF support is experimental — use llama.cpp for those.',
     llamacpp: 'Serves the GGUF named on the right; the pod downloads it itself.',
     coderai: 'A whole coderai serving one capability. Needs a pod image you have pushed.',
+    ds4: 'ds4-server on the engines image, built for every pod GPU. Downloads the configured variant unless the weights are on the volume.',
+    colibri: 'colibri on the engines image (GLM-5.2 on CUDA; DeepSeek-V4 and Kimi-K3 on CPU). Never downloads: the container dir must be on the volume.',
+    k3: 'kimi-k3-in-c on the engines image (CPU). Never downloads: the checkpoint must be on the volume.',
     custom: 'Your image, your docker args, your health path.',
   };
   hint.textContent = HINTS[el.value] || '';
+  const warn = document.getElementById('cfg-rp-engine-warn');
+  if (warn) warn.style.display = (el.value === 'ds4' || el.value === 'colibri' || el.value === 'k3') ? '' : 'none';
 }
 
 // Build the per-model spillover block. Undefined when disabled, so the entry
@@ -4461,6 +4478,7 @@ function _collectRunpod(){
     hf_gguf:            (val('cfg-rp-hf-gguf') || '').trim(),
     pool:               (val('cfg-rp-pool') || '').trim(),
     network_volume_id:  (val('cfg-rp-volume') || '').trim(),
+    volume_path:        (val('cfg-rp-volume-path') || '').trim(),
     venv_on_volume:     document.getElementById('cfg-rp-venv-volume').checked,
     source:             val('cfg-rp-source') || 'auto',
     hf_repo:            (val('cfg-rp-hf-repo') || '').trim(),
@@ -4821,7 +4839,8 @@ async function testRunModel(){
     if (d.ran === 'reachability') {
       show(`${d.model}\nwhere: ${d.where} (${d.target})\n${d.note}\n${d.detail || ''}`, 'busy');
     } else if (d.ok) {
-      show(`OK — ${d.ran} in ${d.seconds}s\nwhere: ${d.where} (${d.target})\n`
+      show((d.warning ? `⚠ ${d.warning}\n\n` : '')
+           + `OK — ${d.ran} in ${d.seconds}s\nwhere: ${d.where} (${d.target})\n`
            + `reply: ${d.sample || '(no sample)'}`, 'ok');
     } else {
       show(`FAILED — ${d.ran || ''} ${d.status || ''} after ${d.seconds || '?'}s\n`

codai/config.py

@@ -1358,6 +1358,40 @@ class ConfigManager:
                 # inherits the decision made here; it cannot make it itself.
                 self.config.ocr.surya_accept_license = True
 
+        # The native engines — ds4, colibri, k3 — ship disabled for the same
+        # reason OCR does, and an engines pod is rented to run exactly one of
+        # them. CODERAI_<ENGINE>_ENABLED switches it on; CODERAI_<ENGINE>_CONFIG
+        # carries the renting install's settings for it (context, expert
+        # cache, download variant…) as JSON, applied field by field onto the
+        # dataclass — only fields it already has, so a stray key cannot plant
+        # anything. Where the binary lives is the image's business
+        # (CODERAI_<ENGINE>_DIR, read by the worker itself), not this block's.
+        for name in ("ds4", "colibri", "k3"):
+            if not _flag(f"CODERAI_{name.upper()}_ENABLED"):
+                continue
+            eng = getattr(self.config, name, None)
+            if eng is None:
+                continue
+            eng.enabled = True
+            raw = (_os.environ.get(f"CODERAI_{name.upper()}_CONFIG") or "").strip()
+            if not raw:
+                continue
+            try:
+                forwarded = json.loads(raw)
+            except Exception as exc:
+                print(f"[seed] CODERAI_{name.upper()}_CONFIG is not valid JSON: {exc}",
+                      flush=True)
+                continue
+            if not isinstance(forwarded, dict):
+                continue
+            for k, v in forwarded.items():
+                if k in ("enabled", "install_dir", "repo_url", "service_url",
+                         "host", "port", "model_path", "auto_build"):
+                    continue            # local to whichever machine set them
+                if hasattr(eng, k):
+                    setattr(eng, k, v)
+            print(f"[seed] {name} enabled with {sorted(forwarded)}", flush=True)
+
     def _seed_models_from_env(self) -> None:
         """Register models named by CODERAI_SEED_MODELS (JSON list of entries).
 

docs/remote-execution.md

@@ -654,6 +654,10 @@ volume at once can corrupt it — coderai's writes are write-once (content-addre
 adapter blobs, named model files), but keep that in mind before pointing many
 pods at one volume for anything else.
 
+**Engine models live here or nowhere.** DeepSeek-V4 on ds4 is ~154 GB, Kimi-K3
+is terabytes; the engines pod image loads them from `volume_path` on the volume
+rather than downloading. See "Engine pods" in `docs/runpod.md` — and its warning.
+
 #### Dependencies on the volume, small image in the registry
 
 A capability image is ~7 GB and nearly all of it is torch. With a volume those

docs/runpod.md

@@ -173,6 +173,73 @@ entirely — CoderAI logs `Model 'X' is RunPod-served — skipping local downloa
 and never fetches weights. So a RunPod model can be configured, enabled and served
 on a machine that has neither the disk space nor the GPU for it.
 
+### Engine pods — ds4, colibri, k3 on a rented GPU
+
+> ⚠ **The weights are the cost, not the GPU.** Models on these engines are
+> 100 GB and up — DeepSeek-V4 is ~154 GB, Kimi-K3 is measured in terabytes.
+> A pod with **no network volume downloads all of it on every cold start**,
+> which is typically one to two hours of rental per boot, before it answers
+> anything. The model page shows this warning the moment you pick one of these
+> engines; the provisioning log repeats it; the test run reports it. Attach a
+> network volume, and put the weights on it once.
+
+The native MoE engines run on RunPod through the **engines image**
+(`ghcr.io/nextime/coderai-engines`), a coderai pod that carries `ds4-server`,
+colibri's `colibri` / `deepseek_v4` / `kimi_k3` and kimi-k3-in-c's `k3`. They
+are **compiled in the image**, not copied from your machine: the local
+`ds4-server` is built for the one card here (sm_86) against the host's CUDA 13,
+and the local colibri has no GPU backend at all — neither would run on an A100.
+The image builds them in a CUDA 12.8 toolchain stage for every generation RunPod
+rents (sm_80 A100, sm_86, sm_89 L40S/4090, sm_90 H100, sm_120a RTX 50 / RTX PRO
+Blackwell) and a portable x86-64-v3 CPU baseline, then keeps only the binaries.
+
+```bash
+./packaging/runpod/build_capability_image.sh engines ghcr.io/nextime/coderai-engines:0.2.18
+```
+
+It stages the engine sources from `~/.coderai/{ds4,colibri,kimi-k3-in-c}` (or
+`CODERAI_DS4_DIR` etc.), so the pod runs the same engine code the local install
+does.
+
+**Naming the engine.** A RunPod-only model has `backend: runpod` — that slot is
+taken — so the engine goes on the runpod block, beside `vllm` and `llamacpp`:
+
+```json
+{ "path": "DeepSeek-V4-Pro-Q4K.gguf", "backend": "runpod",
+  "runpod": { "engine": "ds4", "network_volume_id": "abc123",
+              "volume_path": "models/DeepSeek-V4-Pro-Q4K.gguf",
+              "min_vram_gb": 80, "boot_timeout_s": 900, "load_timeout_s": 3600 } }
+```
+
+A *local* model pinned to `backend: ds4` / `colibri` / `k3` that bursts to
+RunPod carries the pin itself; either way it lands on the engines image with
+that engine switched on (`CODERAI_DS4_ENABLED=1`) and **this install's settings
+for it** forwarded as `CODERAI_DS4_CONFIG` — context, expert cache, extra args,
+the ds4 download variant — minus anything local (install dir, ports, paths).
+`ds4.auto_download` is forced on for the pod: a pod that cannot fetch its own
+weights answers nothing.
+
+**Where the weights come from, per engine:**
+
+| Engine | On the pod | Without `volume_path` |
+|---|---|---|
+| `ds4` | loads `volume_path` if set | runs `download_model.sh <model_variant>` into `<volume>/cache/ds4` (or the container disk — see the warning) |
+| `colibri` | loads the container directory at `volume_path` | **never downloads**: fails with "no model container resolved" |
+| `k3` | loads the checkpoint directory at `volume_path` | **never downloads**: the checkpoint is ~1.56 TB |
+
+`volume_path` is relative to the volume mount (`/workspace` by default) or
+absolute. Put the weights there once — from a pod, with `scp` through the RunPod
+console, or by letting one ds4 pod download and every later one attach.
+
+**ktransformers is not in the image.** It is a Python stack (kt-kernel +
+SGLang) for AMX CPUs, several GB in a venv of its own, and has never been built
+for a pod. A model pinned to `kt` is refused with that reason at provision time;
+serve it from a machine you own with the `host` backend instead.
+
+**The engines are CPU-hungry too.** colibri's DeepSeek-V4 and Kimi-K3 engines
+and `k3` stream experts through RAM by design (no CUDA path); pick a pod with
+the RAM they need, not just the VRAM — `min_vram_gb` says nothing about RAM.
+
 ---
 
 ## 3. Request flow

packaging/runpod/Dockerfile.capability-engines

+# CoderAI engines pod — ds4, colibri and kimi-k3-in-c, compiled for any pod GPU.
+#
+# The local install carries these engines as binaries built on THIS machine:
+# ds4-server for the one card here (sm_86) against the host's CUDA 13, colibri
+# without a GPU backend at all. Copying them into a pod image would ship an
+# engine that fails at first kernel launch on an A100 and cannot find
+# libcudart.so.13 anywhere. So this image builds them itself, in a CUDA 12.8
+# toolchain stage, for every GPU generation RunPod rents (Ampere through
+# Blackwell) and for a portable x86-64-v3 CPU baseline — then keeps only the
+# binaries and the scripts beside them. The toolchain never reaches the image.
+#
+# The runtime side is the ordinary full core: torch's cu128 wheels carry the
+# libcudart.so.12 / libcublas.so.12 the engines link, so nothing else is added.
+#
+# Built by build_capability_image.sh engines, which first stages the engine
+# source trees into packaging/runpod/engines-src/ (they live outside the repo).
+
+ARG CORE_BASE=coderai-capability-base:latest
+ARG CUDA_DEVEL=nvidia/cuda:12.8.1-devel-ubuntu24.04
+
+# ---------------------------------------------------------------- build stage
+FROM ${CUDA_DEVEL} AS engines
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        build-essential git python3 ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+# SASS for every card RunPod rents. colibri's Makefile spells its own list as
+# CUDA_ARCH=portable (plus a PTX fallback); ds4 takes the raw flags, and its
+# Blackwell kernels use block-scaled MXF4 MMA, which exists only on the
+# arch-specific sm_120a target — plain sm_120 fails in ptxas, and so would a
+# PTX fallback, so there is none: a card newer than this list gets no ds4.
+ENV GENCODE="-gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_89,code=sm_89 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_120a,code=sm_120a"
+
+WORKDIR /src
+COPY packaging/runpod/engines-src/ /src/
+COPY packaging/patch-k3.py /src/patch-k3.py
+
+# ds4: CUDA, all archs, portable CPU flags. `-B` because the staged tree may
+# carry the host's .o files, which would "already be built" for the wrong card.
+RUN set -eux; cd /src/ds4; \
+    make -B ds4-server CUDA_ARCH=sm_80 NVCC_ARCH_FLAGS="${GENCODE}" \
+        NATIVE_CPU_FLAG=-march=x86-64-v3 -j"$(nproc)"; \
+    ./ds4-server --help >/dev/null 2>&1 || true; \
+    ls -la ds4-server
+
+# colibri: the GLM engine with the CUDA backend; DeepSeek-V4 and Kimi-K3 are
+# CPU engines by design (their Makefiles have no CUDA path) and only need the
+# portable ARCH so they run on whatever CPU the pod has.
+RUN set -eux; cd /src/colibri/c; \
+    rm -f *.o .build-config; \
+    make colibri CUDA=1 CUDA_ARCH=portable ARCH=x86-64-v3 -j"$(nproc)"; \
+    make deepseek-v4 ARCH=x86-64-v3 -j"$(nproc)"; \
+    make kimi_k3 ARCH=x86-64-v3 -j"$(nproc)"; \
+    ls -la colibri deepseek_v4 kimi_k3
+
+# kimi-k3-in-c: coderai's resident serve-loop patch, then the portable build.
+RUN set -eux; cd /src/kimi-k3-in-c; \
+    rm -rf build; \
+    python3 /src/patch-k3.py src/cli/k3_run.c; \
+    make portable -j"$(nproc)"; \
+    ls -la bin/k3
+
+# What ships: binaries, scripts, docs — no objects, no toolchain, no git.
+RUN set -eux; mkdir -p /out/ds4 /out/colibri /out/kimi-k3-in-c; \
+    cd /src/ds4 && cp -a ds4-server download_model.sh README.md LICENSE /out/ds4/ 2>/dev/null || true; \
+    cd /src/colibri && cp -a . /out/colibri/ && find /out/colibri -name '*.o' -delete && rm -rf /out/colibri/.git; \
+    cd /src/kimi-k3-in-c && mkdir -p /out/kimi-k3-in-c/bin && cp -a bin/k3 /out/kimi-k3-in-c/bin/ \
+        && cp -a README.md LICENSE Makefile /out/kimi-k3-in-c/ 2>/dev/null || true; \
+    du -sh /out/*
+
+# -------------------------------------------------------------- runtime stage
+FROM ${CORE_BASE}
+
+ARG PROFILE=engines
+ARG TORCH_INDEX=https://download.pytorch.org/whl/cu128
+ENV PROFILE=${PROFILE}
+ARG IMAGE_TAG=unknown
+ENV CODERAI_IMAGE_TAG=${IMAGE_TAG}
+
+WORKDIR /opt/coderai/app
+COPY packaging/runpod/profiles/ /tmp/profiles/
+RUN --mount=type=cache,target=/root/.cache/pip set -eux; \
+    python -m pip install --extra-index-url ${TORCH_INDEX} -r /tmp/profiles/${PROFILE}.txt; \
+    python -c "import torch; print('torch', torch.__version__, 'cuda', torch.version.cuda)"; \
+    rm -rf /tmp/profiles
+
+COPY --from=engines /out/ds4/ /opt/coderai/ds4/
+COPY --from=engines /out/colibri/ /opt/coderai/colibri/
+COPY --from=engines /out/kimi-k3-in-c/ /opt/coderai/kimi-k3-in-c/
+
+# Where the workers look (codai/api/{ds4,colibri,k3}_worker.default_install_dir),
+# and the CUDA runtime the engines link, from torch's own wheels.
+ENV CODERAI_DS4_DIR=/opt/coderai/ds4 \
+    CODERAI_COLIBRI_DIR=/opt/coderai/colibri \
+    CODERAI_K3_DIR=/opt/coderai/kimi-k3-in-c \
+    LD_LIBRARY_PATH=/usr/local/lib/python3.12/site-packages/nvidia/cuda_runtime/lib:/usr/local/lib/python3.12/site-packages/nvidia/cublas/lib:${LD_LIBRARY_PATH}
+
+# Fail the BUILD if a binary is missing a library: the alternative is a pod that
+# boots, takes the request, and dies at the engine's first exec.
+RUN set -eux; for b in /opt/coderai/ds4/ds4-server /opt/coderai/colibri/c/colibri \
+        /opt/coderai/colibri/c/deepseek_v4 /opt/coderai/colibri/c/kimi_k3 \
+        /opt/coderai/kimi-k3-in-c/bin/k3; do \
+        test -x "$b"; \
+        if ldd "$b" | grep -q 'not found'; then echo "ERROR: $b:" >&2; ldd "$b" | grep 'not found' >&2; exit 1; fi; \
+        echo "ok: $b"; \
+    done; \
+    n=$(grep -a -o '\.target sm_[0-9]*a\?' /opt/coderai/ds4/ds4-server | sort -u | wc -l); \
+    echo "ds4-server SASS targets: $n"; test "$n" -ge 5
+
+COPY . /opt/coderai/app
+RUN python -c "import codai.api.app; print('coderai imports OK for profile ${PROFILE}')"
+
+EXPOSE 8000
+HEALTHCHECK --interval=30s --timeout=5s --start-period=120s \
+    CMD curl -fsS http://127.0.0.1:8000/healthz || exit 1
+
+COPY packaging/runpod/boot.sh /opt/coderai/boot.sh
+RUN chmod +x /opt/coderai/boot.sh
+CMD ["/opt/coderai/boot.sh"]

packaging/runpod/build_all_capability_images.sh

@@ -15,7 +15,7 @@ VERSION="${VERSION:-$(python3 -c 'import re,pathlib; print(re.search(r"__version
 # A profile's extra-venv specs live beside it as <profile>.venv-<name>.txt, so
 # a bare *.txt glob would invent a profile called "audio.venv-audiocraft" and
 # try to build an image for it.
-PROFILES="${PROFILES:-$(ls packaging/runpod/profiles/*.txt | xargs -n1 basename | sed 's/\.txt$//' | grep -v '^core$' | grep -v '\.venv-' | tr '\n' ' ')}"
+PROFILES="${PROFILES:-$(ls packaging/runpod/profiles/*.txt | xargs -n1 basename | sed 's/\.txt$//' | grep -v '^core' | grep -v '\.venv-' | tr '\n' ' ')}"
 
 if [[ -z "$NS" ]]; then
     echo "usage: $0 <registry/namespace>   e.g. ghcr.io/nextime" >&2

packaging/runpod/build_capability_image.sh

@@ -64,9 +64,41 @@ else
     echo "==> reusing the shared core $CORE_BASE"
 fi
 
-echo "==> building $TAG from profile '$PROFILE'"
+# A profile may name its own Dockerfile (`<profile>.dockerfile`) when the generic
+# one cannot express it — the engines image compiles C/CUDA in a toolchain stage.
+DOCKERFILE=packaging/runpod/Dockerfile.capability
+if [[ -f "$PROFILES_DIR/$PROFILE.dockerfile" ]]; then
+    DOCKERFILE="packaging/runpod/$(head -1 "$PROFILES_DIR/$PROFILE.dockerfile")"
+fi
+
+# The engine sources live outside the repo (the same trees the local install
+# builds from: ~/.coderai/ds4, colibri, kimi-k3-in-c). Stage them into the build
+# context without weights, objects or git history, so the image builds the SAME
+# code the local engines run, for every pod GPU instead of this one.
+if [[ "$PROFILE" == "engines" ]]; then
+    SRC=packaging/runpod/engines-src
+    rm -rf "$SRC"; mkdir -p "$SRC"
+    for pair in "ds4:${CODERAI_DS4_DIR:-$HOME/.coderai/ds4}" \
+                "colibri:${CODERAI_COLIBRI_DIR:-$HOME/.coderai/colibri}" \
+                "kimi-k3-in-c:${CODERAI_K3_DIR:-$HOME/.coderai/kimi-k3-in-c}"; do
+        name="${pair%%:*}"; dir="${pair#*:}"
+        if [[ ! -d "$dir" ]]; then
+            echo "engine source tree missing: $dir (set CODERAI_$(echo "$name" | tr a-z- A-Z_ | sed 's/KIMI_K3_IN_C/K3/')_DIR)" >&2
+            exit 2
+        fi
+        rsync -a --exclude '.git/' --exclude 'build/' --exclude '*.o' --exclude '*.gguf' \
+              --exclude '*.gguf.*' --exclude 'gguf/' --exclude '*.safetensors' --exclude '*.bin' \
+              --exclude 'glm52_i4/' --exclude 'web/node_modules/' --exclude 'containers/' \
+              --exclude '/ds4flash.gguf' --exclude '/ds4-server' --exclude '/ds4' --exclude '/ds4-agent' \
+              --exclude '/ds4-bench' --exclude '/ds4-eval' --exclude '/speed-bench' \
+              "$dir/" "$SRC/$name/"
+        echo "    staged $name from $dir ($(du -sh "$SRC/$name" | cut -f1))"
+    done
+fi
+
+echo "==> building $TAG from profile '$PROFILE' ($DOCKERFILE)"
 # BuildKit for the pip cache mount: nine profiles, one torch download.
-DOCKER_BUILDKIT=1 docker build -f packaging/runpod/Dockerfile.capability \
+DOCKER_BUILDKIT=1 docker build -f "$DOCKERFILE" \
     --build-arg "PROFILE=$PROFILE" \
     --build-arg "IMAGE_TAG=$TAG" \
     --build-arg "CORE_BASE=$CORE_BASE" \

packaging/runpod/profiles/engines.dockerfile

+Dockerfile.capability-engines

packaging/runpod/profiles/engines.txt

+# Native MoE engines — ds4 (DeepSeek-V4, CUDA), colibri (GLM-5.2 CUDA;
+# DeepSeek-V4 and Kimi-K3 CPU) and kimi-k3-in-c — a coderai pod that serves
+# frontier-size models through the same C engines the local install uses.
+#
+# The engines are not pip packages: they are compiled in a CUDA 12.8 toolchain
+# stage of Dockerfile.capability-engines (see the `engines.dockerfile` marker)
+# for EVERY pod GPU generation (sm_80/86/89/90/120), not the one card this
+# machine has, and copied in beside their download scripts. The Python side
+# needs nothing beyond the core: the engines bring their own tokenizers.
+#
+# THE WEIGHTS ARE THE COST. DeepSeek-V4 is ~154 GB, Kimi-K3 is measured in
+# terabytes. Put them on a network volume (docs/runpod.md, "Engine pods") —
+# a pod without one downloads all of it on every cold start.
+sentencepiece>=0.1.99
+protobuf>=3.20.0

packaging/runpod/publish_capability_images.sh

@@ -22,7 +22,7 @@ HERE="packaging/runpod"
 NS="${NS:-ghcr.io/nextime}"
 REGISTRY="${NS%%/*}"
 VERSION="${VERSION:-$(python3 -c 'import re,pathlib; print(re.search(r"__version__ = \"([^\"]+)\"", pathlib.Path("codai/__init__.py").read_text()).group(1))')}"
-PROFILES="${PROFILES:-$(ls $HERE/profiles/*.txt | xargs -n1 basename | sed 's/\.txt$//' | grep -v '^core$' | tr '\n' ' ')}"
+PROFILES="${PROFILES:-$(ls $HERE/profiles/*.txt | xargs -n1 basename | sed 's/\.txt$//' | grep -v '^core' | grep -v '\.venv-' | tr '\n' ' ')}"
 REBUILD="${REBUILD:-0}"
 NO_PUSH="${NO_PUSH:-0}"
 YES="${YES:-0}"

tests/test_burst_placement.py

@@ -872,3 +872,116 @@ def test_a_multi_gpu_pod_is_priced_and_sharded_as_a_whole(monkeypatch):
     assert "--split-mode layer" in rw._llamacpp_docker_args(ml, "m", {})
     # A single card asks for neither: no needless flags on the common case.
     assert "tensor-parallel" not in rw._vllm_docker_args(m1, "org/m", {"path": "org/m"})
+
+
+def test_an_engine_model_gets_the_engines_image_and_its_engine_switched_on(monkeypatch):
+    """The local ds4-server is an sm_86 build against CUDA 13 and the local
+    colibri has no GPU backend: neither can be copied into a pod. The engines
+    image builds them for every pod GPU, and a model that names an engine —
+    on its runpod block, since `backend: runpod` takes the pin slot — lands on
+    that image with the engine enabled and this install's settings for it."""
+    import json
+    import codai.api.runpod_worker as rw
+    from codai.api.runpod_worker import pod_plan, parse_model_runpod
+
+    class _Cfg:
+        ctx = 65536; ssd_streaming = True; extra_args = "--foo"; extra_env = ""
+        model_variant = "q4-imatrix"; auto_download = False
+        expert_cache_reserve_gb = 0; model_id = "deepseek-v4"
+        install_dir = "/home/me/.coderai/ds4"; port = 1234
+    class _Root:
+        ds4 = _Cfg()
+    class _CM:
+        config = _Root()
+    monkeypatch.setattr("codai.admin.routes.config_manager", _CM(), raising=False)
+
+    entry = {"path": "/AI/gguf/DeepSeek-V4-Q4.gguf", "model_type": "text_models",
+             "backend": "runpod", "weights_gb": 154,
+             "ds4": {"ssd_streaming": True}}
+    mcfg = parse_model_runpod({"engine": "ds4"})
+    plan = pod_plan(mcfg, "deepseek", entry=entry)
+    assert plan["engine"] == "coderai"
+    assert plan["image"].endswith("coderai-engines:latest")
+    env = plan["env"]
+    assert env["CODERAI_DS4_ENABLED"] == "1"
+    fwd = json.loads(env["CODERAI_DS4_CONFIG"])
+    assert fwd["ctx"] == 65536 and fwd["ssd_streaming"] is True
+    assert fwd["auto_download"] is True, "a pod that cannot fetch answers nothing"
+    assert "install_dir" not in fwd and "port" not in fwd
+
+    seed = json.loads(env["CODERAI_SEED_MODELS"])[0]
+    assert seed["backend"] == "ds4", "the pod must run ds4 for it, not transformers"
+    assert seed["ds4"] == {"ssd_streaming": True}
+    assert "weights_gb" not in seed          # sizes the disk here, not the model there
+    assert seed["path"] == "DeepSeek-V4-Q4.gguf", "a local path means nothing there"
+
+    # Weights already on the volume: the seed points straight at them.
+    mcfg = parse_model_runpod({"engine": "ds4", "network_volume_id": "vol1",
+                               "volume_path": "models/DeepSeek-V4-Q4.gguf"})
+    seed = json.loads(pod_plan(mcfg, "deepseek", entry=entry)["env"]["CODERAI_SEED_MODELS"])[0]
+    assert seed["path"] == "/workspace/models/DeepSeek-V4-Q4.gguf"
+
+    # A local model pinned to the engine itself, bursting: same answer.
+    local = {"path": "/AI/glm52", "model_type": "text_models", "backend": "colibri"}
+    plan = pod_plan(parse_model_runpod({"network_volume_id": "v", "volume_path": "glm52"}),
+                    "glm", entry=local)
+    assert plan["image"].endswith("coderai-engines:latest")
+    assert plan["env"]["CODERAI_COLIBRI_ENABLED"] == "1"
+    assert json.loads(plan["env"]["CODERAI_SEED_MODELS"])[0]["backend"] == "colibri"
+
+    # The other pins describe THIS machine and are stripped from the seed.
+    vk = {"path": "org/m", "model_type": "text_models", "backend": "vulkan"}
+    seed = json.loads(pod_plan(parse_model_runpod({"engine": "coderai", "image": "x"}),
+                               "m", entry=vk)["env"]["CODERAI_SEED_MODELS"])[0]
+    assert "backend" not in seed
+
+    # ktransformers has no pod image, and says so instead of shipping a pod
+    # that cannot run it.
+    import pytest
+    with pytest.raises(RuntimeError, match="ktransformers"):
+        pod_plan(parse_model_runpod({"engine": "kt"}), "m",
+                 entry={"path": "org/m", "model_type": "text_models"})
+
+    # The volume env points ds4's downloader at the volume too.
+    assert rw.volume_env("/workspace")["DS4_GGUF_DIR"] == "/workspace/cache/ds4"
+
+
+def test_a_pod_switches_its_engine_on_from_env_and_takes_the_settings(tmp_path, monkeypatch):
+    import json
+    from codai.config import ConfigManager
+
+    monkeypatch.setenv("CODERAI_DS4_ENABLED", "1")
+    monkeypatch.setenv("CODERAI_DS4_CONFIG", json.dumps(
+        {"ctx": 65536, "auto_download": True, "install_dir": "/nope",
+         "port": 9, "not_a_field": 1}))
+    cm = ConfigManager(str(tmp_path))
+    cm.load()
+    assert cm.config.ds4.enabled is True
+    assert cm.config.ds4.ctx == 65536 and cm.config.ds4.auto_download is True
+    assert cm.config.ds4.install_dir != "/nope" and cm.config.ds4.port != 9
+    assert not hasattr(cm.config.ds4, "not_a_field")
+    assert cm.config.colibri.enabled is False       # only what was asked for
+    on_disk = json.loads((tmp_path / "config.json").read_text())
+    assert on_disk.get("ds4", {}).get("enabled") is not True
+
+
+def test_the_engines_image_is_built_from_source_for_every_pod_gpu():
+    """Shipping this machine's binaries would ship an sm_86-only ds4 linked
+    to CUDA 13. The image compiles in a CUDA 12.8 stage for Ampere through
+    Blackwell, and the build fails if a binary cannot find its libraries."""
+    from pathlib import Path
+    from codai.api.runpod_worker import PUBLISHED_CAPABILITY_IMAGES, default_capability_image
+    assert "engines" in PUBLISHED_CAPABILITY_IMAGES
+    assert default_capability_image("engines").endswith("coderai-engines:latest")
+    prof = Path("packaging/runpod/profiles")
+    assert (prof / "engines.txt").exists()
+    assert (prof / "engines.dockerfile").read_text().strip() == "Dockerfile.capability-engines"
+    df = Path("packaging/runpod/Dockerfile.capability-engines").read_text()
+    for arch in ("sm_80", "sm_86", "sm_89", "sm_90", "sm_120"):
+        assert arch in df
+    assert "CUDA_ARCH=portable" in df and "x86-64-v3" in df
+    assert "patch-k3.py" in df and "not found" in df
+    for var in ("CODERAI_DS4_DIR", "CODERAI_COLIBRI_DIR", "CODERAI_K3_DIR"):
+        assert var in df
+    sh = Path("packaging/runpod/build_capability_image.sh").read_text()
+    assert ".dockerfile" in sh and "engines-src" in sh