docs: nineteen images, their sizes and cores, the CUDA-13 vLLM trap,...

docs: nineteen images, their sizes and cores, the CUDA-13 vLLM trap, uv/apt/cache markers, direct TLS by default
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKm7Z56iKjHgNkHc6VhWKz

README.md

@@ -92,11 +92,20 @@ Plus:
 - **A pod that says what it is doing**: each pod logs timestamped boot phases from the first
   instant the container exists and serves the record at `/boot` — RunPod publishes no pod-log
   API, so the pod reports on itself
-- **Fifteen capability images, every model type served**: ten ordinary ones plus five for
+- **Nineteen capability images, every model type served**: ten ordinary ones, five for
   stacks that cannot share a venv (pyannote, coqui XTTS, NeMo, CrisperWhisper, PaddleOCR),
-  each carrying its own torch on a torch-free core so it stays under RunPod's image size
-  limit. Fourteen have completed a real inference on a rented GPU; `faceswap` is the one
-  that needs real faces to test
+  each carrying its own torch on a torch-free core, and four more for the LLM servers and
+  native engines (below). Sixteen have completed a real inference on a rented GPU on the
+  current build; `faceswap` needs real faces to test and the engine images need 100 GB+
+  weights on a volume
+- **The LLM servers as coderai pods**: `coderai-llama` (llama-cpp-python compiled with CUDA
+  for every pod GPU generation, 5.3 GB) and `coderai-vllm` (vLLM in its own venv, 12.4 GB)
+  serve a GGUF or an HF repo with everything a coderai pod gives — TLS, `/boot`, seeding,
+  local LoRAs, the Tasks page — while the upstream `vllm`/`llamacpp` images stay selectable
+- **No Cloudflare in the way**: a coderai pod is reached at its public ip:port over TLS it
+  brings itself (a per-install CA signs a certificate per pod), so there is no 100 s proxy
+  cutoff — a voice clone that died at 524 through the proxy passes direct. The proxy remains
+  the default for upstream images, which cannot take a certificate
 - **The native MoE engines on rented GPUs**: an `engines` image (2.4 GB) compiles ds4, colibri
   and kimi-k3-in-c for every pod GPU generation (Ampere → Blackwell) rather than copying this
   machine's builds, and `engines-kt` carries ktransformers (SGLang + kt-kernel) in its own

docs/remote-execution.md

@@ -289,13 +289,30 @@ so the toolchain never ships. `audio` does this for deepfilterlib's Rust
 extensions, and costs ~50 MB more than a profile that needs nothing.
 
 Profiles live in `packaging/runpod/profiles/` — `core.txt` (what any pod needs to
-boot) plus one file per capability. There are fifteen images, and they come in
-two kinds:
-
-| image | serves | core |
-|---|---|---|
-| `images`, `video`, `tts`, `stt`, `text`, `voice`, `audio`, `embeddings`, `ocr`, `faceswap` | one capability, in the main venv | full (torch + CUDA) |
-| `speaker`, `tts-xtts`, `stt-nemo`, `stt-crisper`, `ocr-paddle` | a stack that **cannot** share the main venv | light (no torch) |
+boot) plus one file per capability. There are nineteen images, in three kinds:
+
+| image | serves | core | size |
+|---|---|---|---|
+| `images`, `video`, `tts`, `stt`, `text`, `voice`, `audio`, `embeddings`, `ocr`, `faceswap` | one capability, in the main venv | full (torch + CUDA) | 9–10 GB |
+| `speaker`, `tts-xtts`, `stt-nemo`, `stt-crisper`, `ocr-paddle` | a stack that **cannot** share the main venv | light (no torch) + its own venv | 7–8 GB |
+| `llama`, `vllm`, `engines`, `engines-kt` | an LLM server or the native MoE engines as a coderai pod | light + a venv or C binaries | 2.4 / 5.3 / 11 / 12.4 GB |
+
+RunPod refuses an image above a size it does not document; 12.4 GB has booted,
+14.5 GB was refused. A stack built on a recent torch (2.9+, cudnn 9) is about
+3 GB heavier than one on torch 2.5, which is why the vLLM image is the largest.
+
+Three more per-profile markers exist for the venv images: `<profile>.venv-<name>.uv`
+resolves that venv with **uv** instead of pip (pip spent an hour on
+sglang-kt's forty exact pins without installing a byte), `<profile>.apt` lists
+apt packages kept **at run time** (a C compiler: Triton JIT-compiles vLLM's and
+SGLang's kernels on the pod and dies without one), and wheels are cached across
+builds (PyPI's CDN throttles a machine that has pulled 100 GB in a day).
+
+One trap worth naming: **PyPI's vLLM wheel is a CUDA 13 build**, and RunPod's
+hosts run CUDA 12.8 drivers — the pod dies with *"The NVIDIA driver on your
+system is too old (found version 12080)"*. The `vllm` image installs vLLM's own
+`+cu129` variant with torch's cu129 wheels; CUDA 12.x minor-version
+compatibility runs a 12.9 runtime on a 12.8 driver.
 
 The routers import lazily, so a pod only needs the libraries its own endpoints
 touch. The build fails if a profile is missing something the app imports, and

docs/runpod.md

@@ -215,8 +215,8 @@ local LoRAs, nothing on the Tasks page. Two more images run the same servers
 
 | `engine` | Image | What it is |
 |---|---|---|
-| `coderai-vllm` | `coderai-vllm` (light core + vLLM venv, ~10 GB) | vLLM driven by coderai's vLLM backend. Your local **vLLM settings travel** (`dtype`, `quantization`, `gpu_memory_utilization`, `max_num_seqs`, `extra_args`); tensor parallelism follows the pod's `gpu_count`. |
-| `coderai-llama` | `coderai-llama` (full core + llama-cpp-python CUDA, ~9 GB) | A GGUF on coderai's own CUDA backend — llama-cpp-python compiled for every pod GPU generation. Local LoRAs work. |
+| `coderai-vllm` | `coderai-vllm` (light core + vLLM `+cu129` venv, 12.4 GB — PyPI's wheel is CUDA 13 and dies on RunPod's 12.8 drivers) | vLLM driven by coderai's vLLM backend. Your local **vLLM settings travel** (`dtype`, `quantization`, `gpu_memory_utilization`, `max_num_seqs`, `extra_args`); tensor parallelism follows the pod's `gpu_count`. |
+| `coderai-llama` | `coderai-llama` (light core + llama-cpp-python CUDA, 5.3 GB; the GGUF backend asks NVML about the card, not torch) | A GGUF on coderai's own CUDA backend — llama-cpp-python compiled for every pod GPU generation. Local LoRAs work. |
 
 `coderai-text` remains the transformers pod (HF safetensors, 4/8-bit, PEFT
 adapters). Throughput under concurrency is vLLM's game on either image.
@@ -260,7 +260,8 @@ has; AMX hosts are the fast ones. `engine: kt` on the runpod block selects it;
 the pod launches SGLang from that venv (`CODERAI_KT_VENV`). The model path may
 be a HuggingFace repo id — SGLang downloads it into the volume's HF cache — or a
 directory on the volume; `ktransformers.extra_args` travels for the KT knobs
-(`--kt-cpuinfer`, `--kt-weight-path` …). Note the kt backend has not been run
+(`--kt-cpuinfer`, `--kt-weight-path` …). The image carries `gcc` because Triton
+JIT-compiles SGLang's kernels on the pod. Note the kt backend has not been run
 end to end on this machine either: the image is import-checked at build and
 `/healthz`-checked at boot, and the first real inference will be on a pod.