- 20 Sep, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authoredcluster: mDNS discovery with a shared token, prefix-cache routing, Prometheus /metrics + per-key usage, node logs and re-push on return, GGUF row split, cosign-signed images - codai/cluster/discovery.py: _coderai._tcp announce + browse (zeroconf), HMAC fingerprint of cluster.token as membership; members auto-join as nodes, listed nodes with a blank key get the shared token; nodes and engines accept the cluster token (front /cluster/*, engine BearerAuth); env overrides for capability images (CODERAI_CLUSTER_TOKEN/DISCOVERY/NODE_NAME/ADVERTISE_HOST); coderai-docker --host-network - frontproxy/affinity.py + router prefer=: a conversation goes back to the engine/node that served it when the model is resident in several places; engine_for_model now picks the least busy holder - frontproxy/metrics.py: /metrics (requests per key/model/kind/status/engine, engines/nodes up, VRAM, in-flight, RPC, discovery, RunPod spend), usage tables on the Cluster page, server.metrics_public - node logs: /cluster/logs on the node, /admin/api/cluster/logs on the head, log viewer on the Cluster page; a node that comes back gets its assigned models pushed again - per-model split_mode=row (llama.cpp tensor parallel over local + RPC devices); vLLM TP may span nodes (documented) - packaging/sign-images.sh + packaging/cosign.pub: all 21 GHCR images signed - docs/cluster.md §§ discovery, 7 prefix routing, 9 operating, 10 accelerators evaluation; tests/test_cluster_ops.py (11) Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKm7Z56iKjHgNkHc6VhWKz
-
- 19 Sep, 2026 17 commits
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
never ship the local video_editor config (it carried an API token) or .claude settings; scrub them in the image overlay Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packages: install-from-packages guide, Windows installer (Docker Desktop + WSL2 launcher, Inno Setup build under wine) Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
host pools: keep the single-host wording and _run signature; start/stop commands never inherit across machines Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model 'components' (text_encoder / low_noise / vae → cluster node): the prompt is encoded where the text encoder is, the Wan 2.2 low-noise expert takes over at the expert boundary with the latents handed over, the VAE decodes where it lives and returns the finished video. The far side runs the ordinary generation endpoint with a _handoff block, so loading, acceleration and LoRAs happen as usual there; the scheduler is sliced to resume mid-loop (codai/cluster/components.py). Models page block + tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Synchronous data parallel through torch.distributed with a hand-rolled all-reduce of the adapter gradients (codai/cluster/ddp.py): the engine that takes the job is rank 0, sends the same job with the images inlined to each node in 'nodes' (or the model's lora_train_nodes), all ranks start from rank 0's init, average gradients every step, rank 0 saves. Peers write to scratch and are cancelled with the coordinator. All five in-process trainers (SD1.5, SDXL, Z-Image, flow DiT, Wan incl. 4-bit) take the hooks; two-process test. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model 'distribute': n images/videos, embedding and rerank lists, TTS sentences, transcriptions cut at silences, OCR batches are split over every engine that has the model — here and on cluster nodes — and merged back. Files a node generates are addressed through the head. Failed parts retry once elsewhere. Models page block + tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packaging: llama rebuild stage checks with ldd (libcuda is a runtime library), rpc-server script needs no import Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packaging: Dockerfile.update-llama — rebuild llama-cpp-python with GGML_RPC and ship rpc-server over an existing image Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang multi-node, pools, compat-filtered model form Several coderai installs as one: - Cluster nodes (cluster.nodes): another install is an engine of this front — polled on its /cluster/state, assigned models, pinned by name, routed by capability and load; entries pushed to /cluster/reload-config and localized there; Load/Unload reach it through token-guarded twins. Live add/remove. - llama.cpp RPC: codai/backends/ggml_rpc.py registers rpc-server endpoints through ctypes and gives every load an explicit device list (local cards, then the model's rpc_servers); the split and the VRAM pool see remote memory. cluster.rpc_servers runs this machine's rpc-server processes; packaging/build-rpc-server.sh builds it from the vendored llama.cpp; build.sh and the images build llama-cpp-python with GGML_RPC. - vLLM on Ray (pipeline parallel across nodes) and SGLang --nnodes: per-model vllm/kt blocks over the global config, node start/stop commands, ray head and worker mode in boot.sh. - Pools: several URLs per capability remote and several machines per host model, with failover. - Models page: engine/backend selects narrowed live to compatible pairs, impossible pairs refused; Settings → Cluster; /admin/cluster page. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
docs: nineteen images, their sizes and cores, the CUDA-13 vLLM trap, uv/apt/cache markers, direct TLS by default Co-Authored-By:
-
- 18 Sep, 2026 6 commits
-
-
Stefy Lanza (nextime / spora ) authored
PyPI's CDN throttles a machine that has pulled 100 GB of wheels in a day to ~400 KB/s; a venv rebuild that only needed a new apt layer above it then spends hours re-downloading torch. Cache mounts for pip and uv keep the wheels between builds. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Now that a coderai pod serves TLS on its public ip:port, the proxy's 100 s cutoff and body cap buy nothing for our images — and a voice clone already died on it. direct_tcp becomes three-state: unset means direct for coderai images and the proxy for vLLM / llama.cpp / custom (the proxy is the only TLS those have); true/false force it. Model form, save route, harness (--direct-tcp / --proxy) and docs follow. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two verification runs failed with a 404 issued the same second the pod was declared ready — tts on one card, ocr on another — and both passed unchanged on a rerun: RunPod's proxy answers 404 until its route settles, and a catalogue can lag its listing by a moment. A 404/502/503 from a base seen less than a minute ago is retried three times over ~20 s before it is returned. A base up for longer gets no such doubt. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod died with 'The NVIDIA driver on your system is too old (found version 12080)': vLLM 0.27.1 on PyPI pins torch 2.13+cu130. vLLM's own index has a +cu129 variant with torch's cu129 wheels, and CUDA 12.x minor-version compatibility runs a 12.9 runtime on a 12.8 driver. uv resolves it with --index-strategy unsafe-best-match, since the torch index also carries stale copies of ordinary packages. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
vLLM on the pod died with 'Engine core initialization failed. See root cause above' — and the root cause was above the six-line tail the 503 carried. Neither core has a C compiler; Triton JIT-compiles vLLM's (and SGLang's) kernels at run time and needs one. A <profile>.apt marker keeps runtime apt packages in the image; vllm and engines-kt get gcc. The worker's error now surfaces the lines that name an error first. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod terminated behind the pool's back — the harness reaping by API, a reclaimed spot machine — stayed 'healthy' until the next timed probe, and the next request failed on connection refused for something the pool could have known. The backend now discards it immediately and retries the request once on a fresh acquire; a second refusal is a real outage and raises. Co-Authored-By:
-
- 17 Sep, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai-llama came out at 11.6 GB — the CUDA wheel for five GPU generations on top of the full core — over RunPod's size line. The only thing the CUDA GGUF backend needed torch for was three questions about the card (is there one, total memory, free memory); gpu_probe answers them from torch when present and NVML otherwise. The image drops to the light core plus NVIDIA's runtime, cuBLAS and NCCL wheels. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai-text was transformers only, so a GGUF or a vLLM-served repo still went to an upstream image: no TLS on the direct path, no /boot, no local LoRAs, nothing on the Tasks page. coderai-llama compiles llama-cpp-python with CUDA for every pod GPU on the full core; coderai-vllm is vLLM in its own uv-resolved venv on the light core, found by the worker through CODERAI_VLLM_VENV, with the local vLLM settings forwarded. The upstream engines stay selectable as vllm / llamacpp. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
So callers and tests that patch requests.get/post keep working; only https-to-a-bare-IP takes the pinned path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
direct_tcp traded the proxy's 100 s cutoff for plain HTTP. Now a coderai pod on that path brings its own TLS: a per-install CA (openssl, kept beside the pod registry) signs a short-lived leaf per pod, sent as env like the token; boot.sh and the venv-on-volume bootstrap start uvicorn on it. This side verifies https-to-a-bare-IP against that CA only — pod_http, a requests shim every pod-facing call now goes through. No hostname check: the IP is unknown at issue time, and a foreign cert fails the chain before a name is compared. Proven end to end in a test with a real https server. vLLM/llama.cpp images stay plain on this path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Every pod request went through <pod>-<port>.proxy.runpod.net, which sits behind Cloudflare: a 100 s idle limit per request and a body-size cap. A slow non-streaming render can be cut off with a 524 while the pod is still working. direct_tcp (off by default) exposes the port as tcp and talks to the public ip:port the runtime reports — plain HTTP, no limits. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
pip spent an hour and gigabytes of RAM on sglang-kt's forty exact pins without installing a byte; uv resolved the same set in seconds. A <profile>.venv-<name>.uv marker opts a venv in. Also drop __pycache__ from baked venvs. Co-Authored-By:
-
- 16 Sep, 2026 9 commits
-
-
Stefy Lanza (nextime / spora ) authored
kt was refused because it looked unshippable: a Python stack pinning its own torch, thought to be AMX-only. kt-kernel 0.7 ships prebuilt manylinux wheels that dispatch at runtime (AMX, AVX-512, AVX2 llamafile, AMD BLIS), and sglang-kt pins the matching torch — so it is a venv on the light core like the other split images. The worker now launches SGLang from that venv (CODERAI_KT_VENV / ktransformers.venv) instead of assuming coderai's interpreter has it, and accepts a HF repo id, which SGLang downloads into the volume's cache. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The engines are C binaries and coderai only drives them; the full core's torch was dead weight. All they link is the CUDA runtime and cuBLAS, which NVIDIA ships as wheels at the 12.8 they were compiled against. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Twice the build disk filled with 500+ GB of layers no image referenced, pinned by mount records of containers the daemon no longer knew about, while docker system df reported 90 GB. Nothing in docker can see them. This walks imagedb/layerdb/mounts the way a fresh daemon would, reports the stale set, and with --fix (daemon stopped) removes it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The local ds4-server is an sm_86 build against CUDA 13 and the local colibri has no GPU backend: neither could be copied into a pod. The new engines image compiles both, plus kimi-k3-in-c, in a CUDA 12.8 stage for sm_80/86/89/90/120a (ds4's Blackwell kernels need the arch-specific target; plain sm_120 fails in ptxas) on a portable x86-64-v3 baseline, and fails the build if a binary cannot find its libraries. A RunPod-only model names the engine on its runpod block (backend: runpod takes the pin slot); a local model pinned to the engine and bursting carries the pin. Either way the pod arrives with the engine enabled and this install's settings forwarded (CODERAI_<ENG>_CONFIG), and volume_path points it at weights already on the network volume — the only sane way to run 150 GB+ models. The model page warns loudly the moment one of these engines is picked. ktransformers is refused with a reason: it has no pod image. Also: the batch build/publish scripts no longer list core-light and venv profiles as images. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod-only was an entry in the compute-backend dropdown, between OpenCL and wav2vec2 — not where anyone deciding WHERE a model runs would look. The Placement section now asks 'Runs on' (this machine / RunPod only / a machine of yours) and the RunPod block sits under it; the radios and the backend select are one stored value, kept in sync. The host backend existed only as JSON. It now has its fields on the form, a save handler, and — like a pod — no local weight or VRAM bookkeeping. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
gpu_count was passed to RunPod's gpuCount and then ignored: the price ceiling checked one card, min_vram_gb one card's memory, and the engine loaded on GPU 0 while the rest sat idle. Now the pod is ranked as a whole (price x N, VRAM x N, '2x A100'), vLLM gets --tensor-parallel-size N and llama.cpp --split-mode layer. Exposed on the model page, clamped 1-8. Requests forwarded to a pod or host never appeared on the Tasks page, so a running remote job was invisible from the admin. _forward now registers a 'remote' task whose message names the pod it landed on. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
weights_gb, on the model entry or its runpod block, is taken as-is and the HuggingFace estimate is skipped. It is the only honest answer for a URL download, an upload or a local file — none of which the estimator can see — and for an HF repo it beats a heuristic that deliberately errs high (SDXL: estimated 35 GB, actually ~7 for one precision). A nonsense value is ignored rather than trusted, and the field is never seeded to a pod: it describes the pod's disk, not the model. /v1/models/test/state now reports the disk a pod would be rented with, the weight size used, and whether that size was stated or estimated — decided before any money is spent. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
remote-execution.md: the capability images are two kinds now — ten on the full core and five on a torch-free light core for stacks that cannot share a venv — and the doc says which, why each split was measured rather than assumed, what the .venv-*.check scripts catch that a bare import cannot, and why the light core exists (RunPod refuses images above ~10 GB; a specialised image on the full core carried 7 GB of torch it never used). The 'speaker rides stt' line was wrong — that image shipped nothing for speaker work — and is fixed. A new section covers what travels to a pod as environment: the HF token, licence acceptances, baked-venv paths, OCR engine gates. Surya on a pod is documented as served by docTR, with the reason. The host backend gets its own section: always-on and on-demand shapes, what it deliberately lacks, and service_token. runpod.md: the size line with the numbers behind it, EXITED as a dead pod rather than a slow pull, container-disk sizing from HF metadata, and the field table corrected — boot_timeout_s is 900, container_disk_gb grows to fit. README: the fifteen images and the host backend, one line each. Co-Authored-By:
-