- 19 Sep, 2026 12 commits
-
-
Stefy Lanza (nextime / spora ) authoredhost pools: keep the single-host wording and _run signature; start/stop commands never inherit across machines Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKm7Z56iKjHgNkHc6VhWKz
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model 'components' (text_encoder / low_noise / vae → cluster node): the prompt is encoded where the text encoder is, the Wan 2.2 low-noise expert takes over at the expert boundary with the latents handed over, the VAE decodes where it lives and returns the finished video. The far side runs the ordinary generation endpoint with a _handoff block, so loading, acceleration and LoRAs happen as usual there; the scheduler is sliced to resume mid-loop (codai/cluster/components.py). Models page block + tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Synchronous data parallel through torch.distributed with a hand-rolled all-reduce of the adapter gradients (codai/cluster/ddp.py): the engine that takes the job is rank 0, sends the same job with the images inlined to each node in 'nodes' (or the model's lora_train_nodes), all ranks start from rank 0's init, average gradients every step, rank 0 saves. Peers write to scratch and are cancelled with the coordinator. All five in-process trainers (SD1.5, SDXL, Z-Image, flow DiT, Wan incl. 4-bit) take the hooks; two-process test. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model 'distribute': n images/videos, embedding and rerank lists, TTS sentences, transcriptions cut at silences, OCR batches are split over every engine that has the model — here and on cluster nodes — and merged back. Files a node generates are addressed through the head. Failed parts retry once elsewhere. Models page block + tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packaging: llama rebuild stage checks with ldd (libcuda is a runtime library), rpc-server script needs no import Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packaging: Dockerfile.update-llama — rebuild llama-cpp-python with GGML_RPC and ship rpc-server over an existing image Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang multi-node, pools, compat-filtered model form Several coderai installs as one: - Cluster nodes (cluster.nodes): another install is an engine of this front — polled on its /cluster/state, assigned models, pinned by name, routed by capability and load; entries pushed to /cluster/reload-config and localized there; Load/Unload reach it through token-guarded twins. Live add/remove. - llama.cpp RPC: codai/backends/ggml_rpc.py registers rpc-server endpoints through ctypes and gives every load an explicit device list (local cards, then the model's rpc_servers); the split and the VRAM pool see remote memory. cluster.rpc_servers runs this machine's rpc-server processes; packaging/build-rpc-server.sh builds it from the vendored llama.cpp; build.sh and the images build llama-cpp-python with GGML_RPC. - vLLM on Ray (pipeline parallel across nodes) and SGLang --nnodes: per-model vllm/kt blocks over the global config, node start/stop commands, ray head and worker mode in boot.sh. - Pools: several URLs per capability remote and several machines per host model, with failover. - Models page: engine/backend selects narrowed live to compatible pairs, impossible pairs refused; Settings → Cluster; /admin/cluster page. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
docs: nineteen images, their sizes and cores, the CUDA-13 vLLM trap, uv/apt/cache markers, direct TLS by default Co-Authored-By:
-
- 18 Sep, 2026 6 commits
-
-
Stefy Lanza (nextime / spora ) authored
PyPI's CDN throttles a machine that has pulled 100 GB of wheels in a day to ~400 KB/s; a venv rebuild that only needed a new apt layer above it then spends hours re-downloading torch. Cache mounts for pip and uv keep the wheels between builds. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Now that a coderai pod serves TLS on its public ip:port, the proxy's 100 s cutoff and body cap buy nothing for our images — and a voice clone already died on it. direct_tcp becomes three-state: unset means direct for coderai images and the proxy for vLLM / llama.cpp / custom (the proxy is the only TLS those have); true/false force it. Model form, save route, harness (--direct-tcp / --proxy) and docs follow. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two verification runs failed with a 404 issued the same second the pod was declared ready — tts on one card, ocr on another — and both passed unchanged on a rerun: RunPod's proxy answers 404 until its route settles, and a catalogue can lag its listing by a moment. A 404/502/503 from a base seen less than a minute ago is retried three times over ~20 s before it is returned. A base up for longer gets no such doubt. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod died with 'The NVIDIA driver on your system is too old (found version 12080)': vLLM 0.27.1 on PyPI pins torch 2.13+cu130. vLLM's own index has a +cu129 variant with torch's cu129 wheels, and CUDA 12.x minor-version compatibility runs a 12.9 runtime on a 12.8 driver. uv resolves it with --index-strategy unsafe-best-match, since the torch index also carries stale copies of ordinary packages. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
vLLM on the pod died with 'Engine core initialization failed. See root cause above' — and the root cause was above the six-line tail the 503 carried. Neither core has a C compiler; Triton JIT-compiles vLLM's (and SGLang's) kernels at run time and needs one. A <profile>.apt marker keeps runtime apt packages in the image; vllm and engines-kt get gcc. The worker's error now surfaces the lines that name an error first. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod terminated behind the pool's back — the harness reaping by API, a reclaimed spot machine — stayed 'healthy' until the next timed probe, and the next request failed on connection refused for something the pool could have known. The backend now discards it immediately and retries the request once on a fresh acquire; a second refusal is a real outage and raises. Co-Authored-By:
-
- 17 Sep, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai-llama came out at 11.6 GB — the CUDA wheel for five GPU generations on top of the full core — over RunPod's size line. The only thing the CUDA GGUF backend needed torch for was three questions about the card (is there one, total memory, free memory); gpu_probe answers them from torch when present and NVML otherwise. The image drops to the light core plus NVIDIA's runtime, cuBLAS and NCCL wheels. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai-text was transformers only, so a GGUF or a vLLM-served repo still went to an upstream image: no TLS on the direct path, no /boot, no local LoRAs, nothing on the Tasks page. coderai-llama compiles llama-cpp-python with CUDA for every pod GPU on the full core; coderai-vllm is vLLM in its own uv-resolved venv on the light core, found by the worker through CODERAI_VLLM_VENV, with the local vLLM settings forwarded. The upstream engines stay selectable as vllm / llamacpp. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
So callers and tests that patch requests.get/post keep working; only https-to-a-bare-IP takes the pinned path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
direct_tcp traded the proxy's 100 s cutoff for plain HTTP. Now a coderai pod on that path brings its own TLS: a per-install CA (openssl, kept beside the pod registry) signs a short-lived leaf per pod, sent as env like the token; boot.sh and the venv-on-volume bootstrap start uvicorn on it. This side verifies https-to-a-bare-IP against that CA only — pod_http, a requests shim every pod-facing call now goes through. No hostname check: the IP is unknown at issue time, and a foreign cert fails the chain before a name is compared. Proven end to end in a test with a real https server. vLLM/llama.cpp images stay plain on this path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Every pod request went through <pod>-<port>.proxy.runpod.net, which sits behind Cloudflare: a 100 s idle limit per request and a body-size cap. A slow non-streaming render can be cut off with a 524 while the pod is still working. direct_tcp (off by default) exposes the port as tcp and talks to the public ip:port the runtime reports — plain HTTP, no limits. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
pip spent an hour and gigabytes of RAM on sglang-kt's forty exact pins without installing a byte; uv resolved the same set in seconds. A <profile>.venv-<name>.uv marker opts a venv in. Also drop __pycache__ from baked venvs. Co-Authored-By:
-
- 16 Sep, 2026 15 commits
-
-
Stefy Lanza (nextime / spora ) authored
kt was refused because it looked unshippable: a Python stack pinning its own torch, thought to be AMX-only. kt-kernel 0.7 ships prebuilt manylinux wheels that dispatch at runtime (AMX, AVX-512, AVX2 llamafile, AMD BLIS), and sglang-kt pins the matching torch — so it is a venv on the light core like the other split images. The worker now launches SGLang from that venv (CODERAI_KT_VENV / ktransformers.venv) instead of assuming coderai's interpreter has it, and accepts a HF repo id, which SGLang downloads into the volume's cache. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The engines are C binaries and coderai only drives them; the full core's torch was dead weight. All they link is the CUDA runtime and cuBLAS, which NVIDIA ships as wheels at the 12.8 they were compiled against. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Twice the build disk filled with 500+ GB of layers no image referenced, pinned by mount records of containers the daemon no longer knew about, while docker system df reported 90 GB. Nothing in docker can see them. This walks imagedb/layerdb/mounts the way a fresh daemon would, reports the stale set, and with --fix (daemon stopped) removes it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The local ds4-server is an sm_86 build against CUDA 13 and the local colibri has no GPU backend: neither could be copied into a pod. The new engines image compiles both, plus kimi-k3-in-c, in a CUDA 12.8 stage for sm_80/86/89/90/120a (ds4's Blackwell kernels need the arch-specific target; plain sm_120 fails in ptxas) on a portable x86-64-v3 baseline, and fails the build if a binary cannot find its libraries. A RunPod-only model names the engine on its runpod block (backend: runpod takes the pin slot); a local model pinned to the engine and bursting carries the pin. Either way the pod arrives with the engine enabled and this install's settings forwarded (CODERAI_<ENG>_CONFIG), and volume_path points it at weights already on the network volume — the only sane way to run 150 GB+ models. The model page warns loudly the moment one of these engines is picked. ktransformers is refused with a reason: it has no pod image. Also: the batch build/publish scripts no longer list core-light and venv profiles as images. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod-only was an entry in the compute-backend dropdown, between OpenCL and wav2vec2 — not where anyone deciding WHERE a model runs would look. The Placement section now asks 'Runs on' (this machine / RunPod only / a machine of yours) and the RunPod block sits under it; the radios and the backend select are one stored value, kept in sync. The host backend existed only as JSON. It now has its fields on the form, a save handler, and — like a pod — no local weight or VRAM bookkeeping. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
gpu_count was passed to RunPod's gpuCount and then ignored: the price ceiling checked one card, min_vram_gb one card's memory, and the engine loaded on GPU 0 while the rest sat idle. Now the pod is ranked as a whole (price x N, VRAM x N, '2x A100'), vLLM gets --tensor-parallel-size N and llama.cpp --split-mode layer. Exposed on the model page, clamped 1-8. Requests forwarded to a pod or host never appeared on the Tasks page, so a running remote job was invisible from the admin. _forward now registers a 'remote' task whose message names the pod it landed on. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
weights_gb, on the model entry or its runpod block, is taken as-is and the HuggingFace estimate is skipped. It is the only honest answer for a URL download, an upload or a local file — none of which the estimator can see — and for an HF repo it beats a heuristic that deliberately errs high (SDXL: estimated 35 GB, actually ~7 for one precision). A nonsense value is ignored rather than trusted, and the field is never seeded to a pod: it describes the pod's disk, not the model. /v1/models/test/state now reports the disk a pod would be rented with, the weight size used, and whether that size was stated or estimated — decided before any money is spent. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
remote-execution.md: the capability images are two kinds now — ten on the full core and five on a torch-free light core for stacks that cannot share a venv — and the doc says which, why each split was measured rather than assumed, what the .venv-*.check scripts catch that a bare import cannot, and why the light core exists (RunPod refuses images above ~10 GB; a specialised image on the full core carried 7 GB of torch it never used). The 'speaker rides stt' line was wrong — that image shipped nothing for speaker work — and is fixed. A new section covers what travels to a pod as environment: the HF token, licence acceptances, baked-venv paths, OCR engine gates. Surya on a pod is documented as served by docTR, with the reason. The host backend gets its own section: always-on and on-demand shapes, what it deliberately lacks, and service_token. runpod.md: the size line with the numbers behind it, EXITED as a dead pod rather than a slow pull, container-disk sizing from HF metadata, and the field table corrected — boot_timeout_s is 900, container_disk_gb grows to fit. README: the fifteen images and the host backend, one line each. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod was launched with docTR as its default and still answered 'OCR engine surya is not enabled' — because the request carried engine=surya, and the OCR route honours the request's own field over the configured default. Setting the pod's env changed the default; it did not change what the request asked for. A pod target for OCR now rewrites the request's engine field on the way out, to the engine the pod will actually serve (pod_ocr_engine mirrors the launch env). Multipart is edited on the bytes with every other part untouched and Content-Length corrected; verified through FastAPI's own parser with the real probe body, file intact. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
XTTS reached its own loader on a pod and died on 'unparseable worker reply'. coqui prints an interactive CPML licence prompt to stdout and then blocks on input() — on a pipe, that corrupts the JSON protocol and then waits forever. Two fixes. The service now keeps the real stdout for the protocol and sends everything else the process prints to stderr, so no library chatter can reach the caller's parser again. And the CPML acceptance travels as COQUI_TOS_AGREED, locally and to a pod: configuring an XTTS model at all IS the decision, exactly as surya's licence is handled, and the pod inherits it. Verified in the image: the load returns clean JSON with the full speaker list. Surya: 0.22 is a VLM (surya-ocr-2), not a self-contained OCR library. It needs a server — vLLM, which it spawns IN DOCKER, or llama-server on a GGUF — and the pod image has neither; locally it rides coderai's own vLLM engine. A pod asked for surya now serves the request with docTR, which is in the image and has passed, and logs why — instead of loading surya, reaching inference, and dying on 'docker binary not found'. Surya on a pod belongs with the engine images, where a llama-server and the GGUF are the same work as colibri and ds4. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod reported 'No module named surya.ocr' — an API that has not existed for a year. The real error was two tiers up: the loader wrapped each version branch in a bare except, so a failure INSIDE the right branch was swallowed and it fell through to the oldest one. The reported error was from code that never should have run. The real error was surya 0.17.1's config class choking on transformers 5.17. The pin came from a stale requirements-surya.txt. The local install — read from its venv's dist-info — runs surya 0.22.1 on transformers 5.15, and works. Both now pin 0.22.1. 0.22 dropped surya.foundation and takes RecognitionPredictor() alone, which the loader already handled — but only as a fallback reached by swallowed exception. It now selects the API by what IMPORTS: foundation present is 0.17, absent is 0.22 (or pre-0.17). Verified loading 0.22.1 through the engine class on the pod stack beside docTR. tts-xtts: coqui's TTS(name) splits on '/' and wants its own four-part catalogue name; a HuggingFace id has two parts and fails 'expected 4, got 2'. The service translates (coqui/XTTS-v2 -> tts_models/multilingual/multi-dataset/xtts_v2), verified against coqui's catalogue. This means XTTS through _CoquiBackend had never worked with an HF id anywhere, not only on a pod. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The capability images are ordinary containers; nothing about them needs RunPod. The same image can run on a box you own, a server rented by the month, a second GPU in the next room — and coderai should use it the way it uses a pod: per model, with an endpoint and a token, and nothing else. always on: {"backend": "host", "host": {"url": ..., "api_key": ...}} on demand: add start_cmd (+ optional stop_cmd, boot_timeout_s, idle_timeout_s) start_cmd is whatever starts the thing — docker run, ssh box docker start, a systemd unit. coderai runs it, waits for /healthz, uses the host, and runs stop_cmd after idle_timeout_s with nothing in flight. It only ever stops a host it started itself; one that was already up is never touched. Deliberately absent: GPU search, price ranking, budgets, spot, capacity fallback, the orphan reaper. Those exist because a cloud rents an anonymous card by the second and can lose it. A host is a named machine you are responsible for — the machinery would be wrong, not just unnecessary. Verified for real: a capability container started by command from a cold state, healthy in 6s, used, and stopped by the idle timer. Also fixed on the way: a plain service_url carried no token, so a hand-run capability image behind one answered 401 to everything. service_token travels now. Neither host nor service_* is seeded to a pod: they say where a model runs from HERE, and would make a pod try to forward to itself. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
'function object has no attribute tqdm', on a rented pod, after the model had loaded. Current f5-tts defaults its progress argument to the tqdm MODULE and calls progress.tqdm(iterable); older releases called progress(iterable). Our call site passed a bare lambda, written for the old shape, and the new shape went looking for a .tqdm attribute on it. _NoProgress answers both — callable, and with a .tqdm — and shows nothing, which is what a server wants. Confirmed against the f5-tts actually in the voice image: its infer_batch_process calls progress.tqdm(...). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The HF token fix worked: pyannote got past the gate and failed one layer deeper, instantiating the pipeline. Reproduced by running the real service with a GPU: ModuleNotFoundError: No module named 'matplotlib'. pyannote's segmentation tasks import matplotlib.pyplot at module load; it is not declared, and 'import pyannote.audio' never reaches it, so the build's import check passed and the venv shipped broken. This is the fourth venv whose bare import check passed while the real load failed. A <profile>.venv-<name>.check script now runs at build time in the venv's own python and imports what the REAL load reaches — pipeline modules, model classes — without a GPU or a token, which a build has neither of. The pyannote check fails on the shipped image and passes with the fix; the XTTS check passes as-is, so that venv is genuinely complete. Also: every isolated-venv worker reported failures as the LAST FIVE LINES of the subprocess tail. A Python traceback's last five lines are frame headers — the exception is the final line, and cutting to five dropped it whenever the trace was deep. Ten workers said where a process died and never why. Fixed in all of them. Co-Authored-By:
-