- Oct 05, 2026
-
-
Stefy Lanza (nextime / spora ) authored
A pod asked for surya or olmOCR was downgraded to docTR UNCONDITIONALLY. The reasoning was sound — both are VLMs needing a model server, locally they ride coderai's own vLLM engine, and a pod image will not get one (that venv alone is ~10.5 GB against a ~12 GB ceiling, and it would be a second torch beside the one docTR and Surya already need) — but the conclusion was too broad. In their SERVED modes these engines only speak HTTP, and PIL and requests are already in the image. So a pod can run them whenever the operator has pointed them at a reachable endpoint, including at a vLLM pod, which is the composition to prefer over a fat image. - _ocr_external_server(engine) decides it: olmocr_serve="server" + olmocr_server_url, or surya_serve="llamacpp" + surya_server_url. The local-only modes ("vllm", "model") still downgrade, and the log now names the setting to fix instead of only the symptom. - The mode, URL and model ids are seeded to the pod. config.py gained the env overrides for surya_serve/surya_server_url and fallback_engines; only olmOCR's existed. - fallback_engines is pinned to the engines the image actually carries: "auto" would hand a failed page to an engine with no server on that pod, which the fallback added in 0.2.28 made newly reachable. - Engine selection is alias-aware (`alias or served or path`, the convention used throughout the manager). An entry aliased "olmocr" over an HF repo path was read as an unknown engine and silently downgraded. Aliases are how one model carries several configs and how several models share an endpoint, so this path has to honour them. - The licence decision now survives a downgrade: `wanted` was reassigned to "doctr" before the `wanted == "surya"` check, making that branch's own documented intent ("asking for surya by name IS the decision") unreachable. olmOCR needs no pip dependency anywhere — it imports only PIL and requests — which is why nothing flagged its absence from the profiles; ocr.txt now records that and why a pod has no vLLM. 19 new tests; 278 pass across the OCR, front-gate, vLLM-share, cluster, remote-execution, placement and routing suites. Co-Authored-By:Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
- Sep 30, 2026
-
-
Stefy Lanza (nextime / spora ) authored
CoderAI could make every piece of a short video and nothing turned the pieces into a reel — assembly needs ffmpeg and two facts only the server has (how long the narration it just synthesised is, and when each word falls), so a detached client cannot do it. Requested by the Motus project over the HTTP API. - codai/api/compose.py: scenes -> one uploadable video. Scene length = narration + padding (min_duration floor), visuals split it evenly; async job with stage/progress/message/warnings, cancel (the ffmpeg child is killed), a queue that reports its position, and a blocking form for tests. Narration, music and word timings come from this server's own endpoints, called on a loop this module owns — never the request's loop, which may be closed before the render needs it. 400 names the offending scene/field; 501 when a sub-feature has no model (music.generate, timing=stt, TTS for narration) or ffmpeg is missing. - codai/compose/render.py: one normalised clip per visual (so a failed visual becomes a gradient and a warning instead of a lost render), concat or xfade that preserves the total duration, narration padded per scene and loudness-normalised, music looped/faded/ducked with sidechaincompress, burned captions + overlays with the final H.264 High/AAC/+faststart encode, thumbnail. - codai/compose/captions.py: word timings from STT aligned onto the CALLER's words (a mis-heard word never reaches the screen) or estimated; line breaking; ASS with six presets inside a 10%/6% safe area; SRT/VTT with the same text. - codai/compose/media.py: resolve a URL / /v1/files path / data URI / base64 (never a server path), probe with ffprobe, resolve fonts through fontconfig. - codai/api/uploads.py: POST /v1/files/upload, content-addressed by sha256 with GET /v1/files/blob/{hash} so a client can skip a re-upload; type decided by content, not by the client's claim. - compose and uploads are pinned local in the capability gateway (a pod has ffmpeg but not this install's models or files); both advertised in the capability document; caption fonts (Montserrat/Inter/Noto CJK) added to the three image builds. - docs/video-compose.md; tests/test_compose.py (30, real ffmpeg, no model/GPU). Co-Authored-By:
-
- Sep 20, 2026
-
-
Stefy Lanza (nextime / spora ) authored
cluster: mDNS discovery with a shared token, prefix-cache routing, Prometheus /metrics + per-key usage, node logs and re-push on return, GGUF row split, cosign-signed images - codai/cluster/discovery.py: _coderai._tcp announce + browse (zeroconf), HMAC fingerprint of cluster.token as membership; members auto-join as nodes, listed nodes with a blank key get the shared token; nodes and engines accept the cluster token (front /cluster/*, engine BearerAuth); env overrides for capability images (CODERAI_CLUSTER_TOKEN/DISCOVERY/NODE_NAME/ADVERTISE_HOST); coderai-docker --host-network - frontproxy/affinity.py + router prefer=: a conversation goes back to the engine/node that served it when the model is resident in several places; engine_for_model now picks the least busy holder - frontproxy/metrics.py: /metrics (requests per key/model/kind/status/engine, engines/nodes up, VRAM, in-flight, RPC, discovery, RunPod spend), usage tables on the Cluster page, server.metrics_public - node logs: /cluster/logs on the node, /admin/api/cluster/logs on the head, log viewer on the Cluster page; a node that comes back gets its assigned models pushed again - per-model split_mode=row (llama.cpp tensor parallel over local + RPC devices); vLLM TP may span nodes (documented) - packaging/sign-images.sh + packaging/cosign.pub: all 21 GHCR images signed - docs/cluster.md §§ discovery, 7 prefix routing, 9 operating, 10 accelerators evaluation; tests/test_cluster_ops.py (11) Co-Authored-By:
-
- Sep 19, 2026
-
-
Stefy Lanza (nextime / spora ) authored
cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang multi-node, pools, compat-filtered model form Several coderai installs as one: - Cluster nodes (cluster.nodes): another install is an engine of this front — polled on its /cluster/state, assigned models, pinned by name, routed by capability and load; entries pushed to /cluster/reload-config and localized there; Load/Unload reach it through token-guarded twins. Live add/remove. - llama.cpp RPC: codai/backends/ggml_rpc.py registers rpc-server endpoints through ctypes and gives every load an explicit device list (local cards, then the model's rpc_servers); the split and the VRAM pool see remote memory. cluster.rpc_servers runs this machine's rpc-server processes; packaging/build-rpc-server.sh builds it from the vendored llama.cpp; build.sh and the images build llama-cpp-python with GGML_RPC. - vLLM on Ray (pipeline parallel across nodes) and SGLang --nnodes: per-model vllm/kt blocks over the global config, node start/stop commands, ray head and worker mode in boot.sh. - Pools: several URLs per capability remote and several machines per host model, with failover. - Models page: engine/backend selects narrowed live to compatible pairs, impossible pairs refused; Settings → Cluster; /admin/cluster page. Co-Authored-By:
-
- Sep 18, 2026
-
-
Stefy Lanza (nextime / spora ) authored
PyPI's CDN throttles a machine that has pulled 100 GB of wheels in a day to ~400 KB/s; a venv rebuild that only needed a new apt layer above it then spends hours re-downloading torch. Cache mounts for pip and uv keep the wheels between builds. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod died with 'The NVIDIA driver on your system is too old (found version 12080)': vLLM 0.27.1 on PyPI pins torch 2.13+cu130. vLLM's own index has a +cu129 variant with torch's cu129 wheels, and CUDA 12.x minor-version compatibility runs a 12.9 runtime on a 12.8 driver. uv resolves it with --index-strategy unsafe-best-match, since the torch index also carries stale copies of ordinary packages. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
vLLM on the pod died with 'Engine core initialization failed. See root cause above' — and the root cause was above the six-line tail the 503 carried. Neither core has a C compiler; Triton JIT-compiles vLLM's (and SGLang's) kernels at run time and needs one. A <profile>.apt marker keeps runtime apt packages in the image; vllm and engines-kt get gcc. The worker's error now surfaces the lines that name an error first. Co-Authored-By:
-
- Sep 17, 2026
-
-
Stefy Lanza (nextime / spora ) authored
coderai-llama came out at 11.6 GB — the CUDA wheel for five GPU generations on top of the full core — over RunPod's size line. The only thing the CUDA GGUF backend needed torch for was three questions about the card (is there one, total memory, free memory); gpu_probe answers them from torch when present and NVML otherwise. The image drops to the light core plus NVIDIA's runtime, cuBLAS and NCCL wheels. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai-text was transformers only, so a GGUF or a vLLM-served repo still went to an upstream image: no TLS on the direct path, no /boot, no local LoRAs, nothing on the Tasks page. coderai-llama compiles llama-cpp-python with CUDA for every pod GPU on the full core; coderai-vllm is vLLM in its own uv-resolved venv on the light core, found by the worker through CODERAI_VLLM_VENV, with the local vLLM settings forwarded. The upstream engines stay selectable as vllm / llamacpp. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
direct_tcp traded the proxy's 100 s cutoff for plain HTTP. Now a coderai pod on that path brings its own TLS: a per-install CA (openssl, kept beside the pod registry) signs a short-lived leaf per pod, sent as env like the token; boot.sh and the venv-on-volume bootstrap start uvicorn on it. This side verifies https-to-a-bare-IP against that CA only — pod_http, a requests shim every pod-facing call now goes through. No hostname check: the IP is unknown at issue time, and a foreign cert fails the chain before a name is compared. Proven end to end in a test with a real https server. vLLM/llama.cpp images stay plain on this path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
pip spent an hour and gigabytes of RAM on sglang-kt's forty exact pins without installing a byte; uv resolved the same set in seconds. A <profile>.venv-<name>.uv marker opts a venv in. Also drop __pycache__ from baked venvs. Co-Authored-By:
-
- Sep 16, 2026
-
-
Stefy Lanza (nextime / spora ) authored
kt was refused because it looked unshippable: a Python stack pinning its own torch, thought to be AMX-only. kt-kernel 0.7 ships prebuilt manylinux wheels that dispatch at runtime (AMX, AVX-512, AVX2 llamafile, AMD BLIS), and sglang-kt pins the matching torch — so it is a venv on the light core like the other split images. The worker now launches SGLang from that venv (CODERAI_KT_VENV / ktransformers.venv) instead of assuming coderai's interpreter has it, and accepts a HF repo id, which SGLang downloads into the volume's cache. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The engines are C binaries and coderai only drives them; the full core's torch was dead weight. All they link is the CUDA runtime and cuBLAS, which NVIDIA ships as wheels at the 12.8 they were compiled against. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The local ds4-server is an sm_86 build against CUDA 13 and the local colibri has no GPU backend: neither could be copied into a pod. The new engines image compiles both, plus kimi-k3-in-c, in a CUDA 12.8 stage for sm_80/86/89/90/120a (ds4's Blackwell kernels need the arch-specific target; plain sm_120 fails in ptxas) on a portable x86-64-v3 baseline, and fails the build if a binary cannot find its libraries. A RunPod-only model names the engine on its runpod block (backend: runpod takes the pin slot); a local model pinned to the engine and bursting carries the pin. Either way the pod arrives with the engine enabled and this install's settings forwarded (CODERAI_<ENG>_CONFIG), and volume_path points it at weights already on the network volume — the only sane way to run 150 GB+ models. The model page warns loudly the moment one of these engines is picked. ktransformers is refused with a reason: it has no pod image. Also: the batch build/publish scripts no longer list core-light and venv profiles as images. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod reported 'No module named surya.ocr' — an API that has not existed for a year. The real error was two tiers up: the loader wrapped each version branch in a bare except, so a failure INSIDE the right branch was swallowed and it fell through to the oldest one. The reported error was from code that never should have run. The real error was surya 0.17.1's config class choking on transformers 5.17. The pin came from a stale requirements-surya.txt. The local install — read from its venv's dist-info — runs surya 0.22.1 on transformers 5.15, and works. Both now pin 0.22.1. 0.22 dropped surya.foundation and takes RecognitionPredictor() alone, which the loader already handled — but only as a fallback reached by swallowed exception. It now selects the API by what IMPORTS: foundation present is 0.17, absent is 0.22 (or pre-0.17). Verified loading 0.22.1 through the engine class on the pod stack beside docTR. tts-xtts: coqui's TTS(name) splits on '/' and wants its own four-part catalogue name; a HuggingFace id has two parts and fails 'expected 4, got 2'. The service translates (coqui/XTTS-v2 -> tts_models/multilingual/multi-dataset/xtts_v2), verified against coqui's catalogue. This means XTTS through _CoquiBackend had never worked with an HF id anywhere, not only on a pod. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The HF token fix worked: pyannote got past the gate and failed one layer deeper, instantiating the pipeline. Reproduced by running the real service with a GPU: ModuleNotFoundError: No module named 'matplotlib'. pyannote's segmentation tasks import matplotlib.pyplot at module load; it is not declared, and 'import pyannote.audio' never reaches it, so the build's import check passed and the venv shipped broken. This is the fourth venv whose bare import check passed while the real load failed. A <profile>.venv-<name>.check script now runs at build time in the venv's own python and imports what the REAL load reaches — pipeline modules, model classes — without a GPU or a token, which a build has neither of. The pyannote check fails on the shipped image and passes with the fix; the XTTS check passes as-is, so that venv is genuinely complete. Also: every isolated-venv worker reported failures as the LAST FIVE LINES of the subprocess tail. A Python traceback's last five lines are frame headers — the exception is the final line, and cutting to five dropped it whenever the trace was deep. Ten workers said where a process died and never why. Fixed in all of them. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod refuses images above a size line — measured: 10.1 GB boots, 14.5 GB is 'Exited by Runpod' two seconds after rent on every machine, no other reason given. Container disk (tried 80 GB), layer format and the full image config were each ruled out; size was the only difference. All five specialised images were above the line. They were above it for one reason: each was built on the full core, so it carried 7 GB of CUDA, torch and triton the main venv never used on that pod, beside the venv that did the work. The speaker image was 14.8 GB, of which pyannote's own venv was 5.6. The light core is coderai's Python dependencies with no GPU stack: 1.2 GB. codai.api.app imports with no torch present (verified first), so the main interpreter still boots the API, serves /healthz and /boot, and drives the subprocess worker. A profile marked <name>.light builds on it, its main venv carries only what the API layer needs, and the build FAILS if a torch reaches that venv — because that is exactly the leak that would put the image back over the line. speaker: 14,796 MB -> 7,077 MB, complete, nothing on a volume. Co-Authored-By:
-
- Sep 15, 2026
-
-
Stefy Lanza (nextime / spora ) authored
A speaker pod reported EXITED with uptime 0s from its very first poll. The loop waited out the full budget on it — three machines in a row — printing 'image pull in progress', which it never was: wait_ready treated only TERMINATED and FAILED as terminal. EXITED past a short restart grace now fails fast and says what it means: the image pulled and the process died on boot. The budget it waited out was 300s, not the 900s set on the pool class: the dataclass default was never updated. Two numbers for one thing, and the smaller one silently won. There is one now. Why the process died: boot.sh ran under set -u, and every line before the exec is a diagnostic — it exists to describe the pod, and a diagnostic must never be able to stop the boot it describes. The image boots fine locally; on RunPod an environment difference tripped a strict-mode abort before uvicorn was reached. That is the best-supported explanation, not a proven one — RunPod publishes no log API to confirm it — so the script is now unable to abort before its exec under any environment, verified against a bare sh with nothing set. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The venv built and then failed its import check with ModuleNotFoundError. Verified the whole spec in a throwaway container first this time: with click, 'from TTS.api import TTS' works on torch 2.5.1+cu124 with transformers 4.46.3 — the stack this image exists to provide. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod images were built to satisfy imports, not to serve models. The audit found coqui/XTTS-v2, every whisper.cpp model, diarization, speaker ID, SALAD place recognition and Surya all unservable on a pod, and the OCR image shipping no OCR engine at all. Three of the eight 'isolated' stacks are not isolated any more. Surya, speechbrain and pytorch_lightning all import cleanly beside torch 2.11 — their venvs were decided against an older torch and never re-checked, exactly like audiocraft's torch==2.1.0. They go straight into the ocr, stt and embeddings profiles: place recognition, speaker ID and Surya now work for a few hundred MB instead of three new images. Surya's subprocess engine is pointed at the pod's own interpreter, since building a venv to hold what is already installed would only cost rental time. Five stacks genuinely cannot share, each verified rather than assumed: pyannote (torchaudio.AudioMetaData is gone in 2.11), coqui-tts (a transformers 4.x symbol), CrisperWhisper (transformers 4.46 AND torch 2.5), NeMo (large, own torch) and paddle (its own CUDA runtime). Each gets an image carrying a baked venv, rather than 6 GB added to an image everyone pulls. The `speaker` capability no longer points at the STT image: that was the assumption that it was 'the STT deps plus a bit', and it shipped nothing for diarization or voiceprints at all. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The OCR pod image installed NO OCR engine — only transformers and pypdfium2. A pod could boot, pass its health check, accept a request and have nothing to do it with. Three gates in front of that each reported their own refusal first — 'OCR subsystem is disabled', then 'OCR engine surya is not enabled' — so it took three rounds to see that the engine was never there. docTR goes in, because it is the only one of the three that runs in-process. Surya caps pillow<11 and lives in an isolated venv by design; paddle wants paddlepaddle in one of its own. Either would put a second torch in the image — 6 GB to avoid a Pillow conflict — and the venv mechanism is there if a pod ever needs Surya's layout quality specifically. Each engine also has its own enable flag on top of the subsystem's, and two of the three default to off. A pod now enables the engine it was asked for, falls back to docTR when the request names something the image cannot serve, and still inherits the Surya licence decision from here rather than making it itself. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The isolated venv was solving a problem that was not real. audiocraft is pure Python: its torch==2.1.0 is a declared pin, not an ABI requirement, so --no-deps sidesteps it. The only genuine constraint was xformers, which it imports at module load — and 0.0.35 ships a py39-none wheel requiring torch>=2.10, which this image satisfies. I had ruled that out by grepping wheel filenames for cp312 and missing that abi3/py39 wheels cover 3.12. Verified in the built core image: torch 2.11.0+cu128 with xformers 0.0.35, MusicGen and AudioGen import, ops.memory_efficient_attention present. Cost in the main venv: 950 MB. The second venv cost 6.2 GB, because it carried a duplicate torch and a duplicate CUDA stack. The audio image drops from 15.8 GB to about 10.2 GB, which also removes the reason to split it in two. audio_backend (per model, or CODERAI_AUDIO_BACKEND for a pod) selects audiocraft or transformers explicitly, because they are not equivalent — melody conditioning and AudioGen exist only in the first. Asking for audiocraft when it is absent raises rather than quietly substituting something less capable. A model's own setting beats the environment it lands in, and the choice travels to a pod instead of being decided by what happens to be installed there. The isolated-venv worker stays in the tree: if audiocraft 1.3.0 turns out to misbehave against torch 2.11 at generation time, restoring one profile file brings it back. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three build cycles, three different missing modules — xformers, then torch at install time, then transformers. The list had been trimmed against what the MAIN venv provides, which is exactly backwards: an isolated venv starts with nothing, so it needs the whole declared set, minus only the torch pins we choose and the dev-only tools. Verified in a throwaway container against the real core image before building anything: audiocraft 1.3.0 on torch 2.5.1+cu121 with xformers 0.0.28.post3, and MusicGen and AudioGen both import. That check takes two minutes; the image build it replaces takes fifteen, which is why the last three rounds were so expensive. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Voice cloning failed with 'Could not load libtorchcodec', whose message blames FFmpeg and the torch version. It was neither: the CUDA build links against NVIDIA Performance Primitives (libnppicc.so.12), torch does not use NPP, and torchcodec arrives as a transitive dependency of f5-tts — so nothing ever installed it. Installing it is only half: torch preloads its own nvidia libs and NPP is not among them, so the library sits in site-packages and the loader never looks there. LD_LIBRARY_PATH covers the other half. Verified in the built image: torch 2.11.0+cu128, torchcodec 0.11.1+cu128, AudioDecoder imports. Two wrong theories preceded this — an FFmpeg major mismatch and a torch/torchcodec ABI mismatch — both suggested by that error message, and both disproved by loading each .so by hand with torch imported. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Second failure in that venv: 'No module named torch' while installing. xformers below 0.0.27 has no cp312 wheel, so pip tried to build it from source, and its setup.py imports torch — which does not exist yet in a fresh venv. The binding constraint was never audiocraft's torch==2.1.0 (unsatisfiable on this Python) but xformers, which audiocraft imports at module load: its cp312 wheels start at 0.0.27, and 0.0.28.post3 requires torch==2.5.1 exactly. So the venv is torch 2.5.1 + xformers 0.0.28.post3 — chosen by what actually has wheels for this interpreter rather than by what the libraries would prefer. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Dropped with the rest of its unsatisfiable pins, but audiocraft imports xformers at module load, so the venv built cleanly and then failed its import check with ModuleNotFoundError — which is what that check exists for: on a rented GPU this would have surfaced minutes into a generation instead. Its pin (<0.0.23) pairs with the torch it cannot have here; 0.0.25.post1 is the build that matches torch 2.2.2. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The audio build failed with 'No matching distribution found for torch==2.1.0': audiocraft's pin is unsatisfiable on Python 3.12 at all, because torch support for 3.12 starts at 2.2. So the venv uses the oldest torch this Python can run and installs audiocraft --no-deps, with the deps it actually uses listed explicitly. --no-deps is reckless in a shared venv and routine in an isolated one: nothing in there can reach the server's stack, and the build's import check fails the image if the version judgement turns out to be wrong. The step is now generic — any <profile>.venv-<name>.txt builds /opt/coderai/venvs/<name>, with an optional .nodeps beside it — rather than hardcoding audiocraft into the Dockerfile. The next library that pins an impossible torch will not need a new build step. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The transformers fallback keeps MusicGen working on a pod, but it is not the same thing: audiocraft has melody conditioning and AudioGen, and transformers has neither. Supporting it is not a matter of adding a requirement — it pins torch==2.1.0, torchvision==0.16.0 and torchtext==0.16.0, and torch 2.1.0 has no build for the CUDA these images run, so installing it beside the server would leave the pod with no working GPU torch at all. So it gets its own venv and is driven as a subprocess, the same answer parler-tts and the OCR engines already use here. AudiocraftModel deliberately wears audiocraft's own interface (set_generation_params / generate / generate_with_chroma / sample_rate) so the generation path cannot tell whether the library is in this process, in a venv next door, or absent. The venv is baked into the audio image rather than built on first use: a pod is rented by the second, and pip-installing a library during a cold start costs mor...
-
Stefy Lanza (nextime / spora ) authored
None of these were placement or pod bugs: each image booted, pulled its model and got as far as executing. The remote machinery worked; the payloads did not. voice — torchcodec arrived as a transitive dependency of f5-tts, PyPI served the CUDA-13 build, and it died with 'libnvrtc.so.13: cannot open shared object file' after loading the model. The same class of bug as the torch pin: a CUDA-coupled package escaping the index. It now installs from the CUDA index with the rest, resolving to torchcodec 0.11.1+cu128. audio — 'No module named audiocraft'. audiocraft pins an old torch, so putting it in the image would drag torch backwards and undo the CUDA pin. transformers ships MusicGen itself, so a pod serves it through transformers instead; an install that has audiocraft still uses it. Melody conditioning stays audiocraft-only rather than silently doing something different. video — 'TextToVideoSDPipeline.__call__() got an unexpected keyword argument callback_on_step_end'. Older diffusers pipelines do not take it and raise inside the call, minutes in, with the model loaded and the GPU paid for. The kwarg is now added only when the pipeline's signature accepts it. The try/except that used to wrap it caught nothing: assigning to a dict never raises. This one bites local generations too, not just pods. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
--extra-index-url only ADDS an index; pip then picks the highest version across all of them, and PyPI's default torch is built for CUDA 13. So the pin never applied: every capability image shipped a CUDA-13 torch and failed on any machine with an older driver — 'The NVIDIA driver on your system is too old (found version 12080)'. RunPod's fleet is mixed, so the same GPU model passed or failed depending on which machine the pod landed on, which is exactly why this looked like flakiness for so long. torch, torchvision and torchaudio now install from the CUDA index alone, via --index-url, before anything else; installed later from PyPI they would pull a matching CUDA-13 torch back in. cu128 resolves to torch 2.11 — the version the local install already runs — and runs on the driver that failed. Both Dockerfiles print the resolved torch build at each layer, so a silent substitution cannot happen again. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod could not say WHICH build it was running, which is exactly the question a cached ':latest' raises — an images pod reported a dependency missing that had already been published under that tag. The build now bakes IMAGE_TAG in, so the boot record names the image. /boot also showed only 'lifespan start': the seed registration and the end of startup were plain prints. They are phases now, so one request covers the whole boot including how many seeded models registered and whether that step failed. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The port a pod exposes appears only AFTER its image is pulled, so the 300s budget had to cover the download — it was written assuming the container was already there. The same 7.3 GB image took 133s on a machine with the layers cached and failed repeatedly at 300s on cold ones, and each failure rented a fresh machine and paid for the whole pull again, three times. 900s, overridable per model; being wrong now costs six cents instead of three pods. RunPod publishes no pod-log API: the REST OpenAPI spec lists 23 routes and none serve logs, and GraphQL refuses introspection. Every attempt returning HTTP 400 was code guessing at URLs that never existed. So the pod reports on itself — boot.sh is the entrypoint and logs timestamped phases from the first instant the container exists (including the seed list, which is where "not available. Use one of: " would have been caught in one line), and /boot serves that record plus the app's own phases. A failed boot now asks the pod before falling back to a console link a human has to open. Nothing covers the pull window — no code of ours is running yet — so wait_ready reports status and uptime every 60s instead of going silent for minutes. Co-Authored-By:
-
- Sep 14, 2026
-
-
Stefy Lanza (nextime / spora ) authored
A rented images pod downloaded SDXL for ten minutes and then failed with "No package metadata was found for bitsandbytes". models.json carries quantization settings for EVERY model type, not just LLMs, but bitsandbytes was only in the text profile. It moves to core, the shared layer under all ten images. stt and ocr were reachability-only, which reported ok:true while proving nothing beyond "a URL is configured" — a pass that would survive a completely broken pod. Both inputs can be synthesised honestly: espeak speaks a known sentence, PIL draws known words. The result is checked against what went in, so a model that returns fluent nonsense fails too. When the synthesis itself cannot run (no espeak, no fonts) the probe falls back to the reachability check rather than blaming the model for a gap on this side. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A capability image is ~7 GB and nearly all of it is torch, pulled again by every cold pod. With a network volume those libraries can live there instead: `venv_on_volume` boots a slim image — OS, Python and coderai only — and runs coderai from a venv on the volume. The first pod builds it; every pod after skips both the pull and the install. Opt-in per pool and per model, never a default, because the trade is real and unmeasured: the pull gets much smaller, but importing torch off network storage is slower than off local disk, so some of the saving comes back at import time. Whether that is a net win depends on how often pods cold-start, which is the user's workload, not mine to assume. Two hazards are handled rather than left to chance. The ready marker is written AFTER the install, so a pod that dies mid-build leaves nothing a later pod would import from. And a lock directory — mkdir is atomic — makes a second pod wait for the first instead of two installs interleaving into the same directory. `venv_on_volume` without a volume is refused with the reason, rather than booting a pod that installs 7 GB onto container disk and discards it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ten images built, none pushed: the guard checks free space against every profile before doing anything, but a push uploads what already exists and needs no disk. It now measures only what actually has to be BUILT — and says so when there is nothing to build — so a full disk can no longer turn 'publish' into a no-op. It also asks docker where its root dir is rather than assuming /var/lib/docker, which on this machine is /storage/docker. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A text LoRA that exists only on this machine had nowhere to go: vLLM and llama.cpp resolve adapters at launch and have no endpoint to upload one to, so the previous commit could only report and skip it. Such a model is now routed to a coderai TEXT pod — the one pod image that can be sent an adapter — and the transfer is arranged around the fact that a pod is created before anything can be sent to it: 1. the pod is told the adapter's sha256 id, computed HERE, so both sides agree on the name before the pod exists; 2. the bytes go to the pod's content-addressed blob store before the first request, once per pod (a repeat costs one existence check); 3. the pod resolves sha256:<hex> back to a file — text_loras.local_path now understands the content-addressed forms — and applies it. A published adapter still goes to vLLM, which fetches it itself: cheaper and faster, so publishing an adapter you use often remains the better path, and the log says so. A PEFT adapter DIRECTORY is still refused with a reason — the blob store holds single files. packaging/runpod/profiles/text.txt adds the text profile (transformers, peft, bitsandbytes for QLoRA bases). It has to be built and pushed before use; the other nine are published already. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The package name is the last path segment of ghcr.io/<owner>/<name>:<tag>. Stripping only the registry left the owner in the name, so every URL the script printed after a push 404'd — on the one step the user has to do by hand, because GitHub has no API for package visibility. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The Dockerfile split core into its own RUN so the nine images would share it. Measuring the built result showed they did not: only 4 of 11 layers were shared, and each image carried its own ~7 GB copy. An identical `pip install` does not write identical bytes (dist-info metadata, RECORD files), so the layer digests differ even when the command and inputs match. Sharing by coincidence does not work. Dockerfile.capability-base builds the core (OS + apt + torch + the import-time deps) ONCE as its own image, and each profile is FROM it, so the sharing is structural. build_capability_image.sh builds it on first use and reuses it after (REBUILD_CORE=1 to force, needed when core.txt changes). Measured after rebuilding all nine: every image now carries all 8 core layers and adds 2 of its own (audio 5, from its build-deps apt churn). 29 distinct layers instead of 67, local disk 40.9 GB instead of 94, and a push uploads the 6.79 GB core once rather than nine near-identical copies. A pod pulling a second capability image already has most of it. Also corrected the publish script's claim that `gh` could make the packages public: GitHub's Packages REST API is list/get/delete/restore only, there is no visibility endpoint, and new packages are always private. It now prints the per-package settings URLs and says plainly that this is a click each. Co-Authored-By:
-
- Sep 13, 2026
-
-
Stefy Lanza (nextime / spora ) authored
voice needed build-essential: f5-tts/seed-vc pull webrtcvad, a C extension with no cp312 wheel. Same purged-in-one-layer mechanism audio uses for its Rust toolchain. All nine built and healthy (each answers /healthz): ocr 6.93 embeddings 7.26 audio 7.31 faceswap 7.57 tts 7.61 stt 7.63 images 7.74 video 7.83 voice 8.33 (GB) They share the core layer, so the disk and push cost is one core plus nine small deltas, not nine whole images. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
publish_capability_images.sh does the whole thing: disk check, registry login (with the exact ghcr token instructions), build, smoke test, push, and the per-package URLs for flipping GHCR's default-private packages to public. It is resumable — an image already built for this version is skipped, so re-running after a failure doesn't rebuild 7 GB of torch — and it pushes nothing until every image has answered /healthz. Profiles can now declare apt packages their wheels need to COMPILE, in `profiles/<name>.build-deps`, installed and purged inside a single layer. `audio` needed it: deepfilternet pulls deepfilterlib, which has no cp312 wheel and compiles Rust extensions. It now builds at 7309 MB against 7260 for a profile that needs nothing, so the toolchain is provably not in the image. Built and healthy so far: images (7.74 GB), faceswap (7.57 GB), embeddings (7.26 GB), audio (7.31 GB). ocr, stt, tts, video and voice are untried — the script resumes into them. Co-Authored-By:
-
