- 14 Sep, 2026 20 commits
-
-
Stefy Lanza (nextime / spora ) authoredStop sibling engines reaping each other's RunPod pods: a shared pod registry plus a reap grace period. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
Stefy Lanza (nextime / spora ) authored
The first real pod test found this in seconds: coderai runs one engine per GPU, each with its own pool registry AND its own stale-pod reaper, so each engine sees its siblings' pods as untracked orphans. The nvidia engine created a pod; the radeon engine terminated it four seconds later; radeon created one, nvidia killed that — a mutual kill loop that rented and destroyed pods until I disabled RunPod. Every boot then "failed" after its 300s port timeout, because the pod had been deleted underneath it, and the pool dutifully retried on the next machine. Two fixes, because one is not enough: A shared registry (config_dir/runpod-pods.json, flock'd) records every pod this DEPLOYMENT owns. _known_pod_ids now reads it, so an engine with no pools of its own still knows its siblings' pods are not orphans. A grace period (REAP_GRACE_SECONDS, 20 min) refuses to reap any pod younger than a boot budget, whatever the registry says — the backstop for the race a registry cannot close, such as a process that died between creating a pod and recording it. No pods were left billing: the mutual reaping cleaned up after itself and the account shows zero pods. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Test-run fixes (forced-remote verified before dispatch, empty output is a failure) and config propagation to the engine and front. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Configuring a remote capability in the UI did nothing until a restart, which makes the feature unusable as shipped. The admin API is served by the SYSTEM worker, so a save updates that process's config object. The remote gateway, the engine blocks and the front's burst-to-RunPod path all read a DIFFERENT process's copy. The engine's /internal/reload-config — which the front pushes on a config change — re-read models.json only, and the front's own refresh copied a fixed field list that included neither `runpod` nor `remotes`. Both now carry them: the engine re-reads config.json into its live config on reload, and the front refreshes runpod/remotes along with the rest. Found by trying to use the feature on the running instance rather than by reading the code, which is the only way this class of bug shows up. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Running the new test against the live instance found both immediately, which is the point of it. A FORCED REMOTE TEST COULD PASS LOCALLY. /v1/chat/completions is deliberately outside the gateway (RemoteOpenAIBackend owns it), so the x-coderai-placement header is ignored on that path: a text model with no remote configuration was served HERE while the result reported where=remote, ok=true. A test that passes in the wrong place is worse than no test. Forcing remote is now verified before anything is dispatched, and refuses with what is actually missing. A 200 WITH NO OUTPUT WAS A PASS. The text probe allowed 8 tokens, which a reasoning model spends thinking — Qwen3.5-9B answered with completion_tokens=0, finish_reason=stop, and the test called it ok. The budget is now 64, and an empty result is a failure carrying the numbers that explain it, for text, embeddings, images and rerank alike. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Test run: prove a model works where it is configured to run, with a per-request placement override. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Placement now has many moving parts (image, weights, adapters, auth, budgets, staging) and most of them fail minutes into a pod boot, in a log nobody is watching. POST /v1/models/test runs the smallest real request for the model's kind and reports what happened, with a Test run button on every model page. It dispatches THROUGH THE FRONT, so it exercises the same routing, auth, pod provisioning and staging a real request would — a test that simulated the path would prove nothing about the path. `where` forces placement for that one request via a new x-coderai-placement header the gateway honours: "local" ignores any remote configuration, "remote" refuses to fall back to local. Without that you cannot verify a pod at all: the request would be served wherever the config decided, and a green result would mean nothing. Kinds with no cheap synthetic input — video (minutes of GPU and real money), faceswap, STT, OCR — report a reachability check and say plainly that no generation was run, instead of implying more than was proven. Also fixed test pollution I had introduced: the gateway fixture replaced module-level functions without restoring them, and one test assigned _model_entry_for unscoped. Both leaked into later tests, which passed alone and failed in the suite — the failure mode that teaches you to distrust the suite. Verified stable across repeated runs. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model placement for any model kind, text LoRA/QLoRA locally and on RunPod, weight sources (HF id, URL, upload, pod-side staging), shared pools and optional warm pods. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Your idea, and it removes the last dead end. vLLM and llama.cpp pods could not be given a local model or adapter: no endpoint to upload to, and their server needs the file at launch. But a pod CAN fetch it before the server runs — put the weights on any HTTP host it can reach and stage them at boot. The obstacle was that those images run their server as the ENTRYPOINT, so everything we passed became an argument to the server. RunPod's REST pod API accepts dockerEntrypoint/dockerStartCmd (GraphQL does not), so runpod_client now creates a pod through REST when an override is needed and stays on GraphQL otherwise. The staged start command fetches with curl, wget or python — whichever the image actually has rather than whichever we assumed — extracts a tar for a multi-file model, and then execs the server pointed at what it staged. One definition of the staged path (staged_model_dest) is used by both the download and the server args: computing it twice is exactly how a pod downloads to one path and serves another, which the first cut did, along with emitting two --model flags. Adapters given as a URL ride the same path: --lora-modules for vLLM, --lora/--lora-scaled for llama.cpp. So a local adapter reaches a vLLM pod after all — publish it to a URL instead of uploading it. `source: upload` against a vLLM pod now says what to do instead, rather than only that it cannot. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A text LoRA that exists only on this machine had nowhere to go: vLLM and llama.cpp resolve adapters at launch and have no endpoint to upload one to, so the previous commit could only report and skip it. Such a model is now routed to a coderai TEXT pod — the one pod image that can be sent an adapter — and the transfer is arranged around the fact that a pod is created before anything can be sent to it: 1. the pod is told the adapter's sha256 id, computed HERE, so both sides agree on the name before the pod exists; 2. the bytes go to the pod's content-addressed blob store before the first request, once per pod (a repeat costs one existence check); 3. the pod resolves sha256:<hex> back to a file — text_loras.local_path now understands the content-addressed forms — and applies it. A published adapter still goes to vLLM, which fetches it itself: cheaper and faster, so publishing an adapter you use often remains the better path, and the log says so. A PEFT adapter DIRECTORY is still refused with a reason — the blob store holds single files. packaging/runpod/profiles/text.txt adds the text profile (transformers, peft, bitsandbytes for QLoRA bases). It has to be built and pushed before use; the other nine are published already. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
lora_path and lora_model_dir have been in the model config for a long time, but only the VRAM estimator ever read them: vulkan.py, cuda.py and vllm.py contained zero LoRA references, so configuring a LoRA on a text model silently did nothing. Now all three apply them. codai/models/text_loras.py answers the one question they share — which adapters does this model want, and where are they — across every config shape the field has grown (lora_path, a list, lora_model_dir, a `loras` list of dicts), de-duplicated so the same adapter is not applied twice. Each runtime then does its own thing: transformers PEFT load_adapter + set_adapters, several at once llama.cpp lora_path + lora_scale, ONE adapter and it must be GGUF-converted — a PEFT safetensors is reported and skipped rather than failing deep inside the load vLLM --enable-lora --lora-modules, with --max-lora-rank 64 because the default 16 refuses most trained adapters and blames the flag rather than the adapter QLoRA needs nothing extra: the quantisation is how the BASE model loads (load_in_4bit), the adapter is an ordinary LoRA. The manager now passes the model's entry to the backend as `model_config`, which is how any of this reaches them. On RunPod the asymmetry is the interesting part. A diffusion LoRA is sent with the request, because that API carries adapters per request. A text LoRA is part of how the model LOADS, so it must resolve when the pod starts: only HuggingFace repo ids travel to a vLLM/llama.cpp pod, and a local path is reported and skipped instead of being baked into a launch command that would fail minutes later inside a pod nobody can inspect. A coderai pod gets the adapters in its seed and applies them itself. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pod cannot see this machine's disk, so "use my local models on RunPod" needed a way for the pod to GET them. Three, now, picked by `source`: auto — an explicit hf_repo, a path that already is a repo id, or a repo id recovered from the HuggingFace cache path (models--Owner--Repo). Most of a normal catalogue resolves with nothing typed: weights downloaded through coderai live in that cache, and the directory name still says where from. hf — download hf_repo from HuggingFace. url — download model_url directly, which is how most one-off GGUFs are had. upload — send the weights from here. Each server gets only what it can use, refused at configuration time rather than after five minutes of pod boot: vLLM is launched `--model` and takes a repo id only; llama.cpp takes `-hf` or downloads a URL with `-mu`; upload works only on a coderai pod, because vLLM and llama.cpp need the file before their container starts while a coderai pod loads models on demand. Upload is a real option, not a token one — on symmetric gigabit a 30 GB model is a few minutes — but it is the cold cost of every NEW pod, so it is never chosen automatically. /v1/models/upload streams to disk in chunks (never into memory), extracts a tar for multi-file models with every member checked against the destination, and registers the entry in memory only. An existence check by name and size makes a repeat free and refuses a half-copy. keep_warm keeps one pod running so nothing waits for a cold boot. OFF by default and deliberately: a warm pod bills every hour of every day, including the ones nobody uses it. Warm pods start at startup, since a pool created on first use would still pay the boot it exists to avoid. All of it is in the UI: weights source with the right field shown per choice, keep-warm, and an "always run this model locally" pin. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
PER-MODEL PLACEMENT FOR ANY KIND. Sending a model to RunPod used to work only for text: resolve_pod_engine looked at whether the path ended in .gguf, and RunpodBackend implements chat only. A diffusion model pinned to runpod would have been handed to vLLM, which cannot serve it. Now the model's OWN KIND picks the pod: an image/video/TTS/STT/embedding model gets that capability's published coderai image, a GGUF gets llama.cpp, an HF repo gets vLLM. The gateway routes by model BEFORE capability, so two models of the same kind can differ — one video model pinned local with `placement: "local"` while another runs on its own pod with its own budget, which capability-level routing could never express. A pod rented for one model starts with an EMPTY catalogue and would refuse every request for it, so pod_plan seeds CODERAI_SEED_MODELS with that model's entry and the pod registers it at startup (in memory only — a pod is disposable, and writing it would surprise anyone setting the variable on a real install). Only fields that describe the model travel: never local paths, which mean nothing there, and never this deployment's placement settings, which would have the pod renting pods of its own. ORCHESTRATION STAYS LOCAL. /v1/pipelines, /v1/characters and /v1/environments are no longer gatewayable capabilities. They are a sequence of calls to other endpoints, and they already dispatch each step through the front as its own /v1 request — so every step is placed by its own model's config, and a chain can mix local and remote freely. Forwarding the chain would have moved the logic to a pod and placed every step by that pod's catalogue instead. LORAS FOLLOW THE REQUEST. A LoRA is a file on this machine; a pod has never seen it. The blob store is content-addressed, so each adapter is hashed, checked against the remote's /v1/loras/blob/<hash>, uploaded only when missing, and the reference rewritten to that id — repeat requests upload nothing. QLoRA needs nothing extra: the quantisation is in how the base model loads, the adapter is the same safetensors file. An HF id or URL is left alone for the remote to fetch. Also: a vLLM pod now refuses a local model path up front, with the same reasoning as the GGUF guard — otherwise it boots for minutes and dies — and passes `quantization` through, since a local load_in_4bit does not carry over to vLLM. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Now that ghcr.io/nextime/coderai-<capability> is public, a capability pod needs no image name: leaving it blank uses the published image for that capability. Enabling a capability on RunPod is one dropdown. Capabilities that ride another profile's dependency set are mapped — rerank to embeddings, speaker to stt, stems/audio_clean/audio_gen to audio. The four with no published image (loras, characters, environments, pipelines) return no default and say so, rather than defaulting to a tag that 404s minutes into a pod boot. CODERAI_CAPABILITY_IMAGE_REPO / _TAG point the whole set at a fork or a private registry; a row's own `image` still wins. The UI shows the default as each row's placeholder, so it is visible rather than implied. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The package name is the last path segment of ghcr.io/<owner>/<name>:<tag>. Stripping only the registry left the owner in the name, so every URL the script printed after a push 404'd — on the one step the user has to do by hand, because GitHub has no API for package visibility. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The Dockerfile split core into its own RUN so the nine images would share it. Measuring the built result showed they did not: only 4 of 11 layers were shared, and each image carried its own ~7 GB copy. An identical `pip install` does not write identical bytes (dist-info metadata, RECORD files), so the layer digests differ even when the command and inputs match. Sharing by coincidence does not work. Dockerfile.capability-base builds the core (OS + apt + torch + the import-time deps) ONCE as its own image, and each profile is FROM it, so the sharing is structural. build_capability_image.sh builds it on first use and reuses it after (REBUILD_CORE=1 to force, needed when core.txt changes). Measured after rebuilding all nine: every image now carries all 8 core layers and adds 2 of its own (audio 5, from its build-deps apt churn). 29 distinct layers instead of 67, local disk 40.9 GB instead of 94, and a push uploads the 6.79 GB core once rather than nine near-identical copies. A pod pulling a second capability image already has most of it. Also corrected the publish script's claim that `gh` could make the packages public: GitHub's Packages REST API is list/get/delete/restore only, there is no visibility endpoint, and new packages are always private. It now prints the per-package settings URLs and says plainly that this is a click each. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Every `base-pre-<stamp>` rollback tag pins a whole ~26 GB image, and they were never retired: twenty builds had accumulated 245 GB of bases that nothing would ever read. The flatten now keeps KEEP_BACKUPS (default 1) and retires the rest, so there is always exactly one image to roll back to. KEEP_BACKUPS=0 keeps none. It also drops the intermediate `flat-<stamp>` tag by default (KEEP_FLAT_TAG=1 to keep it): it points at the same image the base tag does, so it frees nothing and only makes `docker image ls` unreadable. Tested end to end against ubuntu:24.04 — a real export/import/verify cycle in seconds rather than six minutes: config equality passes, the new backup is made, three older ones are retired, and KEEP_BACKUPS=0 leaves none. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Several models can share one RunPod pod until it saturates. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two models that both offload rented a card each, even when neither came close to filling one. They can now name a shared `pool` — on the model's runpod block or on a burst target — and land on the same pod; each request carries its own model name, and the pool rents a second pod only once the first is carrying scale_up_inflight_per_pod requests. Sharing is refused, loudly, for pod servers pinned to one model at launch: a vLLM pod is `--model X` and a llama.cpp pod `-hf one.gguf`. Allowing it would send requests to a pod that answers with the wrong model's weights, or 404s — a clear error at configuration time beats debugging that. `engine: coderai` (a whole coderai, which picks the model per request) is what makes a shared pool possible. The first model to create a shared pool sets its shape; later joiners do not overwrite it, so whichever model happens to load last cannot silently redefine the budget everyone is sharing. Capability pools now use the same rule instead of their own ad-hoc guard. Settable in the UI on both the model's pod section and its burst target. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model placement (local / remote / burst), pod auth and real concurrency scaling, and a UI for all of it. Co-Authored-By:
-
- 13 Sep, 2026 20 commits
-
-
Stefy Lanza (nextime / spora ) authored
None of this session's work had any UI, so all of it was JSON-only. Now: MODEL PAGE — a Placement section: serve this model from a remote URL (with its name and token on that remote), and burst-to-RunPod with the three triggers (no free slot / queue full / no local GPU) and a target that is either a managed pod or your own serverless endpoint. The pod form gained engine (auto/vLLM/llama.cpp/coderai/custom), the GGUF to download, image, health path, docker args, the private-registry credential, the pod's bearer token with an explicit "leave it open" opt-out, and the scaling knobs (scale-up threshold, max in-flight per pod, sticky sessions). SETTINGS PAGE — a remote service URL for each of the five engines (ds4, colibri, k3, ktransformers, vLLM), the account-wide registry credential, and a Remote capabilities card: one row per capability, each Local / Remote URL / RunPod pod, with image, shared pool name, VRAM, $/hr, max pods and idle timeout. Clearing a remote field now REMOVES it. The save path only writes keys that are present, so sending an empty value as "unchanged" would have left a stale remote in place and the model would have kept going to a machine the user thought they had detached. Tests assert what is easy to get silently wrong in UI work: every getElementById reads an id that exists (counting ids the page assigns to elements it creates), every new field is both populated and collected, every capability the gateway can route has a row, and the admin API keeps every key the form sends. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three things, one of which is a security fix. POD AUTH. backends/runpod.py said "the pod's vLLM server needs no auth" and sent no credential. A RunPod proxy URL is reachable by anyone who learns it, so every pod we have ever started was an open GPU on the public internet. Pods are now launched locked to a bearer token, and blank does NOT mean open: the pool generates one per pool unless `allow_open_pod` is set explicitly. Each server takes it its own way — `--api-key` for vLLM and llama.cpp, CODERAI_API_TOKEN for a coderai pod, which required teaching BearerAuthMiddleware to accept an env token since a pod has no auth.json to seed. The health probe sends it too: a locked pod answers 401 on /v1/models, which would otherwise look exactly like "never became ready". All three callers (always-remote backend, capability gateway, front spill) now carry it. SCALING. scale_up_inflight_per_pod was parsed and stored but never used: the pool grew only when it had NO healthy pod, so with max_pods=3 fifty concurrent requests all piled onto pod #1. It now grows when the least-loaded pod reaches the threshold, in the BACKGROUND — the request that proved the pool needs to grow is served by an existing pod rather than waiting minutes for a cold one. `max_inflight_per_pod` (0 = let the remote's batching absorb it) makes requests wait for a slot instead of piling on, for engines that degrade under concurrency rather than queueing internally. CACHE. `sticky_sessions` sends a conversation back to the pod that already served it, so the remote's prefix/KV cache still holds its context; round-robin would re-prefill every turn from scratch. Bounded to the newest 4096 conversations. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Spillover already existed, but not in the shape that is actually useful: it fired only when the WAIT QUEUE overflowed (on_concurrency_full) or when no GPU could serve the model, and it could only reach a serverless endpoint — the code said outright that a pods target "goes through the engine and is not handled here", which meant bursting to our own pods was not implemented at all. Two additions: `on_busy` — offload the moment there is no free local slot, instead of queueing behind the running generation. Built on FrontQueue.try_acquire, a non-blocking take: the front has to learn the model is occupied WITHOUT committing to wait for it. Wired into both the buffered and the SSE dispatch paths. `target.mode: "pods"` — borrow a pod from the managed pool (provision on demand, health-gate, scale, reap, bill against the same caps) for one request and hand it straight back, so an overlap that never recurs is reaped rather than left running. Serverless and pods now share one `_spill_open` primitive, with a Response wrapper for the buffered path and a chunk generator for the SSE path. Local stays the safety net in both directions: a burst that cannot be opened falls back to the local queue, never the reverse. Two bugs caught before shipping, both in my own first cut: the direct path referenced _swap_tok before it is assigned (NameError on every burst), and the SSE path left _qkey set after bursting, so cleanup would have released a slot it never held — handing a phantom slot to the next waiter and over-subscribing the model. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
voice needed build-essential: f5-tts/seed-vc pull webrtcvad, a C extension with no cp312 wheel. Same purged-in-one-layer mechanism audio uses for its Rust toolchain. All nine built and healthy (each answers /healthz): ocr 6.93 embeddings 7.26 audio 7.31 faceswap 7.57 tts 7.61 stt 7.63 images 7.74 video 7.83 voice 8.33 (GB) They share the core layer, so the disk and push cost is one core plus nine small deltas, not nine whole images. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
publish_capability_images.sh does the whole thing: disk check, registry login (with the exact ghcr token instructions), build, smoke test, push, and the per-package URLs for flipping GHCR's default-private packages to public. It is resumable — an image already built for this version is skipped, so re-running after a failure doesn't rebuild 7 GB of torch — and it pushes nothing until every image has answered /healthz. Profiles can now declare apt packages their wheels need to COMPILE, in `profiles/<name>.build-deps`, installed and purged inside a single layer. `audio` needed it: deepfilternet pulls deepfilterlib, which has no cp312 wheel and compiles Rust extensions. It now builds at 7309 MB against 7260 for a profile that needs nothing, so the toolchain is provably not in the image. Built and healthy so far: images (7.74 GB), faceswap (7.57 GB), embeddings (7.26 GB), audio (7.31 GB). ocr, stt, tts, video and voice are untried — the script resumes into them. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three fixes the first real build exposed. The import check caught what a lazy-import survey had missed: app.py mounts every router, and a few touch libraries at module level rather than lazily — soundfile (voice_convert), cv2/PIL (faceswap), litellm (parser), websockets (broker client). A static walk of the import graph from codai.api.app found the full set; they are now in core.txt with a note saying why. torch and transformers are NOT among them, so they stay per-profile. torch moved into core so it becomes ONE shared layer. Nine images each building their own copy would duplicate ~5 GB nine times — on disk and on every push. Core is now its own RUN, identical across profiles, so all nine share it. A BuildKit pip cache mount means building all nine downloads torch once rather than nine times, which matters when the link is also pulling models. build_all_capability_images.sh drives the set. Sequential on purpose: they share the cache and the core layer, so parallel builds would race on both. Each image must answer /healthz before the next starts. Validated: coderai-embeddings built, imported and answered /healthz (7259 MB) before this restructuring. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The export/import flatten has been a manual ritual. `docker export | docker import` drops the image config entirely, so it has to be rebuilt by hand every time — and a silently lost ENTRYPOINT or PATH produces an image that builds, ships, and then fails to start. This scripts it, emits every value in JSON form (so CODERAI_EXTRA_ARGS' spaces survive), and then VERIFIES the flattened config matches the source before it moves the base tag. It refuses to retag on a mismatch. It also strips secret-valued env vars. HF_TOKEN was baked into the image config — readable by anyone who can pull it, `docker image inspect` is enough — and had been carried forward since at least the 20260909 base. It arrives via `docker commit` of a container started with -e; the repo itself is clean (not in any tracked file, not in history). run_oci.sh already passes the token at runtime, so removing it from the image changes nothing. Names are reported, values never echoed, and the equality check knows the strips are intentional. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Remote execution across the board: every model and capability can be served from another machine, and RunPod pods can be rented per model, per capability or shared between them. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two things that both come down to not renting more GPU than the work needs. Shared pools: a pod is a whole GPU, and one coderai can serve images, video and TTS from it. Capabilities naming the same `pool` now share one set of pods and one budget; without it, three capabilities rented three cards to do what one can. The pool is configured once on whichever capability carries the settings — a bare `{"pool": "media"}` reference will not reset it to defaults. Trimmed images: packaging/runpod/ builds a coderai carrying ONE capability's dependencies, since the flattened base is ~26.6 GB and every cold pod pays that pull. Profiles under packaging/runpod/profiles/ — core plus images/video/tts/stt/voice/audio/embeddings/ocr/faceswap — work because the routers import lazily, so a pod only needs what its own endpoints touch. The Dockerfile fails the BUILD if a profile is missing something the app imports, and build_capability_image.sh then boots the container and waits for /healthz. Both checks exist because the alternative is finding out as a boot timeout on a rented GPU, with nothing in the logs saying why. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Audited all 66 /v1 endpoints against the capability map. One was unmapped: /v1/audio/progress, so polling an audio job would be answered locally while the job ran on the remote — always empty. Mapped to audio_gen, and a test now asserts the whole set, so a newly added endpoint cannot quietly stay local. The bigger hole was results. With the default response_format "url" a render answers with a URL built from the REMOTE's base (http://pod-xyz:8000/v1/files/out.png): unreachable from the client, and a 404 if fetched here because the file is on the pod. The gateway now rewrites those URLs to this instance and remembers which remote produced each file, so GET /v1/files/<name> follows back to it. Only JSON bodies are buffered for the rewrite; everything else still streams untouched. The map is a bounded routing hint (newest 4096), not a store: a pod reaped for idleness takes its outputs with it, so fetch what you need or ask for b64_json. Documented. Admin, /internal, /chat, /login and /healthz stay local by design — they administer or identify THIS instance. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Capability pods run YOUR image, and nobody publishes their whole stack publicly — but create_pod never sent a credential, so a private image would fail to pull and the pod would just never become healthy, with nothing in the logs pointing at why. RunPod stores registry credentials for you (Settings -> Container Registry Credentials) and gives each set an id; that id now rides along as `containerRegistryAuthId`, from `runpod.registry_auth_id` account-wide or a pod block's own `registry_auth_id`. Public images (vLLM, llama.cpp) are unaffected — they send nothing. docs/remote-execution.md now says where the images come from: LLM pods need no publishing at all, a capability pod needs a coderai image in a registry RunPod can reach, and the 26.6 GB flattened image is paid on every cold pull. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod pool never cared that it was serving one LLM — it provisions, health-checks, scales and reaps whatever image it is given. Two things were hardcoded: the image and the /v1/models readiness probe. Both are now per-pool, via pod_plan(): auto vLLM, or llama.cpp for a GGUF -> /v1/models coderai your image (required) -> /healthz custom your image + docker_args -> health_path So a capability can now be a pool instead of a fixed URL. Configure `remotes.pods: {"images": {...}}` (the same fields a model's runpod block takes) and the gateway borrows a pod per request and hands it back; `"endpoints": {"images": "runpod"}` is the shorthand. Everything that already guards LLM pods now guards these: $/hr ceilings, rolling spend limits, the global budget, min/max autoscaling, idle teardown, the stale -pod reaper, and the stats page (as `capability:<name>`). That closes the gap this work left open: an idle image or video pod is exactly the kind you forget about, and until now nothing would reap it. A pod that cannot be provisioned — RunPod disabled, budget hit, no GPU under the ceiling — returns 502 rather than falling back to running the model locally. Silent local fallback is a failure mode you only notice by watching your own VRAM vanish. 22 tests, including pod acquire/release (the handle is returned even on the happy path, so the reaper can see it), the runpod shorthand, the provisioning-failure 502, and every pod_plan branch. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
18 tests over local stub servers — no GPU, no weights, no network: worker service_url health enforcement, the mux service + RemoteMuxEngine (streaming, stats, mid-stream cancel, error propagation, dead service), remote GGUF text models (routing, /v1 normalisation, auth, chat, SSE, no-local-weights), RunPod engine selection (HF->vLLM, GGUF->llama.cpp, pins, the missing-hf_gguf error), and the gateway (transparency when nothing is remote, capability and per-model routing, verbatim multipart forwarding, chat exclusion, clean 502). Also stop the mux service dumping a broken-pipe traceback when a client cancels a stream — hanging up IS the cancel signal there. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Images, video, embeddings, rerank, OCR, TTS, STT, voice cloning, audio generation, stems, 3D and pipelines all load their model in-process, so none of them had a service to point elsewhere. Writing thirteen services would be the long way round; another coderai already exposes all of them, so it can be the service. RemoteGatewayMiddleware sits in front of the routes and, when a request is addressed to something configured as remote, replays it verbatim to that endpoint and streams the answer back — JSON and multipart alike, since the body is forwarded as raw bytes. Routing is per model (`service_url` on the models.json entry) first, then per capability (`remotes.endpoints` in config.json, or CODERAI_REMOTE_<CAP>_URL). Placement matters and is deliberate: it is registered first, so it ends up innermost — auth, rate limiting and request logging all run before it, and a request still has to be authorised here before being forwarded. Chat and completions are excluded: those go through RemoteOpenAIBackend, which keeps usage accounting, tool parsing and the manager's bookkeeping. When nothing is configured remote it returns on its first branch and never buffers a body (cached 5s so it doesn't walk the model list per request). Bodies over `max_body_mb` are served locally rather than held in RAM. A non-forwarded request replays its buffered body downstream untouched — tested, including multipart uploads that still parse locally. docs/remote-execution.md documents all four remotization mechanisms together. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two additions, one goal: no text model is stuck on this box. 1. codai/backends/remote_openai.py — set `service_url` on a model entry and coderai proxies that model to any OpenAI-compatible endpoint (another coderai, llama-server, vLLM, SGLang, a rented pod) instead of loading it. `served_model` renames it for the remote, `api_key` adds a bearer token. It reuses RunPod's request/SSE machinery and only replaces endpoint resolution, so streaming and usage accounting are the same code path. This is what makes the GGUF/llama.cpp catalogue remotizable: those models load in-process through llama-cpp-python, so there is no service to redirect — pointing the *model* at a URL is the only redirect available. The manager checks it first, ahead of every engine route, and `_model_is_runpod` grew into `_model_is_remote` so such models are never downloaded, cached, or charged local VRAM (old name kept as an alias). 2. RunPod pods pick their server per model. A GGUF cannot simply go to vLLM: its GGUF loader is experimental — single file only, a limited architecture list, and it still wants the original repo's tokenizer. So `engine: auto` sends an HF repo to vLLM and a GGUF to llama.cpp's own server image (ghcr.io/ggml-org/llama.cpp:server-cuda), which downloads the weights itself from `hf_gguf` ("user/repo:Q4_K_M") — a local /AI/….gguf path means nothing on a rented machine, and we don't upload multi-GB weights. `engine: vllm|llamacpp` overrides. Both servers answer /v1/models, so the readiness probe is unchanged. Tested against fake servers: routing, /v1 normalisation, auth header, chat, SSE streaming, usage, and each engine-selection branch. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
These two were the exception in the last commit: they are driven over a stdin/stdout mux protocol, so there was no URL to redirect. This adds the missing HTTP boundary. tools/mux_service.py hosts a real MuxEngine (colibri or k3 — both use the same class) and exposes /health, /run (NDJSON: text chunks then stats), /pause and /resume. codai/api/mux_remote.RemoteMuxEngine is the client and presents the exact surface the backends use — run/is_alive/pause/ resume/close/log_tail/family — so ensure_engine() can hand one back and nothing downstream can tell local from remote. Cancellation travels as a dropped connection: the service's write fails and it stops generating, matching the local `cancelled` callback semantics. `service_url` is now a real config field on all five engine backends (ds4, colibri, k3, ktransformers, vllm), settable through the admin API and persisted, rather than only an env var. Tested end-to-end against a stub engine: streaming, stats, mid-stream cancellation, pause/resume, and error propagation all round-trip. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Each `ensure_service()` already returns a base URL, so a worker can be pointed at a service running elsewhere — another host, a container, a rented RunPod pod — without anything downstream noticing. Only the TTS backends had that escape hatch (parler/melo, checked at the caller); now the check lives in the workers themselves, per-model via `service_url` in the config plus a CODERAI_*_SERVICE_URL env var: vllm, ds4, kt, h3, pyannote, melotts, canary, crisperwhisper, parler Each verifies the remote's health check before returning it, so a stale or wrong URL fails loudly instead of surfacing as a confusing timeout later. vLLM additionally checks that the remote actually serves the requested model — the OCR path asks for surya-2 by name, and a generic remote vLLM would otherwise be proxied silently. colibri/k3 are left out: they speak a stdin/stdout mux protocol, not HTTP, so there is no URL to redirect. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ships the LoRA trainers for LTX-2, MiniMax-H3 and Krea 2, and the overlay venv that lets the last two run without moving the server off diffusers 0.38. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
H3 and Krea 2 need diffusers >= 0.40 while the server runs 0.38. Upgrading in place is not free: measured on this box, SDXL comes out bit-identical but Z-Image drifts (mean |diff| 0.0146, max 0.32, against a same-version control that is bit-identical) — so the upgrade would silently change image output for a model that is working fine. Since diffusers is imported once per process, the split only has to be per-process. The H3 venv is now created with --system-site-packages and given nothing but diffusers>=0.40 and huggingface_hub>=1.23; torch, transformers and the rest are inherited from the server's venv. That is 75 MB instead of the several GB a standalone venv costs, and keeps one torch on the machine. CODERAI_H3_STANDALONE_VENV=1 still builds a fully independent venv from requirements-h3.txt, for the day the parent's torch is too old. Training follows the same boundary: tools/lora_train_worker.py runs one job in the overlay — references written out as PNGs, request flattened to JSON, progress returned as JSON lines and mirrored onto the usual /v1/loras/progress so polling is unchanged. _train_via_overlay() drives it. Routing is by capability, not hardcoded: _diffusers_has(arch) tries to load the classes, so wan and ltx2 train in-process (0.38 has them) while h3 and krea go out-of-process — and if the main venv is ever upgraded they move back in-process with no config change. Verified: the overlay resolves diffusers 0.40 with torch/transformers inherited and leaves the parent untouched; a probe job ran under it, reached _train_flow_dit and reported back through the job file. The trainers themselves are still unrun against real weights. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
We had trainers for SD1.x/SDXL, Z-Image and Wan — none for the architectures the LoRAs people actually publish for video today. All three new ones are the same training problem as Wan (encode references to VAE latents, encode the prompt once, adapt the transformer's attention projections, regress the rectified-flow velocity), so rather than a fourth, fifth and sixth copy of that loop there is one shared _train_flow_dit driven by a per-architecture description in codai/api/lora_archs.py: which classes to import, the latent shape, the timestep shift, the text-encoder contract. Selection: `target` accepts image | video | wan | ltx2 | h3 | krea. `video` keeps meaning Wan for existing configs but now reads the model id, so an LTX-2 or H3 path routes itself; the explicit names force one. The character studio exposes the same choice as --target and a dropdown. Availability differs per architecture and fails with a message that says so rather than an ImportError from inside a training job: LTX-2 works on this diffusers (0.38 has LTX2VideoTransformer3DModel + AutoencoderKLLTX2Video), while H3 and Krea 2 need >= 0.40 — and Krea 2 additionally needs access to the gated krea/Krea-2-* weights. Verified: architecture detection and routing for every target/model combination, and that each trainer's classes either load or explain what they need. NOT run against real weights for any of the three — the contracts match the proven Wan path, but the first real run should be treated as a debugging session. Co-Authored-By:
-