- 14 Sep, 2026 34 commits
-
-
Stefy Lanza (nextime / spora ) authoredThe first RunPod pass over every capability produced four failures, three of which were ours: The TTS probe sent voice "default", which no shipped model has — Kokoro loaded fine on a rented pod and then answered "Voice default not found in available voices". TTSRequest already defaults to a real voice, so the probe stops inventing one. OCR engines are chosen by name (paddle|doctr|surya) and are not models.json entries, so the test refused with "'surya' is not in the model list". _describe now recognises them. For a text model the refusal offered "a remote for its text capability" as a fix. It is not one: /v1/chat/completions is outside the gateway, so that config routes nothing and the test would then report a remote pass for a request served locally — the exact failure the forced-remote check exists to prevent. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
Stefy Lanza (nextime / spora ) authored
A rented images pod downloaded SDXL for ten minutes and then failed with "No package metadata was found for bitsandbytes". models.json carries quantization settings for EVERY model type, not just LLMs, but bitsandbytes was only in the text profile. It moves to core, the shared layer under all ten images. stt and ocr were reachability-only, which reported ok:true while proving nothing beyond "a URL is configured" — a pass that would survive a completely broken pod. Both inputs can be synthesised honestly: espeak speaks a known sentence, PIL draws known words. The result is checked against what went in, so a model that returns fluent nonsense fails too. When the synthesis itself cannot run (no espeak, no fonts) the probe falls back to the reachability check rather than blaming the model for a gap on this side. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two bugs, both found by the first end-to-end embeddings test on a rented pod. A capability image's entrypoint is `uvicorn codai.api.app:app` — it never runs codai/main.py, so set_config_manager() was never called and codai.admin.routes.config_manager stayed None. Both the CODERAI_SEED_MODELS list and the runtime catalogue write read through that global, so a pod seeded with one model answered "not available. Use one of: " with nothing after the colon. The lifespan now builds its own ConfigManager when none was set — loading it is what parses the seed list — and says how many seeds it registered. The bearer token a pool launches its pods with is generated in memory when none is configured. A pod that outlived the process that rented it was then adopted with a DIFFERENT token and answered 401 for the rest of its paid life (the test run reported serves: "(HTTP 401)"). The token is now recorded in the shared pod registry and adopted along with the pod. Verified live: bge-m3 on a rented RTX 2000 Ada returned a real 1024-dimension embedding, 67.9s cold. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Registration reaches the model manager, seeded models survive the engine assignment filter, siblings wait for a booting pod, and pod boot phases plus catalogues are reported for diagnosis. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The last three pod failures were each diagnosed the same way: find the pod URL in the engine log, ask the pod by hand what it serves, work backwards. That is a diagnosis loop measured in minutes of log archaeology per attempt. Boot is now reported in PHASES, because "it took 150s" is not a diagnosis: port open after Ns (image pull + container start), serving after N more. The two have different fixes — a bigger boot_timeout_s versus a bigger load_timeout_s. The moment a pod is up, its /v1/models is logged. An empty catalogue now says so outright — "pod serves NO models — it will refuse every request" — instead of surfacing minutes later as "Model 'x' is not available. Use one of: " with nothing after the colon. And a test run's answer carries the pod diagnostics: which pods are live, their GPU, uptime, spend so far, URL, and the models each one lists. The fact that mattered in every failure so far was the catalogue, and it is now in the response rather than three commands away. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod finally answered, and answered "Model 'bge-m3' is not available. Use one of: " — with NOTHING after the colon, despite being seeded with five models and successfully taught a sixth. Two causes, both only visible on a real pod. REGISTERING A MODEL WAS HALF DONE. Both the boot seed and /v1/models/register wrote models_data, but request validation asks the MODEL MANAGER for its allowed identifiers. A model in the catalogue that the manager never heard of is refused. Registration now calls the right setter per section, maps the alias a request actually uses (bge-m3 -> BAAI/bge-m3), and widens the engine's assigned-model set — a model registered at runtime was assigned by nobody and would otherwise be usable but invisible. AND THE SEED WAS FILTERED OUT BEFORE IT GOT THERE. A pod runs front plus engine; the front computes each engine's assignment from models.json, and the seed is deliberately in-memory only, so the engine was told to register nothing — dropping exactly the models the pod was rented for. Seeded entries now survive that filter. Separately: both engines rented their own pod again, 45 seconds apart, because a pod takes ~2.5 minutes to boot and the sharing check only saw READY pods. A pod is now registered at creation, and a sibling that finds one still booting WAITS for it instead of renting a second card. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Cross-engine pod sharing, network volumes for weights/uploads/staging, and the optional venv-on-volume slim pod image. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A capability image is ~7 GB and nearly all of it is torch, pulled again by every cold pod. With a network volume those libraries can live there instead: `venv_on_volume` boots a slim image — OS, Python and coderai only — and runs coderai from a venv on the volume. The first pod builds it; every pod after skips both the pull and the install. Opt-in per pool and per model, never a default, because the trade is real and unmeasured: the pull gets much smaller, but importing torch off network storage is slower than off local disk, so some of the saving comes back at import time. Whether that is a net win depends on how often pods cold-start, which is the user's workload, not mine to assume. Two hazards are handled rather than left to chance. The ready marker is written AFTER the install, so a pod that dies mid-build leaves nothing a later pod would import from. And a lock directory — mkdir is atomic — makes a second pod wait for the first instead of two installs interleaving into the same directory. `venv_on_volume` without a volume is refused with the reason, rather than booting a pod that installs 7 GB onto container disk and discards it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod's disk dies with the pod, so every cold pod re-downloads its weights and every upload has to be repeated. A network volume — created by hand in the console, coderai never creates one — makes that a one-off. Attaching it is the easy half; the useful half is pointing HF_HOME, the diffusers/transformers caches, CODERAI_MODELS_DIR and the staging directory at the mount. A volume attached without redirecting those changes nothing, which is the mistake this avoids. That also answers the bounce-host question: an upload lands on the volume and every later pod sees it, so a QLoRA or a local model can be sent once and used from any pod — no third-party storage, no public IP. Two RunPod constraints are enforced rather than discovered the hard way: volumes are Secure Cloud only (cloud_types is forced), and a pod must run in the volume's own data center (looked up once and pinned, instead of letting price ranking pick a region where the attach fails as a confusing capacity error). Settable account-wide in Settings or per pool. Container images still come from a registry — a volume holds weights, not the image. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ten images built, none pushed: the guard checks free space against every profile before doing anything, but a push uploads what already exists and needs no disk. It now measures only what actually has to be BUILT — and says so when there is nothing to build — so a full disk can no longer turn 'publish' into a no-op. It also asks docker where its root dir is rather than assuming /var/lib/docker, which on this machine is /storage/docker. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The second live test showed nvidia and radeon each renting their OWN pod for 'capability:embeddings' — two GPUs, two bills, while max_pods said 1. Same root cause as the reaper bug: one engine per GPU, each with its own pools, none able to see the others. Before renting, a pool now looks in the shared registry for a healthy pod another process already has for the SAME pool key, and borrows it. The registry entry carries the URL, recorded once the pod is serving. A borrowed pod is released, never terminated: its owner holds the lifecycle and the bill, and killing it would take it away from the engine still using it. It is also billed at 0 here, so the cost is not double-counted against the caps. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Capability pods are told what they can serve, and any pod can be taught a model at runtime (HF id, URL or upload) instead of being a catalogue frozen at boot. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod knew only what it was seeded with at boot, so any request naming another model got "not available" for the rest of that pod's life. That makes a pod a fixed catalogue rather than an extension of the local system, which is the whole point of it. When a remote refuses a model it does not know, the gateway now teaches it and retries once: resolve the model to something the remote can fetch (HuggingFace id, URL, or an upload when it exists nowhere else), POST it to the new /v1/models/register, and replay the request. Registration is in memory — a pod is disposable — and idempotent, so a model already loaded is never disturbed. Taught once per pod, tracked by (base, model). The retry is bounded to one attempt and only fires on a 400/404 whose body actually says the model is unknown: a 500 or an out-of-memory must never turn into a retry loop against a rented GPU. A model that resolves to no fetchable source is refused with that reason rather than registered as something the pod could never load. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The first pod that survived long enough to answer returned: "Model 'bge-m3' is not available. Use one of: " — nothing after the colon. A pod starts with an EMPTY catalogue. A per-model pod is seeded with its one model, but a capability pod has no model entry at all, so it refused everything. Capability pods are now seeded with this deployment's models OF THAT CAPABILITY — only ones the pod can actually fetch (a HuggingFace repo id or a URL), since registering a local path would turn a clear "not available" into a confusing load failure. The seeding was also unreachable for any pod with an explicit image: the coderai branch returned early only when the image was UNSET, and capability pods always have one now (the published default). Both cases go through one path that carries the auth token and the catalogue together. What the live run did prove, before failing on the catalogue: GPU selection with price ranking and capacity fallback, the boot health gate, pod bearer auth (our token accepted; an unauthenticated call gets 401), gateway forwarding to the pod URL, idle reaping after 120s, and the spend ledger ($0.033 for the run). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Stop sibling engines reaping each other's RunPod pods: a shared pod registry plus a reap grace period. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The first real pod test found this in seconds: coderai runs one engine per GPU, each with its own pool registry AND its own stale-pod reaper, so each engine sees its siblings' pods as untracked orphans. The nvidia engine created a pod; the radeon engine terminated it four seconds later; radeon created one, nvidia killed that — a mutual kill loop that rented and destroyed pods until I disabled RunPod. Every boot then "failed" after its 300s port timeout, because the pod had been deleted underneath it, and the pool dutifully retried on the next machine. Two fixes, because one is not enough: A shared registry (config_dir/runpod-pods.json, flock'd) records every pod this DEPLOYMENT owns. _known_pod_ids now reads it, so an engine with no pools of its own still knows its siblings' pods are not orphans. A grace period (REAP_GRACE_SECONDS, 20 min) refuses to reap any pod younger than a boot budget, whatever the registry says — the backstop for the race a registry cannot close, such as a process that died between creating a pod and recording it. No pods were left billing: the mutual reaping cleaned up after itself and the account shows zero pods. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Test-run fixes (forced-remote verified before dispatch, empty output is a failure) and config propagation to the engine and front. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Configuring a remote capability in the UI did nothing until a restart, which makes the feature unusable as shipped. The admin API is served by the SYSTEM worker, so a save updates that process's config object. The remote gateway, the engine blocks and the front's burst-to-RunPod path all read a DIFFERENT process's copy. The engine's /internal/reload-config — which the front pushes on a config change — re-read models.json only, and the front's own refresh copied a fixed field list that included neither `runpod` nor `remotes`. Both now carry them: the engine re-reads config.json into its live config on reload, and the front refreshes runpod/remotes along with the rest. Found by trying to use the feature on the running instance rather than by reading the code, which is the only way this class of bug shows up. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Running the new test against the live instance found both immediately, which is the point of it. A FORCED REMOTE TEST COULD PASS LOCALLY. /v1/chat/completions is deliberately outside the gateway (RemoteOpenAIBackend owns it), so the x-coderai-placement header is ignored on that path: a text model with no remote configuration was served HERE while the result reported where=remote, ok=true. A test that passes in the wrong place is worse than no test. Forcing remote is now verified before anything is dispatched, and refuses with what is actually missing. A 200 WITH NO OUTPUT WAS A PASS. The text probe allowed 8 tokens, which a reasoning model spends thinking — Qwen3.5-9B answered with completion_tokens=0, finish_reason=stop, and the test called it ok. The budget is now 64, and an empty result is a failure carrying the numbers that explain it, for text, embeddings, images and rerank alike. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Test run: prove a model works where it is configured to run, with a per-request placement override. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Placement now has many moving parts (image, weights, adapters, auth, budgets, staging) and most of them fail minutes into a pod boot, in a log nobody is watching. POST /v1/models/test runs the smallest real request for the model's kind and reports what happened, with a Test run button on every model page. It dispatches THROUGH THE FRONT, so it exercises the same routing, auth, pod provisioning and staging a real request would — a test that simulated the path would prove nothing about the path. `where` forces placement for that one request via a new x-coderai-placement header the gateway honours: "local" ignores any remote configuration, "remote" refuses to fall back to local. Without that you cannot verify a pod at all: the request would be served wherever the config decided, and a green result would mean nothing. Kinds with no cheap synthetic input — video (minutes of GPU and real money), faceswap, STT, OCR — report a reachability check and say plainly that no generation was run, instead of implying more than was proven. Also fixed test pollution I had introduced: the gateway fixture replaced module-level functions without restoring them, and one test assigned _model_entry_for unscoped. Both leaked into later tests, which passed alone and failed in the suite — the failure mode that teaches you to distrust the suite. Verified stable across repeated runs. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model placement for any model kind, text LoRA/QLoRA locally and on RunPod, weight sources (HF id, URL, upload, pod-side staging), shared pools and optional warm pods. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Your idea, and it removes the last dead end. vLLM and llama.cpp pods could not be given a local model or adapter: no endpoint to upload to, and their server needs the file at launch. But a pod CAN fetch it before the server runs — put the weights on any HTTP host it can reach and stage them at boot. The obstacle was that those images run their server as the ENTRYPOINT, so everything we passed became an argument to the server. RunPod's REST pod API accepts dockerEntrypoint/dockerStartCmd (GraphQL does not), so runpod_client now creates a pod through REST when an override is needed and stays on GraphQL otherwise. The staged start command fetches with curl, wget or python — whichever the image actually has rather than whichever we assumed — extracts a tar for a multi-file model, and then execs the server pointed at what it staged. One definition of the staged path (staged_model_dest) is used by both the download and the server args: computing it twice is exactly how a pod downloads to one path and serves another, which the first cut did, along with emitting two --model flags. Adapters given as a URL ride the same path: --lora-modules for vLLM, --lora/--lora-scaled for llama.cpp. So a local adapter reaches a vLLM pod after all — publish it to a URL instead of uploading it. `source: upload` against a vLLM pod now says what to do instead, rather than only that it cannot. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A text LoRA that exists only on this machine had nowhere to go: vLLM and llama.cpp resolve adapters at launch and have no endpoint to upload one to, so the previous commit could only report and skip it. Such a model is now routed to a coderai TEXT pod — the one pod image that can be sent an adapter — and the transfer is arranged around the fact that a pod is created before anything can be sent to it: 1. the pod is told the adapter's sha256 id, computed HERE, so both sides agree on the name before the pod exists; 2. the bytes go to the pod's content-addressed blob store before the first request, once per pod (a repeat costs one existence check); 3. the pod resolves sha256:<hex> back to a file — text_loras.local_path now understands the content-addressed forms — and applies it. A published adapter still goes to vLLM, which fetches it itself: cheaper and faster, so publishing an adapter you use often remains the better path, and the log says so. A PEFT adapter DIRECTORY is still refused with a reason — the blob store holds single files. packaging/runpod/profiles/text.txt adds the text profile (transformers, peft, bitsandbytes for QLoRA bases). It has to be built and pushed before use; the other nine are published already. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
lora_path and lora_model_dir have been in the model config for a long time, but only the VRAM estimator ever read them: vulkan.py, cuda.py and vllm.py contained zero LoRA references, so configuring a LoRA on a text model silently did nothing. Now all three apply them. codai/models/text_loras.py answers the one question they share — which adapters does this model want, and where are they — across every config shape the field has grown (lora_path, a list, lora_model_dir, a `loras` list of dicts), de-duplicated so the same adapter is not applied twice. Each runtime then does its own thing: transformers PEFT load_adapter + set_adapters, several at once llama.cpp lora_path + lora_scale, ONE adapter and it must be GGUF-converted — a PEFT safetensors is reported and skipped rather than failing deep inside the load vLLM --enable-lora --lora-modules, with --max-lora-rank 64 because the default 16 refuses most trained adapters and blames the flag rather than the adapter QLoRA needs nothing extra: the quantisation is how the BASE model loads (load_in_4bit), the adapter is an ordinary LoRA. The manager now passes the model's entry to the backend as `model_config`, which is how any of this reaches them. On RunPod the asymmetry is the interesting part. A diffusion LoRA is sent with the request, because that API carries adapters per request. A text LoRA is part of how the model LOADS, so it must resolve when the pod starts: only HuggingFace repo ids travel to a vLLM/llama.cpp pod, and a local path is reported and skipped instead of being baked into a launch command that would fail minutes later inside a pod nobody can inspect. A coderai pod gets the adapters in its seed and applies them itself. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pod cannot see this machine's disk, so "use my local models on RunPod" needed a way for the pod to GET them. Three, now, picked by `source`: auto — an explicit hf_repo, a path that already is a repo id, or a repo id recovered from the HuggingFace cache path (models--Owner--Repo). Most of a normal catalogue resolves with nothing typed: weights downloaded through coderai live in that cache, and the directory name still says where from. hf — download hf_repo from HuggingFace. url — download model_url directly, which is how most one-off GGUFs are had. upload — send the weights from here. Each server gets only what it can use, refused at configuration time rather than after five minutes of pod boot: vLLM is launched `--model` and takes a repo id only; llama.cpp takes `-hf` or downloads a URL with `-mu`; upload works only on a coderai pod, because vLLM and llama.cpp need the file before their container starts while a coderai pod loads models on demand. Upload is a real option, not a token one — on symmetric gigabit a 30 GB model is a few minutes — but it is the cold cost of every NEW pod, so it is never chosen automatically. /v1/models/upload streams to disk in chunks (never into memory), extracts a tar for multi-file models with every member checked against the destination, and registers the entry in memory only. An existence check by name and size makes a repeat free and refuses a half-copy. keep_warm keeps one pod running so nothing waits for a cold boot. OFF by default and deliberately: a warm pod bills every hour of every day, including the ones nobody uses it. Warm pods start at startup, since a pool created on first use would still pay the boot it exists to avoid. All of it is in the UI: weights source with the right field shown per choice, keep-warm, and an "always run this model locally" pin. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
PER-MODEL PLACEMENT FOR ANY KIND. Sending a model to RunPod used to work only for text: resolve_pod_engine looked at whether the path ended in .gguf, and RunpodBackend implements chat only. A diffusion model pinned to runpod would have been handed to vLLM, which cannot serve it. Now the model's OWN KIND picks the pod: an image/video/TTS/STT/embedding model gets that capability's published coderai image, a GGUF gets llama.cpp, an HF repo gets vLLM. The gateway routes by model BEFORE capability, so two models of the same kind can differ — one video model pinned local with `placement: "local"` while another runs on its own pod with its own budget, which capability-level routing could never express. A pod rented for one model starts with an EMPTY catalogue and would refuse every request for it, so pod_plan seeds CODERAI_SEED_MODELS with that model's entry and the pod registers it at startup (in memory only — a pod is disposable, and writing it would surprise anyone setting the variable on a real install). Only fields that describe the model travel: never local paths, which mean nothing there, and never this deployment's placement settings, which would have the pod renting pods of its own. ORCHESTRATION STAYS LOCAL. /v1/pipelines, /v1/characters and /v1/environments are no longer gatewayable capabilities. They are a sequence of calls to other endpoints, and they already dispatch each step through the front as its own /v1 request — so every step is placed by its own model's config, and a chain can mix local and remote freely. Forwarding the chain would have moved the logic to a pod and placed every step by that pod's catalogue instead. LORAS FOLLOW THE REQUEST. A LoRA is a file on this machine; a pod has never seen it. The blob store is content-addressed, so each adapter is hashed, checked against the remote's /v1/loras/blob/<hash>, uploaded only when missing, and the reference rewritten to that id — repeat requests upload nothing. QLoRA needs nothing extra: the quantisation is in how the base model loads, the adapter is the same safetensors file. An HF id or URL is left alone for the remote to fetch. Also: a vLLM pod now refuses a local model path up front, with the same reasoning as the GGUF guard — otherwise it boots for minutes and dies — and passes `quantization` through, since a local load_in_4bit does not carry over to vLLM. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Now that ghcr.io/nextime/coderai-<capability> is public, a capability pod needs no image name: leaving it blank uses the published image for that capability. Enabling a capability on RunPod is one dropdown. Capabilities that ride another profile's dependency set are mapped — rerank to embeddings, speaker to stt, stems/audio_clean/audio_gen to audio. The four with no published image (loras, characters, environments, pipelines) return no default and say so, rather than defaulting to a tag that 404s minutes into a pod boot. CODERAI_CAPABILITY_IMAGE_REPO / _TAG point the whole set at a fork or a private registry; a row's own `image` still wins. The UI shows the default as each row's placeholder, so it is visible rather than implied. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The package name is the last path segment of ghcr.io/<owner>/<name>:<tag>. Stripping only the registry left the owner in the name, so every URL the script printed after a push 404'd — on the one step the user has to do by hand, because GitHub has no API for package visibility. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The Dockerfile split core into its own RUN so the nine images would share it. Measuring the built result showed they did not: only 4 of 11 layers were shared, and each image carried its own ~7 GB copy. An identical `pip install` does not write identical bytes (dist-info metadata, RECORD files), so the layer digests differ even when the command and inputs match. Sharing by coincidence does not work. Dockerfile.capability-base builds the core (OS + apt + torch + the import-time deps) ONCE as its own image, and each profile is FROM it, so the sharing is structural. build_capability_image.sh builds it on first use and reuses it after (REBUILD_CORE=1 to force, needed when core.txt changes). Measured after rebuilding all nine: every image now carries all 8 core layers and adds 2 of its own (audio 5, from its build-deps apt churn). 29 distinct layers instead of 67, local disk 40.9 GB instead of 94, and a push uploads the 6.79 GB core once rather than nine near-identical copies. A pod pulling a second capability image already has most of it. Also corrected the publish script's claim that `gh` could make the packages public: GitHub's Packages REST API is list/get/delete/restore only, there is no visibility endpoint, and new packages are always private. It now prints the per-package settings URLs and says plainly that this is a click each. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Every `base-pre-<stamp>` rollback tag pins a whole ~26 GB image, and they were never retired: twenty builds had accumulated 245 GB of bases that nothing would ever read. The flatten now keeps KEEP_BACKUPS (default 1) and retires the rest, so there is always exactly one image to roll back to. KEEP_BACKUPS=0 keeps none. It also drops the intermediate `flat-<stamp>` tag by default (KEEP_FLAT_TAG=1 to keep it): it points at the same image the base tag does, so it frees nothing and only makes `docker image ls` unreadable. Tested end to end against ubuntu:24.04 — a real export/import/verify cycle in seconds rather than six minutes: config equality passes, the new backup is made, three older ones are retired, and KEEP_BACKUPS=0 leaves none. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Several models can share one RunPod pod until it saturates. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two models that both offload rented a card each, even when neither came close to filling one. They can now name a shared `pool` — on the model's runpod block or on a burst target — and land on the same pod; each request carries its own model name, and the pool rents a second pod only once the first is carrying scale_up_inflight_per_pod requests. Sharing is refused, loudly, for pod servers pinned to one model at launch: a vLLM pod is `--model X` and a llama.cpp pod `-hf one.gguf`. Allowing it would send requests to a pod that answers with the wrong model's weights, or 404s — a clear error at configuration time beats debugging that. `engine: coderai` (a whole coderai, which picks the model per request) is what makes a shared pool possible. The first model to create a shared pool sets its shape; later joiners do not overwrite it, so whichever model happens to load last cannot silently redefine the budget everyone is sharing. Capability pools now use the same rule instead of their own ad-hoc guard. Settable in the UI on both the model's pod section and its burst target. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model placement (local / remote / burst), pod auth and real concurrency scaling, and a UI for all of it. Co-Authored-By:
-
- 13 Sep, 2026 6 commits
-
-
Stefy Lanza (nextime / spora ) authored
None of this session's work had any UI, so all of it was JSON-only. Now: MODEL PAGE — a Placement section: serve this model from a remote URL (with its name and token on that remote), and burst-to-RunPod with the three triggers (no free slot / queue full / no local GPU) and a target that is either a managed pod or your own serverless endpoint. The pod form gained engine (auto/vLLM/llama.cpp/coderai/custom), the GGUF to download, image, health path, docker args, the private-registry credential, the pod's bearer token with an explicit "leave it open" opt-out, and the scaling knobs (scale-up threshold, max in-flight per pod, sticky sessions). SETTINGS PAGE — a remote service URL for each of the five engines (ds4, colibri, k3, ktransformers, vLLM), the account-wide registry credential, and a Remote capabilities card: one row per capability, each Local / Remote URL / RunPod pod, with image, shared pool name, VRAM, $/hr, max pods and idle timeout. Clearing a remote field now REMOVES it. The save path only writes keys that are present, so sending an empty value as "unchanged" would have left a stale remote in place and the model would have kept going to a machine the user thought they had detached. Tests assert what is easy to get silently wrong in UI work: every getElementById reads an id that exists (counting ids the page assigns to elements it creates), every new field is both populated and collected, every capability the gateway can route has a row, and the admin API keeps every key the form sends. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three things, one of which is a security fix. POD AUTH. backends/runpod.py said "the pod's vLLM server needs no auth" and sent no credential. A RunPod proxy URL is reachable by anyone who learns it, so every pod we have ever started was an open GPU on the public internet. Pods are now launched locked to a bearer token, and blank does NOT mean open: the pool generates one per pool unless `allow_open_pod` is set explicitly. Each server takes it its own way — `--api-key` for vLLM and llama.cpp, CODERAI_API_TOKEN for a coderai pod, which required teaching BearerAuthMiddleware to accept an env token since a pod has no auth.json to seed. The health probe sends it too: a locked pod answers 401 on /v1/models, which would otherwise look exactly like "never became ready". All three callers (always-remote backend, capability gateway, front spill) now carry it. SCALING. scale_up_inflight_per_pod was parsed and stored but never used: the pool grew only when it had NO healthy pod, so with max_pods=3 fifty concurrent requests all piled onto pod #1. It now grows when the least-loaded pod reaches the threshold, in the BACKGROUND — the request that proved the pool needs to grow is served by an existing pod rather than waiting minutes for a cold one. `max_inflight_per_pod` (0 = let the remote's batching absorb it) makes requests wait for a slot instead of piling on, for engines that degrade under concurrency rather than queueing internally. CACHE. `sticky_sessions` sends a conversation back to the pod that already served it, so the remote's prefix/KV cache still holds its context; round-robin would re-prefill every turn from scratch. Bounded to the newest 4096 conversations. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Spillover already existed, but not in the shape that is actually useful: it fired only when the WAIT QUEUE overflowed (on_concurrency_full) or when no GPU could serve the model, and it could only reach a serverless endpoint — the code said outright that a pods target "goes through the engine and is not handled here", which meant bursting to our own pods was not implemented at all. Two additions: `on_busy` — offload the moment there is no free local slot, instead of queueing behind the running generation. Built on FrontQueue.try_acquire, a non-blocking take: the front has to learn the model is occupied WITHOUT committing to wait for it. Wired into both the buffered and the SSE dispatch paths. `target.mode: "pods"` — borrow a pod from the managed pool (provision on demand, health-gate, scale, reap, bill against the same caps) for one request and hand it straight back, so an overlap that never recurs is reaped rather than left running. Serverless and pods now share one `_spill_open` primitive, with a Response wrapper for the buffered path and a chunk generator for the SSE path. Local stays the safety net in both directions: a burst that cannot be opened falls back to the local queue, never the reverse. Two bugs caught before shipping, both in my own first cut: the direct path referenced _swap_tok before it is assigned (NameError on every burst), and the SSE path left _qkey set after bursting, so cleanup would have released a slot it never held — handing a phantom slot to the next waiter and over-subscribing the model. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
voice needed build-essential: f5-tts/seed-vc pull webrtcvad, a C extension with no cp312 wheel. Same purged-in-one-layer mechanism audio uses for its Rust toolchain. All nine built and healthy (each answers /healthz): ocr 6.93 embeddings 7.26 audio 7.31 faceswap 7.57 tts 7.61 stt 7.63 images 7.74 video 7.83 voice 8.33 (GB) They share the core layer, so the disk and push cost is one core plus nine small deltas, not nine whole images. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
publish_capability_images.sh does the whole thing: disk check, registry login (with the exact ghcr token instructions), build, smoke test, push, and the per-package URLs for flipping GHCR's default-private packages to public. It is resumable — an image already built for this version is skipped, so re-running after a failure doesn't rebuild 7 GB of torch — and it pushes nothing until every image has answered /healthz. Profiles can now declare apt packages their wheels need to COMPILE, in `profiles/<name>.build-deps`, installed and purged inside a single layer. `audio` needed it: deepfilternet pulls deepfilterlib, which has no cp312 wheel and compiles Rust extensions. It now builds at 7309 MB against 7260 for a profile that needs nothing, so the toolchain is provably not in the image. Built and healthy so far: images (7.74 GB), faceswap (7.57 GB), embeddings (7.26 GB), audio (7.31 GB). ocr, stt, tts, video and voice are untried — the script resumes into them. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three fixes the first real build exposed. The import check caught what a lazy-import survey had missed: app.py mounts every router, and a few touch libraries at module level rather than lazily — soundfile (voice_convert), cv2/PIL (faceswap), litellm (parser), websockets (broker client). A static walk of the import graph from codai.api.app found the full set; they are now in core.txt with a note saying why. torch and transformers are NOT among them, so they stay per-profile. torch moved into core so it becomes ONE shared layer. Nine images each building their own copy would duplicate ~5 GB nine times — on disk and on every push. Core is now its own RUN, identical across profiles, so all nine share it. A BuildKit pip cache mount means building all nine downloads torch once rather than nine times, which matters when the link is also pulling models. build_all_capability_images.sh drives the set. Sequential on purpose: they share the cache and the core layer, so parallel builds would race on both. Each image must answer /healthz before the next starts. Validated: coderai-embeddings built, imported and answered /healthz (7259 MB) before this restructuring. Co-Authored-By:
-