- 15 Sep, 2026 12 commits
-
-
Stefy Lanza (nextime / spora ) authoredThe transformers fallback keeps MusicGen working on a pod, but it is not the same thing: audiocraft has melody conditioning and AudioGen, and transformers has neither. Supporting it is not a matter of adding a requirement — it pins torch==2.1.0, torchvision==0.16.0 and torchtext==0.16.0, and torch 2.1.0 has no build for the CUDA these images run, so installing it beside the server would leave the pod with no working GPU torch at all. So it gets its own venv and is driven as a subprocess, the same answer parler-tts and the OCR engines already use here. AudiocraftModel deliberately wears audiocraft's own interface (set_generation_params / generate / generate_with_chroma / sample_rate) so the generation path cannot tell whether the library is in this process, in a venv next door, or absent. The venv is baked into the audio image rather than built on first use: a pod is rented by the second, and pip-installing a library during a cold start costs more than carrying the layer. Order of preference: in-process audiocraft, then the isolated venv, then transformers — which needs no venv but cannot do melody. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
Stefy Lanza (nextime / spora ) authored
None of these were placement or pod bugs: each image booted, pulled its model and got as far as executing. The remote machinery worked; the payloads did not. voice — torchcodec arrived as a transitive dependency of f5-tts, PyPI served the CUDA-13 build, and it died with 'libnvrtc.so.13: cannot open shared object file' after loading the model. The same class of bug as the torch pin: a CUDA-coupled package escaping the index. It now installs from the CUDA index with the rest, resolving to torchcodec 0.11.1+cu128. audio — 'No module named audiocraft'. audiocraft pins an old torch, so putting it in the image would drag torch backwards and undo the CUDA pin. transformers ships MusicGen itself, so a pod serves it through transformers instead; an install that has audiocraft still uses it. Melody conditioning stays audiocraft-only rather than silently doing something different. video — 'TextToVideoSDPipeline.__call__() got an unexpected keyword argument callback_on_step_end'. Older diffusers pipelines do not take it and raise inside the call, minutes in, with the model loaded and the GPU paid for. The kwarg is now added only when the pipeline's signature accepts it. The try/except that used to wrap it caught nothing: assigning to a dict never raises. This one bites local generations too, not just pods. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/ocr names its engine in a multipart field called `engine`, not `model`. The gateway only ever looked for `model`, so an OCR request carried no model as far as it was concerned: per-model placement resolved correctly, the test reported 'target: its own pod', and then the gateway refused with 'nothing configures ocr to run remotely'. Two layers disagreeing about the same request. _MODEL_FIELD maps the endpoints whose field is named something else, and the extractor takes the field name for JSON, multipart and urlencoded alike. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Voice cloning runs on a TTS model (XTTS, F5) but is a different endpoint served by a different pod image, so section alone maps it to 'tts' and it could never be placed as 'voice'. That is why the voice image had never been tested: not a missing probe, a model that could not be pointed anywhere. The harness also gains --add-as, to create a temporary catalogue entry for a capability nobody has a model configured for (music generation, a small video model), instead of requiring the real catalogue to be edited by hand first. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three capabilities were listed as un-probeable. Only one of the reasons held. Voice cloning 'needs a reference sample' — and espeak produces one, with a transcript we know exactly because we chose the sentence. Audio generation was 'minutes of GPU': asking for 2 seconds instead of the 10s default proves the model renders audio for a fifth of the cost. Video was 'real money': the smallest clip a model can make — 9 frames, 256x256, 4 steps — proves the path without paying for a film. 'Expensive' was an argument for asking for less, not for proving nothing. Generated media has no ground truth, so these check that media came back at all; a 200 carrying nothing is still a failure. Face swap stays reachability-only and now says why: a drawn face does not survive face detection, and a real person's photo is not ours to use. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
OCR engines are picked by name (paddle|doctr|surya) rather than registered like a model, so nothing could place one individually and the test for them failed with "not in models.json" before it ever reached a pod. The only way to send OCR to a pod was a capability remote — a production routing switch, and the mechanism that once sent live embedding traffic to a rented GPU. An ocr_models section maps to the ocr capability, so an engine gets the same per-model placement every other model type already has, and the harness creates a temporary entry for one when the catalogue has none. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
--extra-index-url only ADDS an index; pip then picks the highest version across all of them, and PyPI's default torch is built for CUDA 13. So the pin never applied: every capability image shipped a CUDA-13 torch and failed on any machine with an older driver — 'The NVIDIA driver on your system is too old (found version 12080)'. RunPod's fleet is mixed, so the same GPU model passed or failed depending on which machine the pod landed on, which is exactly why this looked like flakiness for so long. torch, torchvision and torchaudio now install from the CUDA index alone, via --index-url, before anything else; installed later from PyPI they would pull a matching CUDA-13 torch back in. cu128 resolves to torch 2.11 — the version the local install already runs — and runs on the driver that failed. Both Dockerfiles print the resolved torch build at each layer, so a silent substitution cannot happen again. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The test run synthesises a spoken sentence to probe an STT model with real audio rather than a reachability check. espeak was never in the image, so the probe fell back to "configured, not verified" — correct behaviour (the gap is ours, not the model's) and therefore invisible: no error, no failing test. Found by inspecting a built image. Added to the full build and to the thin update layer, so it is present whichever path produced the running image. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod could not say WHICH build it was running, which is exactly the question a cached ':latest' raises — an images pod reported a dependency missing that had already been published under that tag. The build now bakes IMAGE_TAG in, so the boot record names the image. /boot also showed only 'lifespan start': the seed registration and the end of startup were plain prints. They are phases now, so one request covers the whole boot including how many seeded models registered and whether that step failed. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
README gets the per-model test run and the pod boot record; docs/runpod.md and docs/remote-execution.md get the findings themselves, because each one cost a live failure to learn: - RunPod publishes no pod-log API. Checked against their OpenAPI spec: 23 routes, none for logs, and GraphQL refuses introspection. Code that "tried several routes" was guessing at URLs that never existed. - The port appears only after the image is pulled, so the boot budget covers the download: 133s warm vs repeated failures at 300s cold, each one renting a fresh machine to pull the same image again. - A machine may serve a cached ':latest', so a test against a floating tag cannot say which build it ran — pin a version when the answer has to mean something. - Setting remotes.endpoints[capability] to test ONE model moves every request of that kind; that is how production embeddings ended up on a rented pod. - Writing a config does not change routing until an idle engine takes the push, so a test must confirm the serving process has it before testing. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The port a pod exposes appears only AFTER its image is pulled, so the 300s budget had to cover the download — it was written assuming the container was already there. The same 7.3 GB image took 133s on a machine with the layers cached and failed repeatedly at 300s on cold ones, and each failure rented a fresh machine and paid for the whole pull again, three times. 900s, overridable per model; being wrong now costs six cents instead of three pods. RunPod publishes no pod-log API: the REST OpenAPI spec lists 23 routes and none serve logs, and GraphQL refuses introspection. Every attempt returning HTTP 400 was code guessing at URLs that never existed. So the pod reports on itself — boot.sh is the entrypoint and logs timestamped phases from the first instant the container exists (including the seed list, which is where "not available. Use one of: " would have been caught in one line), and /boot serves that record plus the app's own phases. A failed boot now asks the pod before falling back to a console link a human has to open. Nothing covers the pull window — no code of ours is running yet — so wait_ready reports status and uptime every 60s instead of going silent for minutes. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Testing an image model by setting remotes.endpoints["images"] moves every request of that kind. An embeddings remote left on from an earlier test did exactly that on a live box: production embedding traffic went to a rented pod while the local GPU sat idle, and the operator noticed because the card was quiet. tools/runpod_model_test.py writes a runpod block on the single model entry instead — per-model placement already beats the capability map, which is what lets two models of the same kind sit in different places. Writing the config is also not enough. The front pushes a reload and the engine takes it only when idle, so a harness that writes and immediately tests is testing the PREVIOUS configuration. That produced a whole pass of false "nothing configures this to run remotely" failures, and could just as easily have reported a remote pass for a request served locally. GET /v1/models/test/state answers for the process that serves the request; the harness polls it until the placement it wrote is visible and refuses to test if it never is. The new endpoint is in _SKIP_PATHS: forwarded, it would report a pod's view of a config the pod does not have. Co-Authored-By:
-
- 14 Sep, 2026 28 commits
-
-
Stefy Lanza (nextime / spora ) authored
The first RunPod pass over every capability produced four failures, three of which were ours: The TTS probe sent voice "default", which no shipped model has — Kokoro loaded fine on a rented pod and then answered "Voice default not found in available voices". TTSRequest already defaults to a real voice, so the probe stops inventing one. OCR engines are chosen by name (paddle|doctr|surya) and are not models.json entries, so the test refused with "'surya' is not in the model list". _describe now recognises them. For a text model the refusal offered "a remote for its text capability" as a fix. It is not one: /v1/chat/completions is outside the gateway, so that config routes nothing and the test would then report a remote pass for a request served locally — the exact failure the forced-remote check exists to prevent. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A rented images pod downloaded SDXL for ten minutes and then failed with "No package metadata was found for bitsandbytes". models.json carries quantization settings for EVERY model type, not just LLMs, but bitsandbytes was only in the text profile. It moves to core, the shared layer under all ten images. stt and ocr were reachability-only, which reported ok:true while proving nothing beyond "a URL is configured" — a pass that would survive a completely broken pod. Both inputs can be synthesised honestly: espeak speaks a known sentence, PIL draws known words. The result is checked against what went in, so a model that returns fluent nonsense fails too. When the synthesis itself cannot run (no espeak, no fonts) the probe falls back to the reachability check rather than blaming the model for a gap on this side. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two bugs, both found by the first end-to-end embeddings test on a rented pod. A capability image's entrypoint is `uvicorn codai.api.app:app` — it never runs codai/main.py, so set_config_manager() was never called and codai.admin.routes.config_manager stayed None. Both the CODERAI_SEED_MODELS list and the runtime catalogue write read through that global, so a pod seeded with one model answered "not available. Use one of: " with nothing after the colon. The lifespan now builds its own ConfigManager when none was set — loading it is what parses the seed list — and says how many seeds it registered. The bearer token a pool launches its pods with is generated in memory when none is configured. A pod that outlived the process that rented it was then adopted with a DIFFERENT token and answered 401 for the rest of its paid life (the test run reported serves: "(HTTP 401)"). The token is now recorded in the shared pod registry and adopted along with the pod. Verified live: bge-m3 on a rented RTX 2000 Ada returned a real 1024-dimension embedding, 67.9s cold. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Registration reaches the model manager, seeded models survive the engine assignment filter, siblings wait for a booting pod, and pod boot phases plus catalogues are reported for diagnosis. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The last three pod failures were each diagnosed the same way: find the pod URL in the engine log, ask the pod by hand what it serves, work backwards. That is a diagnosis loop measured in minutes of log archaeology per attempt. Boot is now reported in PHASES, because "it took 150s" is not a diagnosis: port open after Ns (image pull + container start), serving after N more. The two have different fixes — a bigger boot_timeout_s versus a bigger load_timeout_s. The moment a pod is up, its /v1/models is logged. An empty catalogue now says so outright — "pod serves NO models — it will refuse every request" — instead of surfacing minutes later as "Model 'x' is not available. Use one of: " with nothing after the colon. And a test run's answer carries the pod diagnostics: which pods are live, their GPU, uptime, spend so far, URL, and the models each one lists. The fact that mattered in every failure so far was the catalogue, and it is now in the response rather than three commands away. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod finally answered, and answered "Model 'bge-m3' is not available. Use one of: " — with NOTHING after the colon, despite being seeded with five models and successfully taught a sixth. Two causes, both only visible on a real pod. REGISTERING A MODEL WAS HALF DONE. Both the boot seed and /v1/models/register wrote models_data, but request validation asks the MODEL MANAGER for its allowed identifiers. A model in the catalogue that the manager never heard of is refused. Registration now calls the right setter per section, maps the alias a request actually uses (bge-m3 -> BAAI/bge-m3), and widens the engine's assigned-model set — a model registered at runtime was assigned by nobody and would otherwise be usable but invisible. AND THE SEED WAS FILTERED OUT BEFORE IT GOT THERE. A pod runs front plus engine; the front computes each engine's assignment from models.json, and the seed is deliberately in-memory only, so the engine was told to register nothing — dropping exactly the models the pod was rented for. Seeded entries now survive that filter. Separately: both engines rented their own pod again, 45 seconds apart, because a pod takes ~2.5 minutes to boot and the sharing check only saw READY pods. A pod is now registered at creation, and a sibling that finds one still booting WAITS for it instead of renting a second card. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Cross-engine pod sharing, network volumes for weights/uploads/staging, and the optional venv-on-volume slim pod image. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A capability image is ~7 GB and nearly all of it is torch, pulled again by every cold pod. With a network volume those libraries can live there instead: `venv_on_volume` boots a slim image — OS, Python and coderai only — and runs coderai from a venv on the volume. The first pod builds it; every pod after skips both the pull and the install. Opt-in per pool and per model, never a default, because the trade is real and unmeasured: the pull gets much smaller, but importing torch off network storage is slower than off local disk, so some of the saving comes back at import time. Whether that is a net win depends on how often pods cold-start, which is the user's workload, not mine to assume. Two hazards are handled rather than left to chance. The ready marker is written AFTER the install, so a pod that dies mid-build leaves nothing a later pod would import from. And a lock directory — mkdir is atomic — makes a second pod wait for the first instead of two installs interleaving into the same directory. `venv_on_volume` without a volume is refused with the reason, rather than booting a pod that installs 7 GB onto container disk and discards it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod's disk dies with the pod, so every cold pod re-downloads its weights and every upload has to be repeated. A network volume — created by hand in the console, coderai never creates one — makes that a one-off. Attaching it is the easy half; the useful half is pointing HF_HOME, the diffusers/transformers caches, CODERAI_MODELS_DIR and the staging directory at the mount. A volume attached without redirecting those changes nothing, which is the mistake this avoids. That also answers the bounce-host question: an upload lands on the volume and every later pod sees it, so a QLoRA or a local model can be sent once and used from any pod — no third-party storage, no public IP. Two RunPod constraints are enforced rather than discovered the hard way: volumes are Secure Cloud only (cloud_types is forced), and a pod must run in the volume's own data center (looked up once and pinned, instead of letting price ranking pick a region where the attach fails as a confusing capacity error). Settable account-wide in Settings or per pool. Container images still come from a registry — a volume holds weights, not the image. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ten images built, none pushed: the guard checks free space against every profile before doing anything, but a push uploads what already exists and needs no disk. It now measures only what actually has to be BUILT — and says so when there is nothing to build — so a full disk can no longer turn 'publish' into a no-op. It also asks docker where its root dir is rather than assuming /var/lib/docker, which on this machine is /storage/docker. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The second live test showed nvidia and radeon each renting their OWN pod for 'capability:embeddings' — two GPUs, two bills, while max_pods said 1. Same root cause as the reaper bug: one engine per GPU, each with its own pools, none able to see the others. Before renting, a pool now looks in the shared registry for a healthy pod another process already has for the SAME pool key, and borrows it. The registry entry carries the URL, recorded once the pod is serving. A borrowed pod is released, never terminated: its owner holds the lifecycle and the bill, and killing it would take it away from the engine still using it. It is also billed at 0 here, so the cost is not double-counted against the caps. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Capability pods are told what they can serve, and any pod can be taught a model at runtime (HF id, URL or upload) instead of being a catalogue frozen at boot. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod knew only what it was seeded with at boot, so any request naming another model got "not available" for the rest of that pod's life. That makes a pod a fixed catalogue rather than an extension of the local system, which is the whole point of it. When a remote refuses a model it does not know, the gateway now teaches it and retries once: resolve the model to something the remote can fetch (HuggingFace id, URL, or an upload when it exists nowhere else), POST it to the new /v1/models/register, and replay the request. Registration is in memory — a pod is disposable — and idempotent, so a model already loaded is never disturbed. Taught once per pod, tracked by (base, model). The retry is bounded to one attempt and only fires on a 400/404 whose body actually says the model is unknown: a 500 or an out-of-memory must never turn into a retry loop against a rented GPU. A model that resolves to no fetchable source is refused with that reason rather than registered as something the pod could never load. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The first pod that survived long enough to answer returned: "Model 'bge-m3' is not available. Use one of: " — nothing after the colon. A pod starts with an EMPTY catalogue. A per-model pod is seeded with its one model, but a capability pod has no model entry at all, so it refused everything. Capability pods are now seeded with this deployment's models OF THAT CAPABILITY — only ones the pod can actually fetch (a HuggingFace repo id or a URL), since registering a local path would turn a clear "not available" into a confusing load failure. The seeding was also unreachable for any pod with an explicit image: the coderai branch returned early only when the image was UNSET, and capability pods always have one now (the published default). Both cases go through one path that carries the auth token and the catalogue together. What the live run did prove, before failing on the catalogue: GPU selection with price ranking and capacity fallback, the boot health gate, pod bearer auth (our token accepted; an unauthenticated call gets 401), gateway forwarding to the pod URL, idle reaping after 120s, and the spend ledger ($0.033 for the run). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Stop sibling engines reaping each other's RunPod pods: a shared pod registry plus a reap grace period. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The first real pod test found this in seconds: coderai runs one engine per GPU, each with its own pool registry AND its own stale-pod reaper, so each engine sees its siblings' pods as untracked orphans. The nvidia engine created a pod; the radeon engine terminated it four seconds later; radeon created one, nvidia killed that — a mutual kill loop that rented and destroyed pods until I disabled RunPod. Every boot then "failed" after its 300s port timeout, because the pod had been deleted underneath it, and the pool dutifully retried on the next machine. Two fixes, because one is not enough: A shared registry (config_dir/runpod-pods.json, flock'd) records every pod this DEPLOYMENT owns. _known_pod_ids now reads it, so an engine with no pools of its own still knows its siblings' pods are not orphans. A grace period (REAP_GRACE_SECONDS, 20 min) refuses to reap any pod younger than a boot budget, whatever the registry says — the backstop for the race a registry cannot close, such as a process that died between creating a pod and recording it. No pods were left billing: the mutual reaping cleaned up after itself and the account shows zero pods. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Test-run fixes (forced-remote verified before dispatch, empty output is a failure) and config propagation to the engine and front. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Configuring a remote capability in the UI did nothing until a restart, which makes the feature unusable as shipped. The admin API is served by the SYSTEM worker, so a save updates that process's config object. The remote gateway, the engine blocks and the front's burst-to-RunPod path all read a DIFFERENT process's copy. The engine's /internal/reload-config — which the front pushes on a config change — re-read models.json only, and the front's own refresh copied a fixed field list that included neither `runpod` nor `remotes`. Both now carry them: the engine re-reads config.json into its live config on reload, and the front refreshes runpod/remotes along with the rest. Found by trying to use the feature on the running instance rather than by reading the code, which is the only way this class of bug shows up. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Running the new test against the live instance found both immediately, which is the point of it. A FORCED REMOTE TEST COULD PASS LOCALLY. /v1/chat/completions is deliberately outside the gateway (RemoteOpenAIBackend owns it), so the x-coderai-placement header is ignored on that path: a text model with no remote configuration was served HERE while the result reported where=remote, ok=true. A test that passes in the wrong place is worse than no test. Forcing remote is now verified before anything is dispatched, and refuses with what is actually missing. A 200 WITH NO OUTPUT WAS A PASS. The text probe allowed 8 tokens, which a reasoning model spends thinking — Qwen3.5-9B answered with completion_tokens=0, finish_reason=stop, and the test called it ok. The budget is now 64, and an empty result is a failure carrying the numbers that explain it, for text, embeddings, images and rerank alike. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Test run: prove a model works where it is configured to run, with a per-request placement override. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Placement now has many moving parts (image, weights, adapters, auth, budgets, staging) and most of them fail minutes into a pod boot, in a log nobody is watching. POST /v1/models/test runs the smallest real request for the model's kind and reports what happened, with a Test run button on every model page. It dispatches THROUGH THE FRONT, so it exercises the same routing, auth, pod provisioning and staging a real request would — a test that simulated the path would prove nothing about the path. `where` forces placement for that one request via a new x-coderai-placement header the gateway honours: "local" ignores any remote configuration, "remote" refuses to fall back to local. Without that you cannot verify a pod at all: the request would be served wherever the config decided, and a green result would mean nothing. Kinds with no cheap synthetic input — video (minutes of GPU and real money), faceswap, STT, OCR — report a reachability check and say plainly that no generation was run, instead of implying more than was proven. Also fixed test pollution I had introduced: the gateway fixture replaced module-level functions without restoring them, and one test assigned _model_entry_for unscoped. Both leaked into later tests, which passed alone and failed in the suite — the failure mode that teaches you to distrust the suite. Verified stable across repeated runs. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model placement for any model kind, text LoRA/QLoRA locally and on RunPod, weight sources (HF id, URL, upload, pod-side staging), shared pools and optional warm pods. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Your idea, and it removes the last dead end. vLLM and llama.cpp pods could not be given a local model or adapter: no endpoint to upload to, and their server needs the file at launch. But a pod CAN fetch it before the server runs — put the weights on any HTTP host it can reach and stage them at boot. The obstacle was that those images run their server as the ENTRYPOINT, so everything we passed became an argument to the server. RunPod's REST pod API accepts dockerEntrypoint/dockerStartCmd (GraphQL does not), so runpod_client now creates a pod through REST when an override is needed and stays on GraphQL otherwise. The staged start command fetches with curl, wget or python — whichever the image actually has rather than whichever we assumed — extracts a tar for a multi-file model, and then execs the server pointed at what it staged. One definition of the staged path (staged_model_dest) is used by both the download and the server args: computing it twice is exactly how a pod downloads to one path and serves another, which the first cut did, along with emitting two --model flags. Adapters given as a URL ride the same path: --lora-modules for vLLM, --lora/--lora-scaled for llama.cpp. So a local adapter reaches a vLLM pod after all — publish it to a URL instead of uploading it. `source: upload` against a vLLM pod now says what to do instead, rather than only that it cannot. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A text LoRA that exists only on this machine had nowhere to go: vLLM and llama.cpp resolve adapters at launch and have no endpoint to upload one to, so the previous commit could only report and skip it. Such a model is now routed to a coderai TEXT pod — the one pod image that can be sent an adapter — and the transfer is arranged around the fact that a pod is created before anything can be sent to it: 1. the pod is told the adapter's sha256 id, computed HERE, so both sides agree on the name before the pod exists; 2. the bytes go to the pod's content-addressed blob store before the first request, once per pod (a repeat costs one existence check); 3. the pod resolves sha256:<hex> back to a file — text_loras.local_path now understands the content-addressed forms — and applies it. A published adapter still goes to vLLM, which fetches it itself: cheaper and faster, so publishing an adapter you use often remains the better path, and the log says so. A PEFT adapter DIRECTORY is still refused with a reason — the blob store holds single files. packaging/runpod/profiles/text.txt adds the text profile (transformers, peft, bitsandbytes for QLoRA bases). It has to be built and pushed before use; the other nine are published already. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
lora_path and lora_model_dir have been in the model config for a long time, but only the VRAM estimator ever read them: vulkan.py, cuda.py and vllm.py contained zero LoRA references, so configuring a LoRA on a text model silently did nothing. Now all three apply them. codai/models/text_loras.py answers the one question they share — which adapters does this model want, and where are they — across every config shape the field has grown (lora_path, a list, lora_model_dir, a `loras` list of dicts), de-duplicated so the same adapter is not applied twice. Each runtime then does its own thing: transformers PEFT load_adapter + set_adapters, several at once llama.cpp lora_path + lora_scale, ONE adapter and it must be GGUF-converted — a PEFT safetensors is reported and skipped rather than failing deep inside the load vLLM --enable-lora --lora-modules, with --max-lora-rank 64 because the default 16 refuses most trained adapters and blames the flag rather than the adapter QLoRA needs nothing extra: the quantisation is how the BASE model loads (load_in_4bit), the adapter is an ordinary LoRA. The manager now passes the model's entry to the backend as `model_config`, which is how any of this reaches them. On RunPod the asymmetry is the interesting part. A diffusion LoRA is sent with the request, because that API carries adapters per request. A text LoRA is part of how the model LOADS, so it must resolve when the pod starts: only HuggingFace repo ids travel to a vLLM/llama.cpp pod, and a local path is reported and skipped instead of being baked into a launch command that would fail minutes later inside a pod nobody can inspect. A coderai pod gets the adapters in its seed and applies them itself. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pod cannot see this machine's disk, so "use my local models on RunPod" needed a way for the pod to GET them. Three, now, picked by `source`: auto — an explicit hf_repo, a path that already is a repo id, or a repo id recovered from the HuggingFace cache path (models--Owner--Repo). Most of a normal catalogue resolves with nothing typed: weights downloaded through coderai live in that cache, and the directory name still says where from. hf — download hf_repo from HuggingFace. url — download model_url directly, which is how most one-off GGUFs are had. upload — send the weights from here. Each server gets only what it can use, refused at configuration time rather than after five minutes of pod boot: vLLM is launched `--model` and takes a repo id only; llama.cpp takes `-hf` or downloads a URL with `-mu`; upload works only on a coderai pod, because vLLM and llama.cpp need the file before their container starts while a coderai pod loads models on demand. Upload is a real option, not a token one — on symmetric gigabit a 30 GB model is a few minutes — but it is the cold cost of every NEW pod, so it is never chosen automatically. /v1/models/upload streams to disk in chunks (never into memory), extracts a tar for multi-file models with every member checked against the destination, and registers the entry in memory only. An existence check by name and size makes a repeat free and refuses a half-copy. keep_warm keeps one pod running so nothing waits for a cold boot. OFF by default and deliberately: a warm pod bills every hour of every day, including the ones nobody uses it. Warm pods start at startup, since a pool created on first use would still pay the boot it exists to avoid. All of it is in the UI: weights source with the right field shown per choice, keep-warm, and an "always run this model locally" pin. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
PER-MODEL PLACEMENT FOR ANY KIND. Sending a model to RunPod used to work only for text: resolve_pod_engine looked at whether the path ended in .gguf, and RunpodBackend implements chat only. A diffusion model pinned to runpod would have been handed to vLLM, which cannot serve it. Now the model's OWN KIND picks the pod: an image/video/TTS/STT/embedding model gets that capability's published coderai image, a GGUF gets llama.cpp, an HF repo gets vLLM. The gateway routes by model BEFORE capability, so two models of the same kind can differ — one video model pinned local with `placement: "local"` while another runs on its own pod with its own budget, which capability-level routing could never express. A pod rented for one model starts with an EMPTY catalogue and would refuse every request for it, so pod_plan seeds CODERAI_SEED_MODELS with that model's entry and the pod registers it at startup (in memory only — a pod is disposable, and writing it would surprise anyone setting the variable on a real install). Only fields that describe the model travel: never local paths, which mean nothing there, and never this deployment's placement settings, which would have the pod renting pods of its own. ORCHESTRATION STAYS LOCAL. /v1/pipelines, /v1/characters and /v1/environments are no longer gatewayable capabilities. They are a sequence of calls to other endpoints, and they already dispatch each step through the front as its own /v1 request — so every step is placed by its own model's config, and a chain can mix local and remote freely. Forwarding the chain would have moved the logic to a pod and placed every step by that pod's catalogue instead. LORAS FOLLOW THE REQUEST. A LoRA is a file on this machine; a pod has never seen it. The blob store is content-addressed, so each adapter is hashed, checked against the remote's /v1/loras/blob/<hash>, uploaded only when missing, and the reference rewritten to that id — repeat requests upload nothing. QLoRA needs nothing extra: the quantisation is in how the base model loads, the adapter is the same safetensors file. An HF id or URL is left alone for the remote to fetch. Also: a vLLM pod now refuses a local model path up front, with the same reasoning as the GGUF guard — otherwise it boots for minutes and dies — and passes `quantization` through, since a local load_in_4bit does not carry over to vLLM. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Now that ghcr.io/nextime/coderai-<capability> is public, a capability pod needs no image name: leaving it blank uses the published image for that capability. Enabling a capability on RunPod is one dropdown. Capabilities that ride another profile's dependency set are mapped — rerank to embeddings, speaker to stt, stems/audio_clean/audio_gen to audio. The four with no published image (loras, characters, environments, pipelines) return no default and say so, rather than defaulting to a tag that 404s minutes into a pod boot. CODERAI_CAPABILITY_IMAGE_REPO / _TAG point the whole set at a fork or a private registry; a row's own `image` still wins. The UI shows the default as each row's placeholder, so it is visible rather than implied. Co-Authored-By:
-