- 14 Sep, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authoredSeveral models can share one RunPod pod until it saturates. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
Stefy Lanza (nextime / spora ) authored
Two models that both offload rented a card each, even when neither came close to filling one. They can now name a shared `pool` — on the model's runpod block or on a burst target — and land on the same pod; each request carries its own model name, and the pool rents a second pod only once the first is carrying scale_up_inflight_per_pod requests. Sharing is refused, loudly, for pod servers pinned to one model at launch: a vLLM pod is `--model X` and a llama.cpp pod `-hf one.gguf`. Allowing it would send requests to a pod that answers with the wrong model's weights, or 404s — a clear error at configuration time beats debugging that. `engine: coderai` (a whole coderai, which picks the model per request) is what makes a shared pool possible. The first model to create a shared pool sets its shape; later joiners do not overwrite it, so whichever model happens to load last cannot silently redefine the budget everyone is sharing. Capability pools now use the same rule instead of their own ad-hoc guard. Settable in the UI on both the model's pod section and its burst target. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model placement (local / remote / burst), pod auth and real concurrency scaling, and a UI for all of it. Co-Authored-By:
-
- 13 Sep, 2026 27 commits
-
-
Stefy Lanza (nextime / spora ) authored
None of this session's work had any UI, so all of it was JSON-only. Now: MODEL PAGE — a Placement section: serve this model from a remote URL (with its name and token on that remote), and burst-to-RunPod with the three triggers (no free slot / queue full / no local GPU) and a target that is either a managed pod or your own serverless endpoint. The pod form gained engine (auto/vLLM/llama.cpp/coderai/custom), the GGUF to download, image, health path, docker args, the private-registry credential, the pod's bearer token with an explicit "leave it open" opt-out, and the scaling knobs (scale-up threshold, max in-flight per pod, sticky sessions). SETTINGS PAGE — a remote service URL for each of the five engines (ds4, colibri, k3, ktransformers, vLLM), the account-wide registry credential, and a Remote capabilities card: one row per capability, each Local / Remote URL / RunPod pod, with image, shared pool name, VRAM, $/hr, max pods and idle timeout. Clearing a remote field now REMOVES it. The save path only writes keys that are present, so sending an empty value as "unchanged" would have left a stale remote in place and the model would have kept going to a machine the user thought they had detached. Tests assert what is easy to get silently wrong in UI work: every getElementById reads an id that exists (counting ids the page assigns to elements it creates), every new field is both populated and collected, every capability the gateway can route has a row, and the admin API keeps every key the form sends. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three things, one of which is a security fix. POD AUTH. backends/runpod.py said "the pod's vLLM server needs no auth" and sent no credential. A RunPod proxy URL is reachable by anyone who learns it, so every pod we have ever started was an open GPU on the public internet. Pods are now launched locked to a bearer token, and blank does NOT mean open: the pool generates one per pool unless `allow_open_pod` is set explicitly. Each server takes it its own way — `--api-key` for vLLM and llama.cpp, CODERAI_API_TOKEN for a coderai pod, which required teaching BearerAuthMiddleware to accept an env token since a pod has no auth.json to seed. The health probe sends it too: a locked pod answers 401 on /v1/models, which would otherwise look exactly like "never became ready". All three callers (always-remote backend, capability gateway, front spill) now carry it. SCALING. scale_up_inflight_per_pod was parsed and stored but never used: the pool grew only when it had NO healthy pod, so with max_pods=3 fifty concurrent requests all piled onto pod #1. It now grows when the least-loaded pod reaches the threshold, in the BACKGROUND — the request that proved the pool needs to grow is served by an existing pod rather than waiting minutes for a cold one. `max_inflight_per_pod` (0 = let the remote's batching absorb it) makes requests wait for a slot instead of piling on, for engines that degrade under concurrency rather than queueing internally. CACHE. `sticky_sessions` sends a conversation back to the pod that already served it, so the remote's prefix/KV cache still holds its context; round-robin would re-prefill every turn from scratch. Bounded to the newest 4096 conversations. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Spillover already existed, but not in the shape that is actually useful: it fired only when the WAIT QUEUE overflowed (on_concurrency_full) or when no GPU could serve the model, and it could only reach a serverless endpoint — the code said outright that a pods target "goes through the engine and is not handled here", which meant bursting to our own pods was not implemented at all. Two additions: `on_busy` — offload the moment there is no free local slot, instead of queueing behind the running generation. Built on FrontQueue.try_acquire, a non-blocking take: the front has to learn the model is occupied WITHOUT committing to wait for it. Wired into both the buffered and the SSE dispatch paths. `target.mode: "pods"` — borrow a pod from the managed pool (provision on demand, health-gate, scale, reap, bill against the same caps) for one request and hand it straight back, so an overlap that never recurs is reaped rather than left running. Serverless and pods now share one `_spill_open` primitive, with a Response wrapper for the buffered path and a chunk generator for the SSE path. Local stays the safety net in both directions: a burst that cannot be opened falls back to the local queue, never the reverse. Two bugs caught before shipping, both in my own first cut: the direct path referenced _swap_tok before it is assigned (NameError on every burst), and the SSE path left _qkey set after bursting, so cleanup would have released a slot it never held — handing a phantom slot to the next waiter and over-subscribing the model. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
voice needed build-essential: f5-tts/seed-vc pull webrtcvad, a C extension with no cp312 wheel. Same purged-in-one-layer mechanism audio uses for its Rust toolchain. All nine built and healthy (each answers /healthz): ocr 6.93 embeddings 7.26 audio 7.31 faceswap 7.57 tts 7.61 stt 7.63 images 7.74 video 7.83 voice 8.33 (GB) They share the core layer, so the disk and push cost is one core plus nine small deltas, not nine whole images. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
publish_capability_images.sh does the whole thing: disk check, registry login (with the exact ghcr token instructions), build, smoke test, push, and the per-package URLs for flipping GHCR's default-private packages to public. It is resumable — an image already built for this version is skipped, so re-running after a failure doesn't rebuild 7 GB of torch — and it pushes nothing until every image has answered /healthz. Profiles can now declare apt packages their wheels need to COMPILE, in `profiles/<name>.build-deps`, installed and purged inside a single layer. `audio` needed it: deepfilternet pulls deepfilterlib, which has no cp312 wheel and compiles Rust extensions. It now builds at 7309 MB against 7260 for a profile that needs nothing, so the toolchain is provably not in the image. Built and healthy so far: images (7.74 GB), faceswap (7.57 GB), embeddings (7.26 GB), audio (7.31 GB). ocr, stt, tts, video and voice are untried — the script resumes into them. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three fixes the first real build exposed. The import check caught what a lazy-import survey had missed: app.py mounts every router, and a few touch libraries at module level rather than lazily — soundfile (voice_convert), cv2/PIL (faceswap), litellm (parser), websockets (broker client). A static walk of the import graph from codai.api.app found the full set; they are now in core.txt with a note saying why. torch and transformers are NOT among them, so they stay per-profile. torch moved into core so it becomes ONE shared layer. Nine images each building their own copy would duplicate ~5 GB nine times — on disk and on every push. Core is now its own RUN, identical across profiles, so all nine share it. A BuildKit pip cache mount means building all nine downloads torch once rather than nine times, which matters when the link is also pulling models. build_all_capability_images.sh drives the set. Sequential on purpose: they share the cache and the core layer, so parallel builds would race on both. Each image must answer /healthz before the next starts. Validated: coderai-embeddings built, imported and answered /healthz (7259 MB) before this restructuring. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The export/import flatten has been a manual ritual. `docker export | docker import` drops the image config entirely, so it has to be rebuilt by hand every time — and a silently lost ENTRYPOINT or PATH produces an image that builds, ships, and then fails to start. This scripts it, emits every value in JSON form (so CODERAI_EXTRA_ARGS' spaces survive), and then VERIFIES the flattened config matches the source before it moves the base tag. It refuses to retag on a mismatch. It also strips secret-valued env vars. HF_TOKEN was baked into the image config — readable by anyone who can pull it, `docker image inspect` is enough — and had been carried forward since at least the 20260909 base. It arrives via `docker commit` of a container started with -e; the repo itself is clean (not in any tracked file, not in history). run_oci.sh already passes the token at runtime, so removing it from the image changes nothing. Names are reported, values never echoed, and the equality check knows the strips are intentional. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Remote execution across the board: every model and capability can be served from another machine, and RunPod pods can be rented per model, per capability or shared between them. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two things that both come down to not renting more GPU than the work needs. Shared pools: a pod is a whole GPU, and one coderai can serve images, video and TTS from it. Capabilities naming the same `pool` now share one set of pods and one budget; without it, three capabilities rented three cards to do what one can. The pool is configured once on whichever capability carries the settings — a bare `{"pool": "media"}` reference will not reset it to defaults. Trimmed images: packaging/runpod/ builds a coderai carrying ONE capability's dependencies, since the flattened base is ~26.6 GB and every cold pod pays that pull. Profiles under packaging/runpod/profiles/ — core plus images/video/tts/stt/voice/audio/embeddings/ocr/faceswap — work because the routers import lazily, so a pod only needs what its own endpoints touch. The Dockerfile fails the BUILD if a profile is missing something the app imports, and build_capability_image.sh then boots the container and waits for /healthz. Both checks exist because the alternative is finding out as a boot timeout on a rented GPU, with nothing in the logs saying why. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Audited all 66 /v1 endpoints against the capability map. One was unmapped: /v1/audio/progress, so polling an audio job would be answered locally while the job ran on the remote — always empty. Mapped to audio_gen, and a test now asserts the whole set, so a newly added endpoint cannot quietly stay local. The bigger hole was results. With the default response_format "url" a render answers with a URL built from the REMOTE's base (http://pod-xyz:8000/v1/files/out.png): unreachable from the client, and a 404 if fetched here because the file is on the pod. The gateway now rewrites those URLs to this instance and remembers which remote produced each file, so GET /v1/files/<name> follows back to it. Only JSON bodies are buffered for the rewrite; everything else still streams untouched. The map is a bounded routing hint (newest 4096), not a store: a pod reaped for idleness takes its outputs with it, so fetch what you need or ask for b64_json. Documented. Admin, /internal, /chat, /login and /healthz stay local by design — they administer or identify THIS instance. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Capability pods run YOUR image, and nobody publishes their whole stack publicly — but create_pod never sent a credential, so a private image would fail to pull and the pod would just never become healthy, with nothing in the logs pointing at why. RunPod stores registry credentials for you (Settings -> Container Registry Credentials) and gives each set an id; that id now rides along as `containerRegistryAuthId`, from `runpod.registry_auth_id` account-wide or a pod block's own `registry_auth_id`. Public images (vLLM, llama.cpp) are unaffected — they send nothing. docs/remote-execution.md now says where the images come from: LLM pods need no publishing at all, a capability pod needs a coderai image in a registry RunPod can reach, and the 26.6 GB flattened image is paid on every cold pull. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod pool never cared that it was serving one LLM — it provisions, health-checks, scales and reaps whatever image it is given. Two things were hardcoded: the image and the /v1/models readiness probe. Both are now per-pool, via pod_plan(): auto vLLM, or llama.cpp for a GGUF -> /v1/models coderai your image (required) -> /healthz custom your image + docker_args -> health_path So a capability can now be a pool instead of a fixed URL. Configure `remotes.pods: {"images": {...}}` (the same fields a model's runpod block takes) and the gateway borrows a pod per request and hands it back; `"endpoints": {"images": "runpod"}` is the shorthand. Everything that already guards LLM pods now guards these: $/hr ceilings, rolling spend limits, the global budget, min/max autoscaling, idle teardown, the stale -pod reaper, and the stats page (as `capability:<name>`). That closes the gap this work left open: an idle image or video pod is exactly the kind you forget about, and until now nothing would reap it. A pod that cannot be provisioned — RunPod disabled, budget hit, no GPU under the ceiling — returns 502 rather than falling back to running the model locally. Silent local fallback is a failure mode you only notice by watching your own VRAM vanish. 22 tests, including pod acquire/release (the handle is returned even on the happy path, so the reaper can see it), the runpod shorthand, the provisioning-failure 502, and every pod_plan branch. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
18 tests over local stub servers — no GPU, no weights, no network: worker service_url health enforcement, the mux service + RemoteMuxEngine (streaming, stats, mid-stream cancel, error propagation, dead service), remote GGUF text models (routing, /v1 normalisation, auth, chat, SSE, no-local-weights), RunPod engine selection (HF->vLLM, GGUF->llama.cpp, pins, the missing-hf_gguf error), and the gateway (transparency when nothing is remote, capability and per-model routing, verbatim multipart forwarding, chat exclusion, clean 502). Also stop the mux service dumping a broken-pipe traceback when a client cancels a stream — hanging up IS the cancel signal there. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Images, video, embeddings, rerank, OCR, TTS, STT, voice cloning, audio generation, stems, 3D and pipelines all load their model in-process, so none of them had a service to point elsewhere. Writing thirteen services would be the long way round; another coderai already exposes all of them, so it can be the service. RemoteGatewayMiddleware sits in front of the routes and, when a request is addressed to something configured as remote, replays it verbatim to that endpoint and streams the answer back — JSON and multipart alike, since the body is forwarded as raw bytes. Routing is per model (`service_url` on the models.json entry) first, then per capability (`remotes.endpoints` in config.json, or CODERAI_REMOTE_<CAP>_URL). Placement matters and is deliberate: it is registered first, so it ends up innermost — auth, rate limiting and request logging all run before it, and a request still has to be authorised here before being forwarded. Chat and completions are excluded: those go through RemoteOpenAIBackend, which keeps usage accounting, tool parsing and the manager's bookkeeping. When nothing is configured remote it returns on its first branch and never buffers a body (cached 5s so it doesn't walk the model list per request). Bodies over `max_body_mb` are served locally rather than held in RAM. A non-forwarded request replays its buffered body downstream untouched — tested, including multipart uploads that still parse locally. docs/remote-execution.md documents all four remotization mechanisms together. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two additions, one goal: no text model is stuck on this box. 1. codai/backends/remote_openai.py — set `service_url` on a model entry and coderai proxies that model to any OpenAI-compatible endpoint (another coderai, llama-server, vLLM, SGLang, a rented pod) instead of loading it. `served_model` renames it for the remote, `api_key` adds a bearer token. It reuses RunPod's request/SSE machinery and only replaces endpoint resolution, so streaming and usage accounting are the same code path. This is what makes the GGUF/llama.cpp catalogue remotizable: those models load in-process through llama-cpp-python, so there is no service to redirect — pointing the *model* at a URL is the only redirect available. The manager checks it first, ahead of every engine route, and `_model_is_runpod` grew into `_model_is_remote` so such models are never downloaded, cached, or charged local VRAM (old name kept as an alias). 2. RunPod pods pick their server per model. A GGUF cannot simply go to vLLM: its GGUF loader is experimental — single file only, a limited architecture list, and it still wants the original repo's tokenizer. So `engine: auto` sends an HF repo to vLLM and a GGUF to llama.cpp's own server image (ghcr.io/ggml-org/llama.cpp:server-cuda), which downloads the weights itself from `hf_gguf` ("user/repo:Q4_K_M") — a local /AI/….gguf path means nothing on a rented machine, and we don't upload multi-GB weights. `engine: vllm|llamacpp` overrides. Both servers answer /v1/models, so the readiness probe is unchanged. Tested against fake servers: routing, /v1 normalisation, auth header, chat, SSE streaming, usage, and each engine-selection branch. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
These two were the exception in the last commit: they are driven over a stdin/stdout mux protocol, so there was no URL to redirect. This adds the missing HTTP boundary. tools/mux_service.py hosts a real MuxEngine (colibri or k3 — both use the same class) and exposes /health, /run (NDJSON: text chunks then stats), /pause and /resume. codai/api/mux_remote.RemoteMuxEngine is the client and presents the exact surface the backends use — run/is_alive/pause/ resume/close/log_tail/family — so ensure_engine() can hand one back and nothing downstream can tell local from remote. Cancellation travels as a dropped connection: the service's write fails and it stops generating, matching the local `cancelled` callback semantics. `service_url` is now a real config field on all five engine backends (ds4, colibri, k3, ktransformers, vllm), settable through the admin API and persisted, rather than only an env var. Tested end-to-end against a stub engine: streaming, stats, mid-stream cancellation, pause/resume, and error propagation all round-trip. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Each `ensure_service()` already returns a base URL, so a worker can be pointed at a service running elsewhere — another host, a container, a rented RunPod pod — without anything downstream noticing. Only the TTS backends had that escape hatch (parler/melo, checked at the caller); now the check lives in the workers themselves, per-model via `service_url` in the config plus a CODERAI_*_SERVICE_URL env var: vllm, ds4, kt, h3, pyannote, melotts, canary, crisperwhisper, parler Each verifies the remote's health check before returning it, so a stale or wrong URL fails loudly instead of surfacing as a confusing timeout later. vLLM additionally checks that the remote actually serves the requested model — the OCR path asks for surya-2 by name, and a generic remote vLLM would otherwise be proxied silently. colibri/k3 are left out: they speak a stdin/stdout mux protocol, not HTTP, so there is no URL to redirect. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ships the LoRA trainers for LTX-2, MiniMax-H3 and Krea 2, and the overlay venv that lets the last two run without moving the server off diffusers 0.38. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
H3 and Krea 2 need diffusers >= 0.40 while the server runs 0.38. Upgrading in place is not free: measured on this box, SDXL comes out bit-identical but Z-Image drifts (mean |diff| 0.0146, max 0.32, against a same-version control that is bit-identical) — so the upgrade would silently change image output for a model that is working fine. Since diffusers is imported once per process, the split only has to be per-process. The H3 venv is now created with --system-site-packages and given nothing but diffusers>=0.40 and huggingface_hub>=1.23; torch, transformers and the rest are inherited from the server's venv. That is 75 MB instead of the several GB a standalone venv costs, and keeps one torch on the machine. CODERAI_H3_STANDALONE_VENV=1 still builds a fully independent venv from requirements-h3.txt, for the day the parent's torch is too old. Training follows the same boundary: tools/lora_train_worker.py runs one job in the overlay — references written out as PNGs, request flattened to JSON, progress returned as JSON lines and mirrored onto the usual /v1/loras/progress so polling is unchanged. _train_via_overlay() drives it. Routing is by capability, not hardcoded: _diffusers_has(arch) tries to load the classes, so wan and ltx2 train in-process (0.38 has them) while h3 and krea go out-of-process — and if the main venv is ever upgraded they move back in-process with no config change. Verified: the overlay resolves diffusers 0.40 with torch/transformers inherited and leaves the parent untouched; a probe job ran under it, reached _train_flow_dit and reported back through the job file. The trainers themselves are still unrun against real weights. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
We had trainers for SD1.x/SDXL, Z-Image and Wan — none for the architectures the LoRAs people actually publish for video today. All three new ones are the same training problem as Wan (encode references to VAE latents, encode the prompt once, adapt the transformer's attention projections, regress the rectified-flow velocity), so rather than a fourth, fifth and sixth copy of that loop there is one shared _train_flow_dit driven by a per-architecture description in codai/api/lora_archs.py: which classes to import, the latent shape, the timestep shift, the text-encoder contract. Selection: `target` accepts image | video | wan | ltx2 | h3 | krea. `video` keeps meaning Wan for existing configs but now reads the model id, so an LTX-2 or H3 path routes itself; the explicit names force one. The character studio exposes the same choice as --target and a dropdown. Availability differs per architecture and fails with a message that says so rather than an ImportError from inside a training job: LTX-2 works on this diffusers (0.38 has LTX2VideoTransformer3DModel + AutoencoderKLLTX2Video), while H3 and Krea 2 need >= 0.40 — and Krea 2 additionally needs access to the gated krea/Krea-2-* weights. Verified: architecture detection and routing for every target/model combination, and that each trainer's classes either load or explain what they need. NOT run against real weights for any of the three — the contracts match the proven Wan path, but the first real run should be treated as a debugging session. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Read at README level. Its pipeline is topic -> LLM script -> LLM search terms -> stock/AI footage -> TTS -> subtitles -> music -> assemble -> publish, with every heavy step delegated to a cloud vendor. Four real gaps on our side: no planner (our tools start from prompts you write), no stock footage source, no 9:16/1:1 presets, no social publishing. Subtitles, background music and assembly we already have. Recorded with a proposal that takes its shape and keeps the work local. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Folds in the tool-detachment work: GET /v1/audio/speech/capabilities and the video editor asking for it over HTTP instead of importing codai. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Audit for detaching the bundled tools into their own containers, talking to coderai only through the HTTP API (aisbf proxying 1:1, assumed). One coupling existed: video_editor.py imported codai.api.tts_backends to fill its emotion and style pickers, which only works while it shares a process tree with the server. Added GET /v1/audio/speech/capabilities and the editor now asks over HTTP, falling back to the local import on an older server and to empty lists when neither answers — the pickers simply don't appear rather than the tool breaking. Everything else was already clean: no tool sends a local path into the API expecting the server to read it (media goes base64/multipart), none reads coderai's own state (models.json, /AI, ~/.coderai), and generated media is always fetched over HTTP rather than from a shared filesystem. docs/tools-as-services.md records the audit, what each container would need (ffmpeg, own volumes, base-url/api-key, long proxy timeouts, large bodies) and which tools are not API clients at all — review_outputs.py reviews the tools' own output dirs, and the *_service.py workers are coderai's own subprocesses. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ships the voice-cloning work: F5 engine caching and reference trimming, the engine choice on /v1/audio/clone (f5/xtts/chain/melotts), multi-clip voice profiles, AudioSeal watermarking, and MeloTTS as an isolated-venv worker. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Everything OpenVoice does that we didn't, built from models we already run rather than adopting theirs — plus MeloTTS itself so their exact V2 pairing is available. Engines on /v1/audio/clone (`engine`, default `auto`): f5 today's in-context clone — timbre AND prosody, needs the transcript xtts XTTS-v2 zero-shot from the clip, 17 languages, no transcript chain base TTS -> seed-vc timbre transfer: OpenVoice's architecture with seed-vc in place of its tone-colour converter melotts the same chain with MeloTTS as the base voice `auto` picks a transcript-free engine when the profile has no transcript or the target language differs from the reference — the case where F5 carries the wrong accent across. GET /v1/audio/clone/engines reports what is installed. The transcript is no longer a hard requirement: it is now checked per engine instead of rejecting the request outright. Multi-clip profiles: PATCH /v1/audio/voices/{name} takes add_clips/remove_clips, and a clone prompts on the cleanest take. Scored on clipping, level and crest factor — explicitly NOT loudest-wins, since a clipped take has the highest RMS of the set and is the worst possible prompt. Single-clip profiles keep working. Watermarking with AudioSeal (MIT), on by default, per-request or global override, detection at POST /v1/audio/watermark/detect. Missing audioseal degrades to unmarked audio with one warning, never an error. MeloTTS runs in an isolated venv worker like parler-tts — its pins conflict with this stack the same way. _RemoteParlerBackend generalised to _RemoteSpeechBackend since both speak the same POST /speak contract. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two costs on every cloned line. F5TTS(device=…) was constructed per request, so the model and its vocoder were loaded from scratch for each line of dialogue — seconds of latency and a fresh VRAM allocation every time. seed-vc next door already keeps a singleton; F5 now does too, rebuilt only if the device changes, and registered with register_external_vram_releaser so the manager can reclaim it like the LoRA base cache and the OCR engines. And nothing capped the reference. F5-TTS prompts on the clip and gains nothing past roughly 15 s, while a saved profile can hold minutes — the character studio uploads 90 s. Both the /v1/audio/clone path and the dialog path now trim to MAX_REF_SECONDS (15, override with CODERAI_F5_MAX_REF_SECONDS), leaving anything already short untouched. Co-Authored-By:
-
- 12 Sep, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authored
A resident model is no longer evicted to make room for itself. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.1.95 fixed the size — "sizing 'lisa-32k' by its own config" now fires and the requirement fell from 52.0 GB to the correct 22.9 GB — but the second request still evicted: Loading '…gemma-4-26B…gguf': need 22.9 GB VRAM, have 4.6 GB free — evicting On-request VRAM eviction: unloading active model 'lisa-32k' (measured 22.90 GB) Only 4.6 GB was free BECAUSE lisa-32k was resident and holding 22.9 GB of the card. The reservation asked whether there was room for a model that had already been loaded, found none, and evicted that very model to make room for it. It reaches ensure_vram_for at all because residency is checked under one id while the reservation arrives under another — self.models is keyed by the alias, the reservation by the weights path. So the guard checks every id the request answers to (model_key, resolved_name, requested_id) and returns early when any is resident. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Sibling configs are now told apart by their alias field. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Third attempt at this, and the first two failed for the same reason: they asked _config_for_model(requested_id) to find the sibling, but that helper resolves an alias THROUGH self.model_aliases to the shared weights path — landing right back on the wrong sibling. So the comparison was always "same object", the swap never happened, and 0.1.94 still logged [vram-est][debug] key='…/gemma-4-26B-A4B-…gguf' -> 51.97 GB | measured_cfg=51.969 Loading '…gguf': need 52.0 GB VRAM — evicting On-request VRAM eviction: unloading active model 'lisa-32k' (measured 22.47 GB) The estimator now scans self.config for an entry whose `alias` field IS the requested id. Aliases are unique per entry, so it needs no path comparison and doesn't depend on how self.config is keyed — which is what the earlier attempts got wrong. The path-based rule stays as a fallback for entries without an alias. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Same-path sibling sizing, now on the path text generation actually takes (get_model_for_request -> _load_model_by_name -> ensure_vram_for). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The previous two attempts wired request_model — which only video and image generation use. Chat completions go through get_model_for_request -> _load_model_by_name -> ensure_vram_for, and that path collapses the alias the same way, so on 0.1.93 the second lisa-32k request still logged [vram-est] key='…/gemma-4-26B-A4B-…gguf' source=measured(force) -> 51.97 GB Loading '…gguf': need 52.0 GB VRAM, have 4.6 GB free — evicting models On-request VRAM eviction: unloading active model 'lisa-32k' (measured 22.90 GB) get_model_for_request now keeps the requested id, passes it to _load_model_by_name, and from there to both ensure_vram_for and the expected_vram_gb the backend uses to size its GPU-layer split — so lisa-32k is sized as itself (22.9 GB) rather than as lisa (51.97 GB). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Completes the same-path sibling fix: the requested id is now kept through alias resolution, so lisa-32k is sized as itself instead of lisa's 51.97 GB. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The previous attempt at this didn't fire. request_model overwrites the requested name with the alias target on its first resolution step: if requested_model in self.model_aliases: requested_model = self.model_aliases[requested_model] so by the time anything looks a config up, 'lisa-32k' no longer exists — model_key and resolved_name are both the weights path, they compare equal, and the same-path-sibling check had nothing to compare. Confirmed live on 0.1.92: the second lisa-32k request still logged "need 52.0 GB VRAM, have 4.6 GB free" and evicted the model that was serving it. Keep the original id in _requested_id before that overwrite, hand it to ensure_vram_for at all three reservation sites, and let _get_model_used_vram_gb try it ahead of resolved_name. Against the live config: lisa-32k now sizes at 22.44 GB instead of lisa's 51.97 GB, and qwen38-128k at 19.50 instead of qwen38's 18.00. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Adds the same-path sibling fix: a request for lisa-32k was sized with lisa's 51.97 GB footprint (same weights file, 178000 vs 32768 context), which evicted the model that was serving the request and reloaded it at 18 of 65 layers. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The second lisa-32k request evicted the model that was already serving it, then reloaded the same file at 18 of 65 layers: # >>> REQUEST model=lisa-32k [vram-est] key='/AI/guffcache/gemma-4-26B-…gguf' -> 51.97 GB Loading '/AI/guffcache/gemma-4-26B-…gguf': need 52.0 GB, have 12.6 GB free — evicting On-request VRAM eviction: unloading active model 'lisa-32k' (measured 14.33 GB) n_gpu_layers : 18 | n_ctx : 32768 lisa and lisa-32k are the same weights file at two contexts — 178000 and 32768 — so they measure 51.97 GB and 14.33 GB. The VRAM estimate was keyed by the PATH, which matches whichever sibling registered it, so the small config was sized with the big one's footprint: a 52 GB requirement that nothing could satisfy, an eviction of the running model, and a crippled reload. _get_model_used_vram_gb now prefers the config of the name actually requested when it names a same-path sibling. Four pairs on this box were affected, including the qwen38 / qwen38-128k pair added yesterday. Co-Authored-By:
-