- 13 Sep, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authoredEverything OpenVoice does that we didn't, built from models we already run rather than adopting theirs — plus MeloTTS itself so their exact V2 pairing is available. Engines on /v1/audio/clone (`engine`, default `auto`): f5 today's in-context clone — timbre AND prosody, needs the transcript xtts XTTS-v2 zero-shot from the clip, 17 languages, no transcript chain base TTS -> seed-vc timbre transfer: OpenVoice's architecture with seed-vc in place of its tone-colour converter melotts the same chain with MeloTTS as the base voice `auto` picks a transcript-free engine when the profile has no transcript or the target language differs from the reference — the case where F5 carries the wrong accent across. GET /v1/audio/clone/engines reports what is installed. The transcript is no longer a hard requirement: it is now checked per engine instead of rejecting the request outright. Multi-clip profiles: PATCH /v1/audio/voices/{name} takes add_clips/remove_clips, and a clone prompts on the cleanest take. Scored on clipping, level and crest factor — explicitly NOT loudest-wins, since a clipped take has the highest RMS of the set and is the worst possible prompt. Single-clip profiles keep working. Watermarking with AudioSeal (MIT), on by default, per-request or global override, detection at POST /v1/audio/watermark/detect. Missing audioseal degrades to unmarked audio with one warning, never an error. MeloTTS runs in an isolated venv worker like parler-tts — its pins conflict with this stack the same way. _RemoteParlerBackend generalised to _RemoteSpeechBackend since both speak the same POST /speak contract. Co-Authored-By:Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two costs on every cloned line. F5TTS(device=…) was constructed per request, so the model and its vocoder were loaded from scratch for each line of dialogue — seconds of latency and a fresh VRAM allocation every time. seed-vc next door already keeps a singleton; F5 now does too, rebuilt only if the device changes, and registered with register_external_vram_releaser so the manager can reclaim it like the LoRA base cache and the OCR engines. And nothing capped the reference. F5-TTS prompts on the clip and gains nothing past roughly 15 s, while a saved profile can hold minutes — the character studio uploads 90 s. Both the /v1/audio/clone path and the dialog path now trim to MAX_REF_SECONDS (15, override with CODERAI_F5_MAX_REF_SECONDS), leaving anything already short untouched. Co-Authored-By:
-
- 12 Sep, 2026 14 commits
-
-
Stefy Lanza (nextime / spora ) authored
A resident model is no longer evicted to make room for itself. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.1.95 fixed the size — "sizing 'lisa-32k' by its own config" now fires and the requirement fell from 52.0 GB to the correct 22.9 GB — but the second request still evicted: Loading '…gemma-4-26B…gguf': need 22.9 GB VRAM, have 4.6 GB free — evicting On-request VRAM eviction: unloading active model 'lisa-32k' (measured 22.90 GB) Only 4.6 GB was free BECAUSE lisa-32k was resident and holding 22.9 GB of the card. The reservation asked whether there was room for a model that had already been loaded, found none, and evicted that very model to make room for it. It reaches ensure_vram_for at all because residency is checked under one id while the reservation arrives under another — self.models is keyed by the alias, the reservation by the weights path. So the guard checks every id the request answers to (model_key, resolved_name, requested_id) and returns early when any is resident. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Sibling configs are now told apart by their alias field. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Third attempt at this, and the first two failed for the same reason: they asked _config_for_model(requested_id) to find the sibling, but that helper resolves an alias THROUGH self.model_aliases to the shared weights path — landing right back on the wrong sibling. So the comparison was always "same object", the swap never happened, and 0.1.94 still logged [vram-est][debug] key='…/gemma-4-26B-A4B-…gguf' -> 51.97 GB | measured_cfg=51.969 Loading '…gguf': need 52.0 GB VRAM — evicting On-request VRAM eviction: unloading active model 'lisa-32k' (measured 22.47 GB) The estimator now scans self.config for an entry whose `alias` field IS the requested id. Aliases are unique per entry, so it needs no path comparison and doesn't depend on how self.config is keyed — which is what the earlier attempts got wrong. The path-based rule stays as a fallback for entries without an alias. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Same-path sibling sizing, now on the path text generation actually takes (get_model_for_request -> _load_model_by_name -> ensure_vram_for). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The previous two attempts wired request_model — which only video and image generation use. Chat completions go through get_model_for_request -> _load_model_by_name -> ensure_vram_for, and that path collapses the alias the same way, so on 0.1.93 the second lisa-32k request still logged [vram-est] key='…/gemma-4-26B-A4B-…gguf' source=measured(force) -> 51.97 GB Loading '…gguf': need 52.0 GB VRAM, have 4.6 GB free — evicting models On-request VRAM eviction: unloading active model 'lisa-32k' (measured 22.90 GB) get_model_for_request now keeps the requested id, passes it to _load_model_by_name, and from there to both ensure_vram_for and the expected_vram_gb the backend uses to size its GPU-layer split — so lisa-32k is sized as itself (22.9 GB) rather than as lisa (51.97 GB). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Completes the same-path sibling fix: the requested id is now kept through alias resolution, so lisa-32k is sized as itself instead of lisa's 51.97 GB. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The previous attempt at this didn't fire. request_model overwrites the requested name with the alias target on its first resolution step: if requested_model in self.model_aliases: requested_model = self.model_aliases[requested_model] so by the time anything looks a config up, 'lisa-32k' no longer exists — model_key and resolved_name are both the weights path, they compare equal, and the same-path-sibling check had nothing to compare. Confirmed live on 0.1.92: the second lisa-32k request still logged "need 52.0 GB VRAM, have 4.6 GB free" and evicted the model that was serving it. Keep the original id in _requested_id before that overwrite, hand it to ensure_vram_for at all three reservation sites, and let _get_model_used_vram_gb try it ahead of resolved_name. Against the live config: lisa-32k now sizes at 22.44 GB instead of lisa's 51.97 GB, and qwen38-128k at 19.50 instead of qwen38's 18.00. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Adds the same-path sibling fix: a request for lisa-32k was sized with lisa's 51.97 GB footprint (same weights file, 178000 vs 32768 context), which evicted the model that was serving the request and reloaded it at 18 of 65 layers. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The second lisa-32k request evicted the model that was already serving it, then reloaded the same file at 18 of 65 layers: # >>> REQUEST model=lisa-32k [vram-est] key='/AI/guffcache/gemma-4-26B-…gguf' -> 51.97 GB Loading '/AI/guffcache/gemma-4-26B-…gguf': need 52.0 GB, have 12.6 GB free — evicting On-request VRAM eviction: unloading active model 'lisa-32k' (measured 14.33 GB) n_gpu_layers : 18 | n_ctx : 32768 lisa and lisa-32k are the same weights file at two contexts — 178000 and 32768 — so they measure 51.97 GB and 14.33 GB. The VRAM estimate was keyed by the PATH, which matches whichever sibling registered it, so the small config was sized with the big one's footprint: a 52 GB requirement that nothing could satisfy, an eviction of the running model, and a crippled reload. _get_model_used_vram_gb now prefers the config of the name actually requested when it names a same-path sibling. Four pairs on this box were affected, including the qwen38 / qwen38-128k pair added yesterday. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ships the auto-offload headroom fix: a measured footprint already includes the KV cache, so reserving a context-scaled amount on top of it spilled layers to CPU that had room on the card. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
qwen38 (27B, n_ctx 32000, q4_0 KV) loaded with 48 of 66 layers on the GPU and 4.9 GB of weights on the CPU, for no reason: the auto-offload sizer reserves "2 GB + n_ctx/12000" = 4.7 GB of headroom for the KV cache, while llama.cpp's own numbers put the whole KV at 0.55 GB (CUDA0 421.88 MiB + CPU 140.62 MiB). With 20.9 GB free and ~17 GB really needed, everything fitted. Two assumptions behind that reserve are wrong when they don't hold: that the VRAM estimate is weights-only, and that the KV is f16. So the reserve is now 1.5 GB (compute buffers) when the config carries a measured footprint — that number already includes the KV, adding a context-scaled reserve on top double counts — and 1.5 GB + n_ctx/40000 when the KV is quantized. The old heuristic stays for a raw, unmeasured, f16 case. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ships the studio's streaming uploads and local frame sampling, the KV-cache placement planner (with swa_full and the quantization ladder), split-GGUF shard resolution, and the n_ctx fallback that no longer lands on 2048. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The qwen38 request loaded with "n_ctx : 2048" and failed on any real prompt. The config says 32000 — but it was added to models.json after the engine started, so the engine's registry didn't have it. The front DID know the alias and routed the request, the backend resolved the alias to a local GGUF through its bare-path fallback, and with no n_ctx in the load kwargs the code took llama.cpp's 2048 default. Nothing in the log explained why a 27B model suddenly had a 2k window. So when no n_ctx arrives with the request: look the config up by the resolved path (the manager's _config_for_model already resolves id/alias/basename), and only if that finds nothing fall back to min(native context, 32768) with a warning naming both numbers. On all three models on this box that turns 2048 into 32768. The immediate cause still needs an engine reload to pick the new entries up; this is about the failure being silent and the floor being useless. Co-Authored-By:
-
- 11 Sep, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authored
A request for Qwen3.8-Flash-Next failed six times with nothing but "Failed to load model from file". The path was ...-00006-of-00006.gguf: the LAST shard of a 6-part split GGUF. llama.cpp opens a split model at shard 00001 and finds the rest itself; handed any other shard it just fails. And the set was incomplete anyway — only shard 6 was on disk, the download having pulled a single shard rather than the set. So a .gguf path is now resolved before the load: a shard filename maps to shard 1, and the whole set is checked first. An incomplete set raises once, naming which shards are present and which are missing, instead of six identical retries of a message that says nothing. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two levers were missing from the KV plan. The one that costs nothing: llama-cpp-python 0.3.30 accepts `swa_full`, and we never passed it. On a sliding-window model — gemma-4-31B has a 1024-token window on a 5:1 pattern — swa_full=True makes the sliding layers reserve the FULL context like the global ones. Turning it off reserves only the window, and it costs no quality: it limits context reuse across requests, not attention. So it is now both a config passthrough and the FIRST rung of the fallback ladder, ahead of touching precision at all. The other: the ladder was q8_0 then q4_0, symmetric. It is now three rungs — (q8_0,q8_0) 24.3 GB, (q8_0,q4_0) 18.6 GB, (q4_0,q4_0) 12.9 GB on tonight's measured 45.7 GB — keeping K more precise than V in the middle, since V tolerates quantization better. There is no rung below that: 4.5 bits (q4_0 / iq4_nl) is llama.cpp's floor for a KV cache. The sub-4-bit ggml types this build exposes (IQ2_*, Q2_K, IQ3_*) are weight quantizations, not KV cache types. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Four SIGKILLs tonight (19:01, 19:32, 20:42, 21:00), each at ~45 GB RSS on a 54 GB box, every one of them the nvidia-gguf engine mid-load — and with it every model resident on that engine, which is why unrelated requests 502'd. The chain: gemma-4-31B at n_ctx 128000 doesn't fit 24 GB of VRAM, llama.cpp fails to create the context, and the retry silently allocates the KV cache in HOST RAM (`CPU KV buffer size = 46750.00 MiB`). Nothing checked whether host RAM could take it. The existing auto-offload sizes layers by weight only; its KV headroom heuristic (2 GB + n_ctx/12000) predicted ~12.7 GB against a real 45.7. So size the KV first and pick a placement that fits: VRAM when there's room, host RAM when there isn't, a quantized KV when neither can hold it at f16, and a clean refusal when even that won't fit — a failed load beats an OOM-killed engine. The context is never shrunk; that stays the caller's decision. It acts only on a MEASURED size. Predicting KV from GGUF metadata is architecture-specific — Gemma-4 carries a per-layer head_count_kv array, separate SWA key/value lengths and a sliding-window pattern — and the naive formula came out 5x high while the SWA-aware one came out 4x low on this very model. So the load now parses llama.cpp's own "KV buffer size" lines, reports the real total split by where it landed, and prints the measured_kv_gb line to record; estimates are logged for visibility but never block a load. Known gap: the first load of a model at a given n_ctx has no measurement yet, so it can still be the one that OOMs. Tonight's value is known (45.7 GB at 128000) and can be recorded by hand to cover it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Uploads were read in the browser with FileReader, base64'd, and posted as one JSON document holding every source — so a phone video was a ~700 MB string in the browser, another copy in the request, and another in the handler, and nginx's 1 MB default rejected it long before any of that mattered. Now the browser POSTs each File as a raw body to /api/upload, which streams it to disk a megabyte at a time and hands back a path; extraction takes paths. Nothing base64s client-side, and neither end ever holds a whole file. JSON bodies are capped at 32 MB (they only carry settings now) and the upload endpoint is dispatched before the body is parsed, since it is binary. Big videos are also frame-sampled LOCALLY before anything is sent: the extractor only ever looks at evenly-spaced frames, so shipping 24 of them instead of the film is the same result for a fraction of the traffic (--max-upload-mb, default 64; 0 sends the file). Oversized stills are downscaled to 2048px, and voice cloning now sends a 90s mono clip rather than the entire soundtrack. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ships the character studio's LoRA training and voice cloning, and MiniMax-H3 support (in-engine when the engine's diffusers can do it, isolated venv worker otherwise). Co-Authored-By:
-
- 10 Sep, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authored
H3 doesn't need a permanent sidecar. Model work already happens in a supervised, per-GPU engine subprocess with its own CUDA context, VRAM manager and swap gate — so H3 belongs there, registered with the model manager like a Wan or LTX pipeline, and pinned to a CUDA engine. What actually forced the worker was the version floor, not the architecture: diffusers ships H3 as Modular blocks only (0.40+), and this venv is on 0.38. So the route is now chosen by capability, not hardcoded — engine_supports_h3() probes for MiniMaxH3ModularPipeline, and `in_process` in the model config forces either side. The same build works before and after a diffusers upgrade, and the worker stays as the fallback while H3 is unproven (an in-engine OOM or device assert poisons the whole engine's context; a subprocess crash doesn't). _load_h3_pipeline() builds the modular pipeline with ONE workflow (loading with none pulls both 61.7GB partitions), group-offloads the big modules per component since a ModularPipeline has no pipeline-level offload helper, and registers it with record_vram_delta so eviction sees its real footprint. A workflow change unloads first — the two partitions can't be co-resident. The checkpoint contracts now live in tools/h3_common.py, loaded by path from the engine and imported as a sibling module inside the isolated venv, so both routes snap frames, round the canvas, pick the workflow and mux the audio identically. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
MiniMax-H3 is a 33B flow-matching DiT that denoises video and audio in ONE packed sequence. diffusers ships it as Modular blocks only (MiniMaxH3ModularPipeline, 0.40+) — there is no DiffusionPipeline half, so it fits neither _detect_pipeline_class() nor this venv's diffusers 0.38. It runs in an isolated venv, the same way pyannote, NeMo-Canary and vLLM do: tools/h3_service.py generation inside the isolated venv (HTTP) codai/api/h3_worker.py venv bootstrap, process lifecycle, eviction handle requirements-h3.txt the venv's pins (diffusers>=0.40, transformers 5.x) _generate_h3() adapts the request to the checkpoint's contracts instead of forwarding it, and reports every adaptation in the response warnings: guidance is baked in (negative_prompt and guidance_scale dropped), 24 fps is fixed and num_frames snaps up to the next 17n+5 within 5-15s, canvas axes round to 32, and v2v/interp are rejected. character_profiles become native ref2va references (max 9) so H3 skips the identity-keyframe bridge; an explicit keyframe wins, since fl2va and ref2va are separate transformer partitions. The worker picks a workflow per request and never loads without one — that pulls both 61.7GB partitions. The worker registers as an evictable, VRAM-tracked model; eviction stops the subprocess, the only way the transformer's pages actually come back. Character studio: clone the voice off the same footage the face came from (/v1/audio/voices/extract) with `extract --voice`, the new `voice` subcommand or the UI checkbox, then `--say` a line that is lip-synced in that cloned voice. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
IP-Adapter conditioning drifts on long shots and profile angles, so the studio now has the middle step: train a LoRA straight from the profile's reference crops and apply it to the generation. `POST /v1/loras/train` already resolves images server-side from `character: <name>`, so the trainer sees exactly the crops you reviewed and pruned — no re-upload. The job runs detached (`wait: false`); the studio polls `/v1/loras/progress`, persists the server job id per LoRA and re-attaches to a run still in flight after a client restart instead of starting a second one, resubmitting to resume from checkpoint when a job was interrupted. CLI: `train`, plus `--lora NAME[:WEIGHT]` / `--image-lora` on `video` and a `--train` flag on `demo` for extract -> train -> generate in one command. Web UI: a training panel (target, steps, rank, resolution, frames, LR, QLoRA toggle, cancel) and a LoRA multi-select on the scene panel. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The repo root accumulates `coderai-base-<date>.tar.gz` docker-save exports (~13 GB each, 79 GB today) plus LoRA `checkpoints/`, none of which .dockerignore excluded — every image build shipped them to the daemon as build context. Also excludes the character studio's local output dir. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Adds tools/character_studio.py, a demo app over the character-profile endpoints: extract a character from real photos/videos (POST /v1/characters/extract), review and prune the reference crops, then generate a brand-new video starring that person (POST /v1/video/generations with character_profiles + keyframe_identity). CLI core plus a `web` subcommand serving the same pipeline in the browser. The UI honours X-Forwarded-Prefix, so it works at the root or under a sub-path mount. Every generated mp4 gets its request body written alongside it, so the demo also doubles as a worked API example. Wired in as a bundled tool like the other UIs: supervisord program on port 7791 (autostart via CODERAI_TOOL_CHARACTER_STUDIO, on by default), entrypoint env defaults, run_oci.sh --enable/--disable-tool + --tool-arg names, nginx route at /character/, service smoke test, and the run banner/README URLs. Version bumped to 0.1.88 so the in-image upgrade path ships it. Co-Authored-By:
-
- 09 Sep, 2026 9 commits
-
-
Stefy Lanza (nextime / spora ) authored
Ships the audio pipeline work (real music dub, demucs separation, DeepFilterNet restoration), the video fixes (camera_motion, dialog voice cloning, managed Wav2Lip, IP-Adapter identity conditioning + keyframe bridge), the huggingface_hub compat shim, and the documentation refresh. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The keyframe bridge dispatched its nested image request back through the front via execute_api_request. That deadlocks: by the time the engine handler runs, the front's GPU-swap gate has already made the VIDEO model the GPU owner for this request, so the nested request for a different (image) model queues behind the very request that is blocked waiting for it. Observed on a real Wan run: [gpu-swap] queued 'stabilityai/stable-diffusion-xl-base-1.0' behind owner 'wan-ai/wan2.2-i2v-a14b-diffusers' with the GPU sitting at 0% until the request was killed. Call codai.api.images.create_image_generation directly instead. The engine already holds the GPU for this request, and its own model manager handles the load/evict — which is also what we want, since the keyframe must be rendered before the video weights land. Verified end to end on real hardware (Wan2.2-I2V-A14B, 4-bit, model-CPU-offload): t2v request + character_profiles -> "rendering IP-Adapter keyframe" -> "IP-Adapter loaded" + 72 modules aligned to bf16 -> keyframe in 10 steps -> "running ti2v from it" -> Wan loaded (24.45 GB VRAM measured) -> HTTP 200 in 1642s with a real 480x480 h264 video, 17 frames, 2.125s, motion present between first and last frame. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Verified end to end on real hardware: same prompt, same seed, mean absolute pixel difference 46/255 between a generation with a character profile and one without. Before this, that difference was 0 — the profile changed nothing. IP-Adapter had never once loaded successfully. Four separate blockers, each hidden behind the previous one, all found by running it rather than reading it: 1. Attention slicing (71f88d14). diffusers' adapter loader rebuilds every attention processor; re-instantiating SlicedAttnProcessor without slice_size raised, the failure was caught and logged as a warning, and generation continued without conditioning. Slicing is now off across the load. 2. Quantized dtype cast. _load_ip_adapter_weights() ends with self.to(dtype=self.dtype) on the denoiser — a NO-OP cast to the dtype it already has — but bitsandbytes refuses any dtype argument on a quantized model. Tolerated for the duration of the load. 3. That tolerance then broke the forward pass: the suppressed cast was what brought the NEWLY added IP-Adapter modules up to the model dtype, so they stayed fp16 against a bf16 pipeline ("expected mat1 and mat2 to have the same dtype: BFloat16 != Half"). We now perform that cast ourselves on the non-quantized modules only — 72 of them, confirmed in the log. 4. Reference-image shape. A character profile carries several images, but diffusers matches the OUTER list against the number of ADAPTERS, not images: "Got 4 images and 1 IP Adapters". Multiple images for one adapter must be nested one level deeper. New shared helper ip_adapter_image_arg(), used by both the image and video paths. Also fixes a real production bug found while testing: pipelines are cached, and once an adapter is attached the UNet config demands image_embeds forever after, so the NEXT request without character references 500s with "...encoder_hid_dim_type set to 'ip_image_proj' which requires the keyword argument image_embeds". The adapter is now detached when a request doesn't need it (with slicing kept off across the unload too, which trips the same constructor). Verified lifecycle on one cached pipeline: with-profile -> without -> with again, all 200, and the two with-profile results are bit-identical (0.0 diff), so unload/reload restores conditioning exactly. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Found by actually running it, not by reading it. Every IP-Adapter load was failing with SlicedAttnProcessor.__init__() missing 1 required positional argument: 'slice_size' and _ensure_ip_adapter_loaded() caught it, logged a warning and returned False — so character_profiles fell back to the prompt hint on the ONE path that was supposed to have real identity conditioning. The image came back looking plausible, which is why it was never noticed. Cause: we enable attention slicing for VRAM headroom, and diffusers' adapter loader rebuilds every attention processor; re-instantiating a SlicedAttnProcessor without its required slice_size raises. Slicing is now disabled for the load and left off while the adapter is in use (re-enabling would swap the IP-Adapter processors straight back out), and restored if the load fails anyway. Verified: the SlicedAttnProcessor error is gone. NOT yet working — the next blocker, now visible underneath: on a QUANTIZED pipeline the load fails with "Casting a quantized model to a new dtype is unsupported". So IP-Adapter still cannot attach to a 4-bit image model, and identity conditioning needs an unquantized image model (or a follow-up that loads the image encoder with an explicit dtype). Do not describe character identity as working end to end until that is resolved. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Loading the IP-Adapter (958d2e09) fixed the mechanism, but only AnimateDiff-class pipelines have one. On Wan / CogVideoX / LTX / SVD — the models people actually use — character_profiles still did nothing except prepend the profile name to the prompt, so identity was not working where it mattered. The image endpoint does have real IP-Adapter. So when a video model can't condition on references itself, render the first frame there against the same profiles and run the video as ti2v from that keyframe. This is the recipe tools/videogen.py already drives by hand, now done server-side. It runs BEFORE the video pipeline is loaded, so an image model and a video model are never resident at once, and it dispatches through the front (execute_api_request) so the image model may live on a different engine. New request fields: keyframe_identity (auto | always | never), keyframe_model, keyframe_steps. 'auto' skips the bridge when the model has native IP-Adapter or a first frame was already supplied. If the keyframe can't be rendered the request still proceeds — identity degrades to the prompt hint plus any per-character LoRA and the reason is reported in the response warnings — unless the caller asked for 'always', which is an explicit requirement and so returns 502 rather than silently producing a stranger. Verified offline across every branch: bridge fires for Wan and rewrites to ti2v with the keyframe; skipped for AnimateDiff; forced by 'always'; disabled by 'never'; no-op with no profiles or an existing first frame; degrades with a warning on image-endpoint failure and raises 502 under 'always'. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Correction to the previous commit. The OCI image already ships a complete lip-sync integration — packaging/linux/launcher/{wav2lip,sadtalker} shims on PATH, a dedicated lipsync_venv, baked Wav2Lip AND SadTalker source, first-use checkpoint download and a smoke test. The claim that the binary "essentially never exists" was wrong for the packaged deployment; _apply_lipsync() uses those shims and never reaches the managed path there. The managed install is therefore only for from-source installs (build.sh + venv_all), where nothing is on PATH and lip sync did silently degrade. More importantly it had claimed CODERAI_WAV2LIP_DIR — the same variable the shim uses for its writable working copy, seeded from the baked read-only source and run against the lipsync venv's pinned deps. Applying our numpy/librosa/torch rewrites to that tree would have corrupted a working packaged install. The managed copy now lives under CODERAI_WAV2LIP_MANAGED_DIR (~/.coderai/wav2lip-managed), whose basename also cannot collide with the shim's ~/.coderai/Wav2Lip on a case-insensitive filesystem. Re-verified after the move: fetch/patch/run still produces a valid synced mp4. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The last two gaps from the person-to-scene audit. IP-Adapter on the video path never worked. _apply_character_refs() set kw['ip_adapter_image'] without ever calling load_ip_adapter(), so a pipeline that does accept the kwarg (AnimateDiff) failed inside diffusers rather than transferring identity, and it also passed ip_adapter_scale as a call kwarg, which no pipeline accepts — a second latent TypeError. It now reuses the image path's _ensure_ip_adapter_loaded(), sets the scale through set_ip_adapter_scale(), and returns whether conditioning is actually active. Pipelines with no IP-Adapter at all (Wan/CogVideoX/LTX/SVD) now say so instead of looking like they worked. Lip sync required a 'wav2lip' or 'sadtalker' executable on PATH — a file that essentially never exists, since Wav2Lip ships neither a pip package nor a binary and its original checkpoint links are dead. codai/api/lipsync.py now owns the lifecycle the way ds4_worker owns ds4: it fetches code AND weights from the camenduru/Wav2Lip mirror on first use, patches what no longer runs on modern numpy/librosa/torch, creates the scratch dirs inference.py expects, and drives it as a subprocess. An operator-installed binary still takes precedence. Each patched file is compiled before being kept, so a bad rewrite is caught at patch time and reverted rather than surfacing as a SyntaxError from a subprocess (which is exactly how the first attempt failed: two librosa patterns compounded into 'sr=sr='). Lip-sync fallback is no longer silent. The reason travels up a per-request warnings list — not module state, which would cross-talk between concurrent requests — and appears as a `warnings` field on the video response, so a client can tell a synchronised result from a plain audio mux. Verified offline (live container untouched): Wav2Lip downloads, patches, and lip-syncs a real face to audio, producing a valid 768x512 h264+aac mp4 that differs from the source; the second run reuses the install with no download; the fallback populates warnings; IP-Adapter loads its weights and sets the scale for a supporting pipeline and cleanly declines for a Wan-like one. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three defects found while auditing the 'person from a video -> scene with their voice' path. 1. camera_motion returned 500 on every request that used it. _apply_camera_motion() set kw['camera_motion'] unconditionally and nothing ever popped it, so pipe(**kw) raised TypeError — no diffusers video pipeline accepts that kwarg (the 'CogVideoX supports it natively' comment was wrong). tools/videogen.py sends the field, so that path was dead. Now the kwarg is passed only when the pipeline signature really takes it, otherwise the motion is folded into the prompt, which is the only conditioning these models have. 2. dialogs[].voice never cloned the voice it named. _generate_tts_for_line() loaded the profile's reference audio and then used it purely as a flag to force a generic voice (en-US-JennyNeural / af_sky) — F5-TTS was never invoked, so the selected identity was silently discarded. It now clones with F5-TTS when the profile has reference audio AND a transcript, and says why in the log when it falls back. 3. seed-vc and F5-TTS could not load their BigVGAN vocoder. huggingface_hub 1.x stopped forwarding proxies/resume_download to a subclass _from_pretrained; BigVGAN still declares them as required keyword-only args, so loading died with 'missing 2 required keyword-only arguments'. Pinning hub below 1.0 is not possible (transformers 5.x needs the new one), so codai/api/hub_compat.py supplies the dropped arguments for exactly the subclasses whose signature still requires them. This also un-breaks /v1/audio/convert. Also fixed on the way: - edge-tts never ran: get_event_loop().run_until_complete() inside an executor thread raises RuntimeError. In _generate_tts_for_line it was swallowed; in _generate_tts only ImportError was caught, so it propagated out of /v1/video/dub. Both now use asyncio.run() and verify the output file. - music dub adopted a failed Seed-VC result: converted_path was assigned before the write succeeded, so a near-empty conversion got remixed instead of falling back to the F5-TTS vocal. The result is now validated before adoption. Verified offline (live container untouched): camera_motion passes through only to a supporting pipeline and otherwise reaches the prompt; a dialog line clones with the profile's reference audio and transcript and refuses to clone without one; seed-vc loads BigVGAN and converts; the music dub falls back to the full vocal and reports the reason. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three related placeholders were shipping as working features: - audio_stems.separate_with_provider() and audio_clean.restore_with_provider() were one-line stubs that unconditionally raised 501, so /v1/audio/stems and /v1/audio/cleanup only ever worked in ffmpeg fallback_mode despite demucs and deepfilternet being declared dependencies and probed by detect_audio_backends. - run_full_music_dub() ran only the STT step and returned hardcoded paths (vocals.wav, final_mix.wav) plus 'status': 'placeholder' for the other four, while the endpoint reported 'status': 'available'. Now: - separate_with_provider drives demucs (demucs.api, CLI fallback), keeps the vocals stem and sums drums+bass+other into an instrumental bed for vocals-instrumental; 4-stem passes the four through. - restore_with_provider runs DeepFilterNet (VoiceFixer fallback) and then applies the hum/click/loudness DSP with ffmpeg, so those options are no longer lost. - audio-music-dub is a real six-stage chain: separate -> transcribe the ISOLATED vocal -> adapt lyrics (LLM prompted for singable output with matching syllable count, argostranslate fallback) -> re-sing with F5-TTS cloned from the original vocal -> optional Seed-VC f0-conditioned singing pass -> ffmpeg remix + limiter. New request fields: text_model, voice_name/ref_text, sing_convert, diffusion_steps, pitch_shift, speed, seed, fallback_mode. Missing optional deps now degrade to a reported 'skipped'/'failed' step with a reason and a top-level status of 'complete' vs 'degraded' plus warnings[], rather than silently producing placeholders. 501s name the package to install. Verified offline with a fake-model harness: the chain produces a real RIFF wav mix end to end, feeds the vocal stem (not the mix) to STT, and correctly reports degraded when seed-vc is unavailable. Co-Authored-By:
-
- 07 Sep, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authored
The README had not been touched since the OCR/multi-engine work in August and described roughly half of what the server does. Add the missing subsystems and correct the ones that overstated the code: - Inference engines: the per-model backend matrix (transformers/gguf/ds4/ colibri/k3/kt/vllm/runpod) and how one is selected. - RunPod: feature summary plus a full docs/runpod.md — pods vs serverless, per-model config keys, cold start, GPU fallback, stuck-boot retry, budgets and the ledger, the stale-pod reaper, spillover (with its real scope), observability and RunPod-only deployments. - STT backends, per-model languages/translation, word timestamps, diarization and enrolled speaker recognition; embeddings families (BGE-M3 multi-vector, GME-Qwen2-VL, GeoCLIP, VPR) and /v1/rerank; the OCR subsystem and its schema store; LoRA training. - API tables for /v1/ocr*, /v1/rerank, /v1/loras*, the audio/speaker routes and the two missing pipelines. - models.json reference incl. multi-config (config_id/alias), the front/engine split, optional-subsystem requirements files, and a docs/ index. Honesty fixes: /v1/pipelines/audio-music-dub is flagged as incomplete (only the transcription step runs); spillover is documented as serverless-target / direct-path only. Co-Authored-By:
-
- 06 Sep, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authored
For a coderai instance that only orchestrates RunPod (no local models), the node that hosts the proxy must advertise the runpod capability. The no-GPU path already builds its engine with backend "auto" (which has it), but an explicit engine_specs backend:"cpu" and the unknown-backend fallback did not — those would 503. - _DEFAULT_CAPS gains "cpu": {transformers, gguf, k3, runpod}. - Unknown-backend fallback now {transformers, gguf, runpod} (runpod needs no local hardware, so every node can host it). Verified by simulating pick_engine for a runpod-pinned model on auto/cpu/nvidia nodes — all route instead of 503. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The 300s boot cap was a hard-coded constant; large models need much more time for the image pull + weight download. Both budgets are now per-model config: - boot_timeout_s (default 300) — until the pod exposes its port; a stuck machine is still dropped fast and the next candidate tried. - load_timeout_s (default 600) — until vLLM answers /v1/models; raise for big models. Parsed in RunpodModelConfig (min 30s), used by _provision_one in place of the class constants (which remain the fallback), whitelisted in api_model_configure, and exposed on the model page with hints. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Live run showed get_pod_logs hitting HTTP 400 on {rest}/pods/{id}/logs?tail=N, so a failed boot fell back to the console link with no output. Now tries the plausible routes in order (no params, ?limit=, containerLogs) and keeps the first that answers, reporting the collected errors otherwise. The console deep-link remains the always-available path. Co-Authored-By:
-