- Oct 29, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Six findings from one 21h30m production log (548,687 lines, 51 loads of 12 models). They share a cause: nothing measured a model directly, so every decision downstream of a measurement was made on a number that was wrong. 1. The footprint was a process-RSS delta --------------------------------------- `measured_ram = self._get_own_ram_gb() - ram_before`, where _get_own_ram_gb sums RSS over the engine and every child. RSS counts mmap'd weight-file pages the loader just walked; summed over a tree it counts pages SHARED between those processes once per process; and as a delta it absorbs anything else that moved in the window. Against ground truth: gemma-2-9b-it.Q4_K_M 5.37 GB on disk -> reported 30.81 GB (5.7x) LTX-2.5, six loads, one unchanged config: 22.11, 22.11, 23.28, 4.50, 19.02, 29.80 GB (6.6x) Every other model sat at 1.0-1.2x, so this was not noise. Wan2.2's 3.4x spread was the real quantization fix and is the one that belonged there. codai/models/footprint.py measures the object instead. Quantization needs no correction factor, which is the point: a bitsandbytes Params4bit IS a uint8 tensor, so numel x element_size already gives packed bytes. What those formats hide is counted explicitly -- bnb's quant_state (absmax, code book, nested double-quant state) lives outside parameters(), and GPTQ/AWQ keep qweight/qzeros/scales in BUFFERS, so a parameters-only walk reports a 4-bit model as nearly empty. Tied weights are deduped by storage pointer; accelerate meta placeholders are sized from their dtype and flagged as declared, keeping the placement-independence the field already documents; a diffusers pipeline is walked through its components, since it is not an nn.Module. Context is costed properly: KV from the KEY/VALUE head count, not the attention head count (4x on a GQA model), with sliding-window layers sized to their window. But a computed KV is only a fallback and says so -- this codebase already learned that "predicting KV from GGUF metadata is architecture-specific ... and was off by 5x in BOTH directions", so measured_kv_gb for the context in question always wins. A GGUF, which no walk can reach, falls back to its file size; it is loaded as stored, so the file IS the weights. 2. A failed load overwrote the good measurement ----------------------------------------------- Device map: cpu: 1 layers Measured footprint: 4.50 GB total Saved measured_vram_gb=4.505 (force_vram_update) generation FAILED: StopIteration interacts badly with generators Full-GPU load looks viable (4.5 GB peak need) <- 4 seconds later A load that placed one layer and raised still reached the recorder, and force_vram_update wrote it to disk as truth. Now a measurement that is implausible on its own (~0 GB) or that collapses to under half of what the SAME quantization signature previously measured is treated as a partial load and discarded. Wan2.2's genuine 40.04 -> 11.73 GB drop is let through because it arrived WITH a signature change, which is exactly what tells the two apart. The rejection names which of the two checks fired. The ratio guard alone was not enough. Walking a pipeline and finding that NO component holds any weights is a failed load outright, and there is nothing to fall back to -- a size taken from process memory measures everything except the model in question. That case is now refused before any estimate is considered, and the refusal is recorded in the history with its reason rather than left looking like a successful load. The hole showed up as one of the new tests passing alone and failing in the full suite: the fallback delta grows with whatever else the process has loaded, so the figure it invented depended on test order. 3. The teardown-leak breaker never looked at host RAM ----------------------------------------------------- It registers under `if _tens.is_cuda` and sweeps under `_obj.is_cuda`. A pipeline loaded with enable_model_cpu_offload keeps its weights in HOST RAM, and 17 of 21 video loads took that path, so none of those weights were tracked and nothing looked for them. With nothing loaded and 482 MiB of VRAM in use: coderai-nvidia 33.14 GB PSS_Anon 30.85 GB anon in 6588 mappings; 6565 of them under 100 MB (20.25 GB) Thousands of mid-sized allocations is retained tensors, not allocator slack -- which is why gc and malloc_trim ran every 15 s for hours with the figure never moving, and why the ladder then evicted healthy models to chase it: RAM eviction: unloading 'video:Lightricks/LTX-2.5-Diffusers' to free host RAM (RSS 40.0 GB > cap target 36.5 GB) codai/models/teardown.py adds the host arm, by weakref rather than by storage pointer: a freed host address is readily handed back out by malloc, so a pointer match could null a live tensor that by then belongs to another model. A referent still alive after teardown IS this pipeline's leak, identity-exact. It runs before the allocator trim, since trimming while the weights are still referenced returns nothing. 4. The RAM cap counted shared pages and page cache -------------------------------------------------- codai/models/proc_mem.py reads Pss_Anon + Pss_Shmem from smaps_rollup, which has already divided shared pages by their sharers and excludes reclaimable page cache. Measured correction on a 5-engine host: 40.06 GB (RSS) -> 37.91 GB (PSS). Modest -- the leak above was the real problem -- but a cap should chase memory that freeing an allocation can actually release. Falls back to the old reading rather than to 0.0, which would read as "no memory in use". 5. A model nobody used stayed loaded ------------------------------------ Ondemand mode - keeping 'audio_gen:ACE-Step' in VRAM alongside new model It held 7.9 GB of a 3090 and 2.1 GB of host RAM for hours after its last request. offload.idle_unload_seconds (default 1800) unloads a model untouched that long. Deliberately generous: reloads are the expensive event -- that day spent 2h04m of 21h30m inside loads -- so a short timeout would trade a VRAM saving nobody asked for against a reload everybody waits on. A key that was never stamped is left alone rather than assumed idle. It runs before the leak-watch gate, so turning that off does not silently disable this. 6. The log left no room for the history anyone needed ----------------------------------------------------- 32,633 of 33,009 requests were embeddings, and one engine produced 426,682 of the 548,687 lines -- 78% -- filling the 2,000-line view in about four minutes. The per-item endpoints are now summarised to one line (model, count, shape) instead of dumped, which also stops a caller's content being written to a shared log. Prompts are deliberately NOT truncated: a 4,000-character script prompt is what identified the music defect below. GET /v1/models/history replaces inferring swaps from a log that rotates: time, event, model, engine, plus seconds, vram_gb, offloaded_gb, strategy for a load and reason, freed_gb for an eviction. Eviction reasons are passed at the call sites. The front merges across engines, because a swap is a cross-engine event -- one engine evicts so a co-located one can load -- so one engine's copy is half the story. Also: ACE-Step was asked for a song with no words ------------------------------------------------- Motus sends a music_prompt and no lyrics, and empty lyrics do not mean instrumental: the pipeline then conditions on a single padding token (lyric_token_idx = [0], lyric_mask = [0]) and the model still tries to sing, which comes out as babbling over the backing track. ACE-Step's own UI documents the way to ask -- "Use [instrumental] or [inst] to generate instrumental music" -- and every genre preset it ships ends in either "male vocals, raw vocals" or "instrumental". Both are now sent when no lyrics are given. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
-
Stefy Lanza (nextime / spora ) authored
`_last_used` is stamped in exactly one place: the `active_in_vram` setter, whose comment calls it the "single chokepoint for 'this model was just used'". `add_model()` goes through it. Seven sites in video.py and images.py wrote `multi_model_manager.models[key] = obj` directly and did not, so for every video pipeline and four image models: - `_lru_order()` sorts by `_last_used.get(k, 0.0)`, so the key sorted FIRST and was evicted ahead of genuinely older models -- permanently, however recently it had been used; - the first eviction pass skips `key == self.active_in_vram`, which an unstamped key never matches, so that protection never applied either. Both ladders iterate `_lru_order()`: the VRAM one and the cross-engine `cosite_evict`. Observed on a production run -- a co-located engine needed room and released the freshly-loaded video pipeline before a TTS model that was older but had gone through `add_model`: [cosite-evict] releasing 'video:Wan-AI/Wan2.2-TI2V-5B-Diffusers' ... [cosite-evict] releasing 'tts:Kokoro-82M' ... Also disambiguate the evict-debug line that led here. It read `getattr(model_obj, 'backend', '∅')`, which returns the default both when the object has no such attribute and when `model_obj` is None -- so a healthy diffusers pipeline (which never carries `.backend`) and a key that had already vanished from `self.models` printed an identical `backend=missing`. Only the latter is a fault. Name the two apart, report the object type, and compare the sentinel by identity rather than `==`, which a tensor-bearing object need not answer with a bool. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The release added in 9715d200 ran after _free_pipeline_vram, which is too late: that function ends in the teardown-leak breaker, which walks live CUDA tensors and nulls whatever still references them. By the time the release ran the components had already been stripped, so it found nothing on the GPU and printed nothing, while the breaker reported the full 16.9 GB of this pipeline's tensors survived teardown; nulled 7581 external reference(s) exactly as before the fix. Verified live 2026-10-29: the new line never appeared and the figure was unchanged. The ordering is not cosmetic, because what the breaker nulls is the modules' own _parameters. It frees the VRAM and leaves the injected components gutted -- and those components are reused by the very next attempt. accelerate's offload hook then does current_device = next(module.parameters()).device on a module with no parameters left, which raises StopIteration, which in a worker thread becomes "StopIteration interacts badly with generators" and fails the generation. Measured: the run whose resident attempt failed died that way at encode_prompt, while the run that skipped the resident attempt (and so never ran the breaker) generated normally on the same pipeline. So the leak, the gutted components and that crash are one bug seen at three removes. Releasing first keeps the components intact, leaves the breaker with nothing to strip, and the retry gets working weights. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two bugs that only bite when something else has already gone wrong, so they turned a recoverable load into a failed one. A fully quantized module could not say what dtype it was. transformers has next(param.dtype for param in self.parameters() if param.is_floating_point()) and a bnb 4-bit module has NO floating-point parameters -- Params4bit are uint8 -- so the generator is empty and next() raises StopIteration. diffusers' DiffusionPipeline.to() reads module.dtype unguarded while deciding whether to warn about fp16-on-cpu, so it escapes enable_model_cpu_offload(); and because loads run in a worker thread, Python turns it into RuntimeError: StopIteration interacts badly with generators and cannot be raised into a Future failing the whole load. Measured on LTX-2.5, whose text encoder is 4-bit: the resident attempt did not fit, the fallback to model CPU offload hit this, and the load restarted from scratch. The recovery path broke exactly when it was needed, and it would do so for any pipeline with an all-4-bit component. The property asks what dtype a module is, and a quantized module has a good answer: the dtype of its storage. The shim falls back to the first parameter of any kind, then to float32, and only when the original raises -- a module with floating-point parameters is answered exactly as before. Guarded at every enable_model_cpu_offload() call site, with a test that fails if one is added without a guard. The injected components kept the GPU between attempts. They are pre-built objects held in a local dict and handed to each attempt through _with_quant, so freeing the pipeline frees nothing: the dict is still a strong reference and whatever the failed attempt moved onto the card stays there. [video] teardown-leak breaker: 16.9 GB of this pipeline's tensors survived teardown; nulled 7581 external reference(s); holders: ["2.01 GB dict['weight', 'bias']", ...] The holders are the modules' own _parameters dicts -- the modules outlived the pipeline. The breaker got the VRAM back by stripping them field by field, which works, but it guts components the next attempt is about to reuse. They are now moved to the CPU at both resident-failure exits: moved, not dropped, because the retry needs these same components, which is the whole point of injecting them. The breaker stays as the backstop for a component that refuses .to('cpu'). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Letting a sized-to-fit cache hit try the GPU (1f42d08c ) is right for a model that fits and expensive for one that does not. Measured on LTX-2.5: 04:55:07 RESIDENT all-GPU load (sized to fit) 04:56:53 [resident] components left off-GPU: transformer(CUDA OOM), vocoder(CUDA OOM), prompt_enhancer(CUDA OOM) 05:11:10 ✓ loaded — strategy: model CPU offload 05:11:12 23.28 GB total, of which 21.32 GB offloaded to host RAM Sixteen minutes to reach exactly the placement that would have been chosen immediately, and it would have paid that on every load. The sizing was not wrong so much as answering a different question: it knows what the weights weigh (22.1 GB against 25.0 GB free), while resident needs every component -- transformer, vocoder, VAE, text encoder, prompt_enhancer -- plus the activation peak, all at once. The real footprint turned out to be 23.28 GB. Only an attempt can answer that, so the answer is recorded as a system-owned learned field beside measured_n_gpu_layers, which exists for the same reason: stop repeating an expensive discovery every load. Both outcomes are recorded, not just the failure. Writing only failures would re-prove a model that fits on every load; writing only successes would never stop the expensive retry. And it is not a one-way door -- a later success clears an earlier failure, because a bigger card, a smaller quantisation or fewer co-resident models all change the answer. Unknown still means try. A model never attempted, or a card that has since changed, gets its chance at the GPU rather than being written off by default. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A production run put 93% of a video model on the CPU beside an idle GPU, nine times. The ladder was not at fault -- it evicted an active model, then a keep-resident one, reached 24.9 GB free and concluded: Full-GPU load looks viable (22.1 GB peak need, 24.9 GB free) — using full GPU and two minutes later the same load reported: [pipeline-cache] injected cached components → forcing hook-based 'model' CPU offload Video load strategy: model CPU offload [after load] VRAM 0.4/25.3 GB (24.9 GB free) footprint 22.11 GB total, of which 20.47 GB was offloaded to host RAM The sizing expressed "use the whole card" by leaving `offload` as None, and None also means "nobody chose". The cached-components path could not tell the two apart, so it fell through to its own default -- 'model' CPU offload, which is there for a dual-expert Wan2.2 whose two ~10 GB experts genuinely do not fit in 24 GB, and which was then applied to every cached pipeline whatever its size. It did not heal either: a pipeline loaded under accelerate offload is never re-cached ("cache must be built from a pre-offload pipeline"), so once a model had been cached, every later load was forced onto the CPU and no clean copy could replace it. The verdict now travels with the request. A cache hit that was sized to fit loads resident; the dual-expert default still applies when it does not fit, and video_resident_experts still forces it on. _full_gpu_viable is assigned unconditionally. Written where the sizing happens it would have lived inside `if _offload is None:` → `try:` → `if cuda.is_available():`, so a per-model strategy, a CPU-only host or any exception there would raise NameError and break EVERY video request -- a much worse failure than the one being fixed. The test asserts the assignment DOMINATES the read, not merely that it precedes it, and fails if it is nested back inside a branch. Co-Authored-By:
-
- Oct 28, 2026
-
-
Stefy Lanza (nextime / spora ) authored
The sequential text-encoder offload reported itself "not enabled" on a card that plainly had `sequential_text_encoder = True`. Four defects, stacked so each one hid the next; the first is much wider than the feature that exposed it. _resolve_config_key could not resolve ANY non-text model by alias. Typed entries are keyed "<kind>:<name>", so the exact-id and alias-map lookups both miss, and the basename fallback misses too: "video:LTX-2.3-PinkCherry" has no slash, so basename() hands back the whole prefixed string. Every per-model image / video / audio / tts / embedding setting was therefore replaced by the global default, silently, for any model addressed by its alias -- which is the only name that distinguishes two configs sharing one checkpoint path. The resolver now tries the typed keys, ordered by the types the model is actually registered as. images.py (6 sites) and tts.py looked their config up with a raw config.get(model_key) and had the same bug; they resolve now, with a test that fails if a raw one reappears. The flag itself lived only in _raw_cfg, where a consumer reading config['sequential_text_encoder'] never saw it. Promoted to a top-level runtime kwarg, as gpu_split beside it already documents. With the gate finally firing, the render failed: "Must provide `prompt_attention_mask` when specifying `prompt_embeds`". encode_prompt's return order had been guessed. Read off the installed sources, LTX-2 returns (embeds, mask, neg_embeds, neg_mask) and Krea 2 returns (embeds, mask) -- the mask comes SECOND, and the old code read it third, filing negative embeddings as the attention mask. Naming is now done by classifying the values (a mask is an integer tensor, embeddings are floating) instead of by position. That rewrite then dropped Krea 2's negative prompt, which would have been worse than the failure it replaced: Krea 2's __call__ encodes "" itself when handed no negative, against an encoder this function has just moved to the CPU. The negative is always encoded up front for those families now. Hardened for families nobody has tried: an SDXL-shaped pipeline returns four float tensors including the pooled pair, and naming only two does not degrade quietly -- the pipeline rejects incomplete embeddings and the request fails. If everything encode_prompt returned cannot be named, the optimisation is abandoned for that request and says so. Measured on LTX-2.3-PinkCherry (480x480, 25 frames, 8 steps, RTX 3090): without 369.4 s peak 23718 MiB 91% util with 345.3 s peak 24098 MiB 92% util 7.3 GiB freed Read as headroom, not speed: the card is 24 GiB and the denoiser fills what it is given, so peak sits at the ceiling either way and the 24 s is one sample. The 7.3 GiB is what stops these renders being pushed to CPU offload, where the same job took tens of minutes. Also here: a clip-audio scene no longer needs an explicit duration (B1 said the duration comes from the clip, and validation refused exactly the request the feature exists for), and _free now reports per component whether it moved, was already off the GPU, or REFUSED the move -- a bnb-quantized encoder can refuse, and all three used to arrive as "nothing was freed". test_sequential_text_encoder's fakes returned bare strings in the wrong tuple order, which is why the bug shipped: the test encoded the mistaken belief rather than the pipeline's contract. They carry dtypes and the real orders now. test_compose's engine-selection test patches shutil.which as well, because the production image ships wav2lip and sadtalker shims on PATH and the test was asserting the absence of a binary the real deployment has. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A text encoder runs ONCE per request, before the first denoising step, and is dead weight for every step after -- but it stays resident and keeps its VRAM. Measured on the cached snapshots, that is not a rounding error: Z-Image's encoder is 7.5 GiB of a 19.2 GiB pipeline, and LTX-2.3's 4-bit Gemma-3-12B is ~6.5 GiB in front of a 15 GiB transformer. PinkCherry declaring exactly 24.0 GB on a 24 GiB card is that sum, and it is why every LTX render was pushed to CPU offload. Encoding first and freeing the encoder hands that VRAM to the denoiser, so a model that "did not fit" fits. This is the sequential text-encoder offload the requirements asked for; it existed only as a side effect before, because diffusers' enable_model_cpu_offload happens to move each component out after use -- so the OFFLOADED path had it and the resident path did not. The embedding argument names differ per family and there is no common base class to ask, so they are read from the pipeline's own signature: Krea 2 takes prompt_embeds + prompt_embeds_mask and needs a second call for the negative, LTX-2 takes prompt_attention_mask and encodes both at once. A family that takes no embeddings at all is left alone rather than having its encoder freed with no way to pass the prompt back. Opt-in per model and settable from the model page, because freeing the encoder costs a host-RAM copy and only pays where the card is tight. The encoder is moved to CPU rather than deleted: the pipeline is cached and the next request needs it again. Any failure returns the request untouched -- a VRAM optimisation must never be the reason a generation fails, and a test holds that. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two bugs of the same shape: a signal that already existed and was not consulted. Same-path siblings. Two cards can share one weights path and differ only in settings -- Wan2.2-I2V-A14B-Lightning and the plain Wan2.2-I2V-A14B both point at Wan-AI/Wan2.2-I2V-A14B-Diffusers, declaring 22 GB and 151 GB. The manager has disambiguated exactly this since the lisa/lisa-32k case, through `requested_id`, and NEITHER placement caller passed it. So the Lightning card was sized by its 151 GB sibling, read as ~127 GB over free VRAM -- far outside the borderline band -- and went straight to CPU offload without trying the card, where it computed at 0% GPU for 26 minutes holding 37 GB of host RAM. Measured: 97.4 GB without the id, 14.1 GB with it, which is the difference between exile and residency on a 24 GiB card. Quantizing the experts was necessary but not sufficient; the ESTIMATE was still the wrong card's. The leak watcher's load signal. _load_in_progress() consulted only the manager's ready event, which diffusers image/video loads never touch -- they load in codai/api/{images,video}.py. A 12-minute multi-shard load therefore looked idle, its monotonically rising RSS looked exactly like a leak, and the mitigation ladder ran against a load doing nothing wrong ("RSS 19.8/43.0 GB LEAK SUSPECTED", every poll). It now also consults the loading reservations, which cover those paths by construction. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
LTX-2.x is the one video family here with an audio VAE and a vocoder, so its output is LTX2PipelineOutput(frames=..., audio=...) -- audio generated jointly with the frames and therefore actually in step with them. Only `frames` was read, so every LTX render came out silent and the only way to get sound was to generate some separately with another model and mux that on top, which does not match the picture. The generated track is now muxed in BEFORE the other post-processing steps, because every one of them re-encodes the frames and the track belongs to these exact ones. `add_audio` becomes tri-state so "say nothing" and "say no" differ: unset keeps what the model generated, false gives a silent video, true runs the older separate-generation path. It defaulted to False, which cannot be told from an explicit false -- so a straight default would have dropped the soundtrack. None is falsy everywhere False was, so existing callers are unaffected. Found while testing it: _has_audio tried `import av` under a bare `except: return False`, and PyAV is in no main requirements file -- only the isolated LongCat venv's. On an install without it EVERY file answered "no audio", and because it gates _mux_original_audio, a clip's own sound was silently dropped from every upscale and interpolate. The audio was there, nothing could see it, and nothing said so. ffprobe cannot be missing the way PyAV can, since ffmpeg is what does the muxing, so it is the fallback, and a failure of both is logged instead of reported as silence. Also: one of my own tests asserted a minimum number of ffmpeg progress reports. `-progress` emits on a time interval, so a small encode can finish having emitted a single block -- it passed alone and failed under the full suite. It now asserts what is guaranteed: frames counted, counted upward. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
13. A request may name itself (`request_id` in the body, or an X-Request-Id header) and poll /v1/{images,video,audio}/progress?request_id=... for its OWN steps. One global slot per media type meant two callers could not tell their progress apart, so a client stopped reading them altogether and showed no step progress at all while the server was drawing. The name is used as the TASK id, so this is a VIEW over the existing task registry -- which already tracks step/total/status per task under a lock and prunes finished ones -- rather than a second place progress is written. Claimed once, because a request registers a short "loading" task as well and both adopting the id would serve the client whichever happened to be last. Unknown id = 404, so a client stops polling on a definite answer instead of reading somebody else's progress. 13b. The encode reports frames written (`encoding 1200/2400 frames`) across the band it owns, so the longest step of a render stops looking like a stall; and the job's own log, kept all along but unreachable, is served as a tail in the status document plus GET /v1/video/compose/{id}/log?since=N for the whole thing incrementally. The record holds the live list, not a snapshot. A4. 72s per 768x1344 still -> 21.4s, measured on an idle 3090. The card carried `offload_strategy: "model"`, which is the model page's DEFAULT, so enable_model_cpu_offload() ran on a model that fits a 23.56 GB card whole and paid PCIe for it on every request. `auto` loads on GPU and falls back to offload only on OOM. The doc's other hypothesis is ruled out rather than assumed: the live progress endpoint reports total=10, so the card's step count IS applied, and an explicit steps=10 takes the same 21.2s. Also: `models.civitai_cache_dir`, configured like hf_cache_dir and gguf_cache_dir because it is the same kind of thing. Unset, these multi-gigabyte checkpoints land beside the other caches -- on this host a 140 GB volume, which a 13.5 GB checkpoint took to 99% until the next model could not write even its config fetch ("No space left on device"). That is what was actually stopping Z-Image-Turbo-IntoRealism from loading at all. Co-Authored-By:
-
- Oct 27, 2026
-
-
Stefy Lanza (nextime / spora ) authored
video.py already knew the hazard and wrote it down: accelerate's device_map only dispatches components it LOADS, so one handed to from_pretrained keeps whatever device it is on with no offload hook, and the forward pass hits "index on cuda:0, weights on cpu". It acted on that only for pipeline-cache injections, while a GGUF component arrives exactly the same way. LTX-2.5 therefore failed under every strategy it tried -- auto, balanced, model and pure-disk, three restarts and half an hour each -- always with Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu and video_resident_experts, the flag that exists for precisely this, was never consulted because the gate in front of it was false. An explicit offload_strategy is still respected; only the strategies that let accelerate place components itself are overridden. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
LTX-2 masks its text conditioning. The model loaded cleanly and died at the first denoising step: ValueError: `attn_mask` is not supported for sage attention so `auto` choosing sage there is not a slower path, it is a broken one. The first version of this guard moved it to flash instead, and it failed again with the same sentence and a different name: ValueError: `attn_mask` is not supported for flash-attn 2. NEITHER fast backend accepts a mask, so falling back from one to the other is not a fix. For these families the only working backend is the default SDPA, and the guard now keeps it. An operator who names sage or flash explicitly is still obeyed -- the override exists so they can force a backend, and silently ignoring it would be a different kind of wrong from silently choosing a broken one. They get the error, which is theirs to get. Wan and the LTX 0.9 family are unaffected and still take the fast backends; tests pin both directions. Suite: 15 failed, 5258 passed -- the same 15 as the baseline. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
9ae240b0 committed its tests and its documentation and none of its implementation: four files instead of twelve. The `git add` that should have staged the rest named a pathspec for requirements-acestep.txt, which did not exist, so git refused the whole invocation and I did not check its output before committing. The message described work that was not in the tree, and the suite at that commit could not have passed. This is that work: * video.py — LTX-2.x gets its own pipeline classes. This diffusers has LTX2Pipeline, LTX2ImageToVideoPipeline, a video AND an audio VAE, but only the LTX 0.9 family was mapped, so every LTX-2 entry in models.json sat parked: the name matched and the class did not. Split on the version in the name, since both families are just called "LTX" and a fine-tune keeps its base version (10Eros is LTX2.3, PinkCherry is LTX23). * hf_loading.py — `civitai:<modelVersionId>` references, and a GGUF component that fails to load now fails the load instead of being skipped. A skipped component is simply absent from the dict, so the base pipeline loaded its own full-precision copy: the operator asked for a 14 GB quantized transformer and got a 38 GB bf16 one, while the VRAM estimate still said 14 GB and the planner therefore freed about a third of what the load needed. * presenter.py — LatentSync and MuseTalk as first-class engines, ahead of wav2lip for a clip (wav2lip works at 96x96 and upscales, which is the soft mouth on a 1080p reel). Both are video-driven, so sadtalker still leads for a still portrait. Recognised from a shim on PATH or a managed install whose weights are already present — never "could be downloaded", because a compose job is the wrong place to fetch 5 GB. * cache/__init__.py + app.py — the loop stall. scan_cache_dir is not a listing: it stats every blob of every revision of every repo, measured at 27.3 SECONDS on this machine's 900 GB cache, and it was called from a per-model VRAM estimate, a GGUF path lookup and a POLLED endpoint, all on the event loop. 6631 stalls over 0.5s, worst 13.0s, serving nothing. Both halves were needed: memoised for 30s (short enough that a model downloaded while the server runs still becomes visible) and the poll moved to asyncio.to_thread, since memoising alone still puts the first 27-second call on the loop. * manager.py — /v1/models reads `modes` from a video entry's long-standing `video_subtypes` as well as the newer spellings. * requirements-acestep.txt — the file whose absence caused this. It is marked NOT YET INSTALLED, because it is: the install was refused as external code, nothing in it has been run, and the audio_gen loader refuses an ACE-Step model explicitly rather than mis-loading it as a MusicGen. Suite: 15 failed, 5194 passed — the same 15 as the baseline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
normalise a language before transcribing, stop inflating weights that are already quantized, and never answer 200 with nothing (v0.5.9) Five bugs and five features from a client report. The bugs share a shape: the server held a belief about a model that nothing checked against the model. A1. A TTS voice is chosen for an accent, so the language a compose render carries is "en-us" / "pt-br". Passed to transformers' Whisper -- which accepts only the English NAME -- it raised "Unsupported language: en-us" and 500'd every captioned scene of every render. Reproduced verbatim against the running server before the fix. New codai/api/lang_codes.py normalises once, per backend: the NAME for the transformers pipeline, the ISO code for whisper.cpp, Canary and faster-whisper, and None (auto-detect) for a tag that maps to nothing -- refusing to transcribe because the caller named a dialect is never the better outcome. Language VALIDATION now compares by meaning, so a model declaring "english" no longer refuses "en-gb", and compose's model picker stops disqualifying the only model that can time words. A3. FLUX.2 (dev/klein/pro) is a unified generate-and-edit model whose name carries neither "edit" nor "kontext", so it fell through to the generation branch and advertised plain img2img -- telling clients to show a `strength` slider for a pipeline that has no such argument. Confirmed live: ['image_generation','image_to_image','inpainting']. diffusers 0.37 agrees it is an instruction edit (Flux2Pipeline takes `image`, not `strength`), which is what the runtime already decided on; only the advertised list disagreed. A4. The estimator normalises a disk size to the load precision. A GGUF was special-cased to 1.0 because its bytes ARE its footprint; an HF checkpoint published pre-quantized is the same case by another route and was not. Z-Image-Turbo-unsloth-bnb-4bit: 16.3 GB of 4-bit weights, 1.70x factor, estimated 30.03 GB against a 24 GB card. Nothing can free 30 of 24, so every still ran partly on the CPU -- ~72 s for one image from a few-step turbo model. A5. Qwen3.5-9B answered content "" with HTTP 200 for four weeks. All three rungs of the generation ladder raised ("Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)") and the last one assigned generated_text = "" and returned normally. Now raises. The Triton error is a quantized model whose weights were CPU-offloaded: such a load succeeds and then fails every request, so it is refused at load time and a quantized model tries full-GPU whatever the estimate says. That also breaks a loop -- measured_vram_gb = GPU + offloaded + reserve means one offloaded load writes a number bigger than the card, and the next load reads it and offloads again. B1. A scene can carry its clip's own sound: visual.keep_audio, scene.audio_mix (voice|clip|both) and clip_volume. "clip" makes the scene as long as the clip and skips the TTS call; "both" ducks the clip under the voice. Folded into the narration track, so the music bed ducks under it like a voice. B2. Two characters talking: scene.speakers, each line in its own voice, each character lip-synced in their own column (or the frame cutting to whoever speaks). One sync run per CHARACTER, not per line -- each gets a track holding only their own lines, so the mouth moves on their words and stays closed while the other talks. Captions never mix two speakers and carry the attribution into the .srt/.vtt. B3. characters: ["alice","bob"] on images and video. The gap was never the field: IP-Adapter conditions on the MEAN of its references, so a second character's pictures produced one person who looked like neither. Reference sets are kept apart and handed to a model that takes several (FLUX.2, Qwen-Image-Edit); a model that cannot do it says so instead of blending. B4. POST /v1/moderation -- nsfw, faces, age, face_match over images or sampled video frames. It fails CLOSED: a missing model is a 503 naming it, not a result with the check quietly absent, because a caller cannot tell a missing check from a passed one. minor_detected is nullable (explicit content with no face cannot be cleared on age) and keys on the LOW end of an age interval, not a point estimate from a model with a multi-year error. C5. Per-model defaults (steps, guidance, sampler) that fill in only what the caller omitted, published in /v1/models alongside modes and licence -- so a client stops keeping a table keyed on the model NAME and an 8-step model stops being run for 30. A sampler switcher that refuses to cross the flow-matching boundary (a FLUX-family pipeline handed a DDIM produces noise, not a worse picture). Stable Audio gets its own pipeline: it was mapped to AudioLDM2, so a configured entry could only fail at load. Two of my own regressions, caught by the suite's guard tests and fixed here: /v1/moderation was unmapped in the remote gateway, and the moderation availability probe LOADED each model -- which turned the /v1/models listing into a 350 MB download on the event loop. Suite: 15 failed, 5146 passed -- the same 15 as the baseline. Co-Authored-By:
-
- Oct 24, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Four ways an edit could answer 200 while quietly doing the wrong thing. The enhancement paths decode a clip to frames and encode a new video, so the audio was left behind: a clip that arrived with speech came back silent. Only the ffmpeg minterpolate fallback -- the path nobody takes by default -- ever kept it. _mux_original_audio() puts the track back on all three in-process paths, stream-copied where MP4 accepts the codec and re-encoded to AAC where it does not. Both paths already preserve the source duration, so nothing is stretched. A silent source, or a remux PyAV cannot do, returns the frames unchanged rather than losing a finished GPU job. The image and video editing endpoints took base64 (and, for video, an http URL), so a caller that had already uploaded a file had to re-send it on every edit -- megabytes a call for a file on the same disk. They now resolve through the compose resolver, so a /v1/files path or a sha256: upload id works everywhere, and our own files are read off disk instead of fetched back over a loopback that would need a token. Responses carry the relative path beside the url, as compose does, so a caller need not depend on the public hostname. Routing through that resolver would have tightened what callers may send: it validates base64 strictly, which rejects the 76-column wrapping that MIME, the base64 tool and encodebytes all produce. Whitespace carries no meaning in base64, so it is now stripped rather than refused. /v1/images/edits defaulted size to the string "1024x1024", so omitting it squashed every photo that was not square. Omitted now keeps the source's shape at the ~1 MP a diffusion model wants, on the 64-grid it needs. /v1/images/upscale fell through to PIL LANCZOS for an unrecognised model and still answered 200, charging for an AI upscale that never ran and returning something merely soft. That is a 400 naming the models that would work. Co-Authored-By:
-
- Oct 17, 2026
-
-
Stefy Lanza (nextime / spora ) authored
0.4.16 fixed one blocking call on the loop by finding it the hard way: a 5,000-document run rented 127 pods and completed nothing for ten minutes, and the only honest signal in the system was that /v1/runpod/status had stopped answering. Nothing in the process said "the loop was blocked", though it had been, for minutes at a time. Both halves of that are addressed here. MEASURED: codai/api/loop_watch.py samples the loop's own behaviour -- a sleep that asks for 0.5 s and returns at 4 s says the loop was busy elsewhere for 3.5 s and served NOTHING in that window. Installed in both the engine and the front (the front is the process the client talks to, so a stall there is the one it experiences), logged per stall with its duration, and exported on /metrics as coderai_loop_lag_seconds, _lag_worst_seconds, _stalls_total and _blocked_seconds_total. It cannot name the culprit, but it turns "the server is slow" into "the server was not running", which is a different investigation. CHECKED: tests/test_no_blocking_calls_on_the_loop.py walks every coroutine in codai/ for calls that block the calling thread. What makes it precise is that it does not look inside a nested def or lambda -- `to_thread(lambda: f())` and `run_on_pod_pool(f, x)` are both correct, so only a call reached without crossing a function boundary is on the loop. Its own ten tests prove it catches what it claims and ignores os.path.join, asyncio primitives and coroutines handed to ensure_future. It found 20 more, all fixed: * ocr/gateway.py: `placement.choose()` in async chat_completions -- the SAME 1200 s pool acquire as 0.4.16, on the chat path instead of the page path, and with no retry loop above it to make the wait visible. * api/ocr.py: a sync urlopen of a caller-supplied URL, so an OCR step was as slow as whoever served that URL, for every request in the process. * eleven `subprocess.run()` ffmpeg/ffprobe calls across faceswap, video, pipelines, voice_clone, custom_pipelines and the outfit video path. A media encode is seconds to minutes, and all of it was everyone's wait. * two HuggingFace API fetches in the admin routes. * two `thread.join()`s on the CUDA streaming path, and a `time.sleep(0.1)` per generated image. The allowlist in that test is empty and should stay empty: an entry in it is a request that cannot be served while the named call runs. Also tools/loadprobe.py, for the outside view: /healthz does no work and returns the instant the loop is free, so its latency IS loop lag as a client sees it. p99 healthz flat while documents crawl is a capacity problem; p99 healthz in seconds is not, and no extra capacity would have helped. Co-Authored-By:
-
- Oct 10, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Township's clip lengths were tuned when the only video model could hold ~81 frames in one call, so a 70-second match became a dozen short cuts. LongCat was pretrained on continuation and the server renders a long shot in segments, so the same match wants a handful of long takes — and the frame numbers that suit one are wrong for the other by a factor of four. Only the two starter templates knew that, and only because someone typed it in. Worse, nothing snapped a picked count onto what the model can emit: random.randint handed the server lengths like 250, which is not 4n+1 for any VAE and not on LongCat's 93+80k segment grid, so the server rounded and the plan's own numbers became fiction. On LongCat that also silently costs block-sparse attention, because every point of 93+80k is a 16n+13 count BSA can tile and nothing between them is. Server — codai/models/video_geometry.py is the geometry once: native rate, the legal frame grid, how much ONE render holds, whether continuation is native, and the side multiple below which the fast attention path is lost. Published per model on /v1/models (ModelInfo.video) and used by the render path itself, so what the server advertises and what it rounds to cannot drift. The numbers were spread across three modules that could not tell a client; the literals here are pinned to each source by tests, since the LongCat and H3 workers run in 3.10 venvs this cannot import. models.json overrides every field. Township — the five frame fields now default to 0 = AUTO: derive the budget from the model's geometry and this tool's seconds band, then snap both ends and every pick onto the model's grid. Seconds, not frames, because clip count falls out of long_target / (frames/fps). An explicit number still wins, so a saved config keeps exactly the budget it had. Entrances and the face-off get their own shorter band instead of borrowing the fight budget (on LongCat that made every entrance an 11-22s take). Outcome shares are snapped per shot and the total rewritten, the per-render chunk comes from the model (Wan keeps its conservative 50), and the chaining decision reads the published continuation rather than matching the model's name. At the defaults a 70s match goes from 9-11 Wan cuts to 4-6 LongCat takes, each one request with native continuation, with BSA still on. 1607 tests pass; the 17 failures and the test_broker_protocol hang are pre-existing and unrelated (verified with the diff stashed). Co-Authored-By:
-
- Oct 08, 2026
-
-
Stefy Lanza (nextime / spora ) authored
A 304-frame request sat at "generating 0/16 (0%)" for its whole nine-minute first segment while the service's log showed Denoising: 15/16. Two defects. The service set _progress["step"] = 0 at the start of every segment and never again, so the step count existed only inside the vendored pipeline's tqdm(total=len(timesteps), desc="Denoising") -- which takes no callback. Nothing reached GET /progress; the poller in video.py faithfully reported the 0 it was handed. Each pipeline module did `from tqdm import tqdm`, so the bar is instrumented by rebinding the name on the MODULE (patching tqdm.tqdm would be too late), and only bars labelled "Denoising" are tracked so a checkpoint-shard bar cannot move a request's progress. The totals also counted ONE segment: a four-segment render would have run 0->16 four times against a denominator of 16. `step` now carries the completed segments' offset so it counts the stage end to end, and the bar spans steps x segments -- with refinement excluded, being a single pass over the finished video. A request that asked for a duration rather than a segment count only learns the real count from the service, so the denominator is corrected on the first report, through a setter that does not restart the bar and lose the elapsed time the rate is derived from. Co-Authored-By:
-
- Oct 07, 2026
-
-
Stefy Lanza (nextime / spora ) authored
File "codai/api/video.py", line 4176, in _generate_longcat payload["audio"] = _as_b64(_audio) NameError: name '_as_b64' is not defined _as_b64 was defined INSIDE _generate_h3, MiniMax H3's handler, and _generate_longcat referenced it across that scope. So every LongCat path that carries binary input raised NameError before reaching the worker: the avatar audio track (at2v), the reference image (ai2v and i2v), and continuation frames. Text-to-video is the only path that never touches it, which is why t2v was the only one that ever worked. It cost a 20-minute wait for the GPU to find out, because the NameError fires after the model is loaded and the swap gate has handed over. Now module level, beside the _decode_b64_or_url it wraps. The test walks the AST for a call site in a function that neither defines it nor is nested in one that does, so the same mistake in another backend fails here instead of on a rented GPU. Co-Authored-By:
-
- Oct 06, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Asked to stop an offloaded render, I killed the service process by hand — and had to, because nothing else could. Every diffusers video path calls task_registry.raise_if_cancelled() between denoise steps and shows up on the Tasks page; LongCat was never registered at all. So the LONGEST-running video job in the system was the only one with no stop button: a 93-frame bf16 render offloading across PCIe can go for a very long time. Its poll loop claimed otherwise. The docstring said it would "carry cancellation and the thermal gate across the process boundary" and the body did neither — it only mirrored progress. A comment describing intent rather than behaviour. It is now registered like the others, reports step progress to the task, and polls for cancellation every 2s. Because the work happens in ANOTHER PROCESS, cancelling cannot raise inside it: the service is stopped instead, which is exactly what eviction already does to it, so that path is well exercised. The request then ends 499 rather than returning a partial video as success, and the task is closed on every exit — cancelled, error or done. I also owe a correction: I told the user there was no cancel for /v1/video/generations. There is, for every other model — I searched for a /v1/video/* endpoint and missed the task registry. The gap was LongCat specifically, which is code I wrote. 7 tests. Co-Authored-By:
-
- Oct 05, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Plan item I, the last of the LongCat work. The tool chains clips because the server generates one clip per request and VACE frame-tail conditioning is the only continuation a Wan model has. That machinery exists to fight drift — the dual-expert and VACE-extension notes are both about chained parts going wrong — and every join is a re-encode. LongCat was pretrained on continuation, so: - the client-side split is LIFTED for it (_chunk_max becomes unbounded): a clip's whole frame budget goes in ONE request and the server runs the segment loop. Chaining it from here would reintroduce exactly the drift the chaining was invented to avoid. Wan/VACE keeps the cap it always had; - between DELIBERATE shots the tool still chains, because a cut is a cut — but LongCat is tail-capable too, so it takes the same cond_frames handover. What differs is underneath, not the call; - the tail length follows the model: 13 of 93 frames, which is what LongCat's continuation pretraining used, against VACE's 5. Handing it 5 would condition it on less than it expects. The constant is asserted against tools/longcat_common.py so the two cannot drift apart; - the log says which continuation produced a shot, because diagnosing a seam starts there. Server side, so a client asks for the frames it wants rather than reimplementing the loop: num_frames is the PER-CALL count, and a request for more than one call's worth now derives total_frames and lets the segment loop cover it. An explicit num_segments or duration_seconds still wins. Passing 400 straight through would have asked the pipeline for a 400-frame single call. The uploader and the odds generation are untouched — they are downstream of the frames. 17 tests. No match has been rendered with it: that needs the weights. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Step 6. Read from upstream's avatar demo rather than assumed, because almost nothing carries over from the base model: - avatar is a DIFFERENT pipeline class, LongCatVideoAvatarPipeline, with its own methods generate_at2v / generate_ai2v / generate_avc; - it takes TWO guidance scales — text and audio — where the base pipeline takes one; - its distilled pass is dmd_lora at 8 steps with lora_network_dim=128/alpha=64 and dit.enable_loras(["dmd"]), NOT cfg_step_lora at 16; - use_int8 is a separate load_quantized_dit(subfolder="base_model_int8"), not a dtype; - both INT8 and the DMD LoRA exist only for avatar-1.5, which also REQUIRES distilled sampling; - the audio is librosa.load(sr=16000) then pipe.get_audio_embedding(...), and the encoder differs per family (chinese-wav2vec2-base for Avatar, whisper-large-v3 for Avatar-1.5). Unsupported combinations are refused BEFORE loading, because the alternative is quietly serving someth...
-
Stefy Lanza (nextime / spora ) authored
Step 5. The requirement is "full configuration in the interface", and the trap is silent: api_model_configure REBUILDS the entry from scratch and copies only an explicit whitelist, so a key with a UI control but no whitelist entry is written once and deleted on the next save from the models page. It looks saved until you reload. That is precisely why H3's documented h3_venv / in_process / dtype keys never survive, and it is not inherited here. - `longcat` is now an option in the hard-coded cfg-backend select — without that the backend cannot be pinned from the UI at all, which is the other half of why H3's `backend: "h3"` switch is unreachable. - A cfg-longcat field group, shown for that backend: variant (bf16/fp8/int8/gguf), default quality, offload, segments, conditioning frames, context-parallel GPUs, repo checkout, venv, and the avatar fields (reference frame index, mask range, INT8, distilled sampling, KV-cache offload). Populated on open and read back on save. - All thirteen keys added to the whitelist, with a comment saying what happens without it. - The settings now DO something: the dispatch reads num_segments, num_cond_frames, offload_kv_cache and quality off the entry as defaults, with the request still winning. A knob that is saved but never read is worse than no knob. - docs/longcat-video.md: why the venv exists, the two prerequisites, the download repos and why file_pattern matters (~83 GB), the checkpoint layout, the stages and presets, the segment arithmetic, the VRAM/eviction/segment-boundary behaviour, and an explicit list of what is NOT done — including that no generation has been verified against real weights. 62 tests, parameterised over every field so a future control added without a whitelist entry fails rather than silently losing data. Not in this step: there is no known-model catalogue in the codebase to pre-register the LongCat repos into — adding a model is operator-driven by model_id — so the repos and their file patterns are documented instead of hardcoded. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Step 4. It also CORRECTS step 2, whose generate() was written against an assumed interface. The real one, read from upstream's demo scripts, differs in four ways that all mattered: - there is no single callable: generate_t2v(height, width, …), generate_i2v(image, resolution, …), generate_vc(video, num_cond_frames, use_kv_cache, offload_kv_cache, …) and generate_refine(stage1_video, spatial_refine_only, …) are separate methods, and t2v takes height/width where i2v and vc take a resolution NAME; - each returns [0]-indexed frames normalised to [0,1], not an object with .frames — upstream converts with (tensor * 255).clamp(0,255).to(uint8); - the distilled stage is a use_distill=True FLAG on the same method, not a LoRA to fuse by hand, so _apply_stage_lora is gone; - the 50-step passes take a negative_prompt and the distilled one does not (its guidance is 1.0, so it would do nothing). On top of that: - The segment loop is upstream's: a call emits num_frames but its first num_cond_frames re-render the previous tail, so only the remainder is new and the fresh frames become the next call's conditioning. torch_gc() between segments, as upstream does. - Modes map onto the tasks: t2v, i2v/ti2v -> i2v, extend/vc/v2v -> vc. Continuation reuses the EXISTING cond_frames field, so a client that already chains clips through the VACE path needs no new vocabulary — and gets native pretrained continuation instead, which is what stops the colour drift chained conditioning suffers. - Long video: num_segments or duration_seconds, and the SERVER owns the loop. The response carries the tail a caller needs to continue by hand if it prefers to. - The segment boundary is now a real handover point. POST /yield asks a generation to stop at the next boundary; the releaser calls it before waiting, so an eviction arriving mid-request costs one segment instead of the whole generation, and the request still returns — shorter, with a warning that reaches the caller — rather than failing. Unsupported modes (the audio-driven avatar tasks) are still refused with a 400 rather than downgraded to t2v, which would read as a quality bug rather than a gap. 23 new tests pinning the upstream call contract so it cannot drift back, plus the segment arithmetic and the yield behaviour. Still no real generation: that needs the venv, the repo checkout and the ~83 GB checkpoint. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Step 2: text-to-video through LongCat-Video, single GPU, bf16. The engine exists now; i2v, continuation/long-video and the avatar variants are the next increment and are REFUSED with a 400 rather than silently downgraded. Step 1 missed a prerequisite, fixed here: LongCatVideoPipeline lives in the repo's own `longcat_video` package, not on PyPI (`from longcat_video.pipeline_longcat_video import LongCatVideoPipeline`), and it vendors its own Wan VAE and scheduler. So the repo SOURCE is bundled like Wav2Lip/SadTalker — code only, no weights — and the service prepends it to sys.path. - tools/longcat_common.py: the contracts both sides share, stdlib-only so it runs on 3.10 and 3.13 alike and is loaded BY PATH from each. Checkpoint layout (tokenizer/text_encoder/vae/scheduler/dit + lora/*.safetensors), the stage table, the presets, and the segment arithmetic — 93 frames per call of which 13 re-render the previous tail, so a segment ADDS 80, which is what makes upstream's "11 segments for a 1-minute video" come out at 59.5 s. Getting that wrong silently truncates long output. - tools/longcat_service.py: runs IN the 3.10 venv, serves /health /progress /generate /unload. Fuses the right LoRA per stage (stages 2 and 3 ARE their LoRAs — the distilled pass is 16 steps because of cfg_step_lora), and refuses 'refinement' with no earlier frames instead of generating something else. - codai/api/longcat_worker.py: venv/source resolution, process lifecycle, and VRAM registration through acquire_stt_backend so eviction reclaims it by stopping the subprocess — the only way to be sure a 13.6B DiT's pages are gone. ensure_built REFUSES by default with the exact manual command, because auto_build pulls torch 2.6 plus its CUDA runtime; the venv probe asserts 3.10/2.6/4.41 rather than a bare `import torch`, since the image rewrites pyvenv.cfg and a mis-pointed venv would import the main venv's torch 2.11. service_key includes config_id/variant: sibling configs on one path are DIFFERENT residencies and must not share a service or a measured footprint. - Detection honours the backend pin, then alias/served-name/path. The dispatch branch in video.py unwraps `_raw_cfg` first, because build_runtime_kwargs emits a FIXED key list for video models with no backend, alias or path in it — which is exactly why H3's documented `backend: "h3"` switch is dead. Not inherited. - The worker path reports progress and is cancellable: the service exposes /progress and the engine polls it onto the existing bar. H3's worker path has neither. - VideoGenerationRequest gains `quality` (draft|fast|best) and `stage` (base|distill|refinement), the preset and resumable-per-stage halves of the staging requirement. 31 tests. NOT covered, and so not claimed: a real generation — that needs the venv, the repo checkout and the ~83 GB checkpoint. Co-Authored-By:
-
- Sep 19, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Per-model 'components' (text_encoder / low_noise / vae → cluster node): the prompt is encoded where the text encoder is, the Wan 2.2 low-noise expert takes over at the expert boundary with the latents handed over, the VAE decodes where it lives and returns the finished video. The far side runs the ordinary generation endpoint with a _handoff block, so loading, acceleration and LoRAs happen as usual there; the scheduler is sliced to resume mid-loop (codai/cluster/components.py). Models page block + tests. Co-Authored-By:
-
- Sep 15, 2026
-
-
Stefy Lanza (nextime / spora ) authored
None of these were placement or pod bugs: each image booted, pulled its model and got as far as executing. The remote machinery worked; the payloads did not. voice — torchcodec arrived as a transitive dependency of f5-tts, PyPI served the CUDA-13 build, and it died with 'libnvrtc.so.13: cannot open shared object file' after loading the model. The same class of bug as the torch pin: a CUDA-coupled package escaping the index. It now installs from the CUDA index with the rest, resolving to torchcodec 0.11.1+cu128. audio — 'No module named audiocraft'. audiocraft pins an old torch, so putting it in the image would drag torch backwards and undo the CUDA pin. transformers ships MusicGen itself, so a pod serves it through transformers instead; an install that has audiocraft still uses it. Melody conditioning stays audiocraft-only rather than silently doing something different. video — 'TextToVideoSDPipeline.__call__() got an unexpected keyword argument callback_on_step_end'. Older diffusers pipelines do not take it and raise inside the call, minutes in, with the model loaded and the GPU paid for. The kwarg is now added only when the pipeline's signature accepts it. The try/except that used to wrap it caught nothing: assigning to a dict never raises. This one bites local generations too, not just pods. Co-Authored-By:
-
- Sep 13, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Two costs on every cloned line. F5TTS(device=…) was constructed per request, so the model and its vocoder were loaded from scratch for each line of dialogue — seconds of latency and a fresh VRAM allocation every time. seed-vc next door already keeps a singleton; F5 now does too, rebuilt only if the device changes, and registered with register_external_vram_releaser so the manager can reclaim it like the LoRA base cache and the OCR engines. And nothing capped the reference. F5-TTS prompts on the clip and gains nothing past roughly 15 s, while a saved profile can hold minutes — the character studio uploads 90 s. Both the /v1/audio/clone path and the dialog path now trim to MAX_REF_SECONDS (15, override with CODERAI_F5_MAX_REF_SECONDS), leaving anything already short untouched. Co-Authored-By:
-
- Sep 10, 2026
-
-
Stefy Lanza (nextime / spora ) authored
H3 doesn't need a permanent sidecar. Model work already happens in a supervised, per-GPU engine subprocess with its own CUDA context, VRAM manager and swap gate — so H3 belongs there, registered with the model manager like a Wan or LTX pipeline, and pinned to a CUDA engine. What actually forced the worker was the version floor, not the architecture: diffusers ships H3 as Modular blocks only (0.40+), and this venv is on 0.38. So the route is now chosen by capability, not hardcoded — engine_supports_h3() probes for MiniMaxH3ModularPipeline, and `in_process` in the model config forces either side. The same build works before and after a diffusers upgrade, and the worker stays as the fallback while H3 is unproven (an in-engine OOM or device assert poisons the whole engine's context; a subprocess crash doesn't). _load_h3_pipeline() builds the modular pipeline with ONE workflow (loading with none pulls both 61.7GB partitions), group-offloads the big modules per component since a ModularPipeline has no pipeline-level offload helper, and registers it with record_vram_delta so eviction sees its real footprint. A workflow change unloads first — the two partitions can't be co-resident. The checkpoint contracts now live in tools/h3_common.py, loaded by path from the engine and imported as a sibling module inside the isolated venv, so both routes snap frames, round the canvas, pick the workflow and mux the audio identically. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
MiniMax-H3 is a 33B flow-matching DiT that denoises video and audio in ONE packed sequence. diffusers ships it as Modular blocks only (MiniMaxH3ModularPipeline, 0.40+) — there is no DiffusionPipeline half, so it fits neither _detect_pipeline_class() nor this venv's diffusers 0.38. It runs in an isolated venv, the same way pyannote, NeMo-Canary and vLLM do: tools/h3_service.py generation inside the isolated venv (HTTP) codai/api/h3_worker.py venv bootstrap, process lifecycle, eviction handle requirements-h3.txt the venv's pins (diffusers>=0.40, transformers 5.x) _generate_h3() adapts the request to the checkpoint's contracts instead of forwarding it, and reports every adaptation in the response warnings: guidance is baked in (negative_prompt and guidance_scale dropped), 24 fps is fixed and num_frames snaps up to the next 17n+5 within 5-15s, canvas axes round to 32, and v2v/interp are rejected. character_profiles become native ref2va references (max 9) so H3 skips the identity-keyframe bridge; an explicit keyframe wins, since fl2va and ref2va are separate transformer partitions. The worker picks a workflow per request and never loads without one — that pulls both 61.7GB partitions. The worker registers as an evictable, VRAM-tracked model; eviction stops the subprocess, the only way the transformer's pages actually come back. Character studio: clone the voice off the same footage the face came from (/v1/audio/voices/extract) with `extract --voice`, the new `voice` subcommand or the UI checkbox, then `--say` a line that is lip-synced in that cloned voice. Co-Authored-By:
-
- Sep 09, 2026
-
-
Stefy Lanza (nextime / spora ) authored
The keyframe bridge dispatched its nested image request back through the front via execute_api_request. That deadlocks: by the time the engine handler runs, the front's GPU-swap gate has already made the VIDEO model the GPU owner for this request, so the nested request for a different (image) model queues behind the very request that is blocked waiting for it. Observed on a real Wan run: [gpu-swap] queued 'stabilityai/stable-diffusion-xl-base-1.0' behind owner 'wan-ai/wan2.2-i2v-a14b-diffusers' with the GPU sitting at 0% until the request was killed. Call codai.api.images.create_image_generation directly instead. The engine already holds the GPU for this request, and its own model manager handles the load/evict — which is also what we want, since the keyframe must be rendered before the video weights land. Verified end to end on real hardware (Wan2.2-I2V-A14B, 4-bit, model-CPU-offload): t2v request + character_profiles -> "rendering IP-Adapter keyframe" -> "IP-Adapter loaded" + 72 modules aligned to bf16 -> keyframe in 10 steps -> "running ti2v from it" -> Wan loaded (24.45 GB VRAM measured) -> HTTP 200 in 1642s with a real 480x480 h264 video, 17 frames, 2.125s, motion present between first and last frame. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Verified end to end on real hardware: same prompt, same seed, mean absolute pixel difference 46/255 between a generation with a character profile and one without. Before this, that difference was 0 — the profile changed nothing. IP-Adapter had never once loaded successfully. Four separate blockers, each hidden behind the previous one, all found by running it rather than reading it: 1. Attention slicing (71f88d14). diffusers' adapter loader rebuilds every attention processor; re-instantiating SlicedAttnProcessor without slice_size raised, the failure was caught and logged as a warning, and generation continued without conditioning. Slicing is now off across the load. 2. Quantized dtype cast. _load_ip_adapter_weights() ends with self.to(dtype=self.dtype) on the denoiser — a NO-OP cast to the dtype it already has — but bitsandbytes refuses any dtype argument on a quantized model. Tolerated for the duration of the load. 3. That tolerance then broke the forward pass: the suppressed cast was what brought the NEWLY added IP-Adapter modules up to the model dtype, so they stayed fp16 against a bf16 pipeline ("expected mat1 and mat2 to have the same dtype: BFloat16 != Half"). We now perform that cast ourselves on the non-quantized modules only — 72 of them, confirmed in the log. 4. Reference-image shape. A character profile carries several images, but diffusers matches the OUTER list against the number of ADAPTERS, not images: "Got 4 images and 1 IP Adapters". Multiple images for one adapter must be nested one level deeper. New shared helper ip_adapter_image_arg(), used by both the image and video paths. Also fixes a real production bug found while testing: pipelines are cached, and once an adapter is attached the UNet config demands image_embeds forever after, so the NEXT request without character references 500s with "...encoder_hid_dim_type set to 'ip_image_proj' which requires the keyword argument image_embeds". The adapter is now detached when a request doesn't need it (with slicing kept off across the unload too, which trips the same constructor). Verified lifecycle on one cached pipeline: with-profile -> without -> with again, all 200, and the two with-profile results are bit-identical (0.0 diff), so unload/reload restores conditioning exactly. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Loading the IP-Adapter (958d2e09) fixed the mechanism, but only AnimateDiff-class pipelines have one. On Wan / CogVideoX / LTX / SVD — the models people actually use — character_profiles still did nothing except prepend the profile name to the prompt, so identity was not working where it mattered. The image endpoint does have real IP-Adapter. So when a video model can't condition on references itself, render the first frame there against the same profiles and run the video as ti2v from that keyframe. This is the recipe tools/videogen.py already drives by hand, now done server-side. It runs BEFORE the video pipeline is loaded, so an image model and a video model are never resident at once, and it dispatches through the front (execute_api_request) so the image model may live on a different engine. New request fields: keyframe_identity (auto | always | never), keyframe_model, keyframe_steps. 'auto' skips the bridge when the mode...
-
Stefy Lanza (nextime / spora ) authored
The last two gaps from the person-to-scene audit. IP-Adapter on the video path never worked. _apply_character_refs() set kw['ip_adapter_image'] without ever calling load_ip_adapter(), so a pipeline that does accept the kwarg (AnimateDiff) failed inside diffusers rather than transferring identity, and it also passed ip_adapter_scale as a call kwarg, which no pipeline accepts — a second latent TypeError. It now reuses the image path's _ensure_ip_adapter_loaded(), sets the scale through set_ip_adapter_scale(), and returns whether conditioning is actually active. Pipelines with no IP-Adapter at all (Wan/CogVideoX/LTX/SVD) now say so instead of looking like they worked. Lip sync required a 'wav2lip' or 'sadtalker' executable on PATH — a file that essentially never exists, since Wav2Lip ships neither a pip package nor a binary and its original checkpoint links are dead. codai/api/lipsync.py now owns the lifecycle the way ds4_worker owns ds4: it fetches code AND weights from the camenduru/Wav2Lip mirror on first use, patches what no longer runs on modern numpy/librosa/torch, creates the scratch dirs inference.py expects, and drives it as a subprocess. An operator-installed binary still takes precedence. Each patched file is compiled before being kept, so a bad rewrite is caught at patch time and reverted rather than surfacing as a SyntaxError from a subprocess (which is exactly how the first attempt failed: two librosa patterns compounded into 'sr=sr='). Lip-sync fallback is no longer silent. The reason travels up a per-request warnings list — not module state, which would cross-talk between concurrent requests — and appears as a `warnings` field on the video response, so a client can tell a synchronised result from a plain audio mux. Verified offline (live container untouched): Wav2Lip downloads, patches, and lip-syncs a real face to audio, producing a valid 768x512 h264+aac mp4 that differs from the source; the second run reuses the install with no download; the fallback populates warnings; IP-Adapter loads its weights and sets the scale for a supporting pipeline and cleanly declines for a Wan-like one. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three defects found while auditing the 'person from a video -> scene with their voice' path. 1. camera_motion returned 500 on every request that used it. _apply_camera_motion() set kw['camera_motion'] unconditionally and nothing ever popped it, so pipe(**kw) raised TypeError — no diffusers video pipeline accepts that kwarg (the 'CogVideoX supports it natively' comment was wrong). tools/videogen.py sends the field, so that path was dead. Now the kwarg is passed only when the pipeline signature really takes it, otherwise the motion is folded into the prompt, which is the only conditioning these models have. 2. dialogs[].voice never cloned the voice it named. _generate_tts_for_line() loaded the profile's reference audio and then used it purely as a flag to force a generic voice (en-US-JennyNeural / af_sky) — F5-TTS was never invoked, so the selected identity was silently discarded. It now clones with F5-TTS when the profile has reference audio AND a transcript, and says why in the log when it falls back. 3. seed-vc and F5-TTS could not load their BigVGAN vocoder. huggingface_hub 1.x stopped forwarding proxies/resume_download to a subclass _from_pretrained; BigVGAN still declares them as required keyword-only args, so loading died with 'missing 2 required keyword-only arguments'. Pinning hub below 1.0 is not possible (transformers 5.x needs the new one), so codai/api/hub_compat.py supplies the dropped arguments for exactly the subclasses whose signature still requires them. This also un-breaks /v1/audio/convert. Also fixed on the way: - edge-tts never ran: get_event_loop().run_until_complete() inside an executor thread raises RuntimeError. In _generate_tts_for_line it was swallowed; in _generate_tts only ImportError was caught, so it propagated out of /v1/video/dub. Both now use asyncio.run() and verify the output file. - music dub adopted a failed Seed-VC result: converted_path was assigned before the write succeeded, so a near-empty conversion got remixed instead of falling back to the F5-TTS vocal. The result is now validated before adoption. Verified offline (live container untouched): camera_motion passes through only to a supporting pipeline and otherwise reaches the prompt; a dialog line clones with the profile's reference audio and transcript and refuses to clone without one; seed-vc loads BigVGAN and converts; the music dub falls back to the full vocal and reports the reason. Co-Authored-By:
-
- Jul 22, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Teardown leak: _free_pipeline_vram now records the pipeline's CUDA storage pointers up front and, after the normal teardown+gc, walks gc.get_objects() and nulls any surviving tensors out of their list/dict referrers (the proven pass from the CUDA text backend's cleanup) — the "~22 GB untracked (teardown leak; referenced elsewhere)" VRAM is now reclaimed instead of poisoning every later load until a restart. Names the holders when it fires so future leak sources identify themselves. Tracking: a video pipeline only registers in manager.models AFTER its multi-minute load, so concurrent loads raced the free-VRAM check into mutual OOM. note_loading()/clear_loading() reserve the model's estimate for the whole load window; _get_free_vram_gb subtracts reservations and the orphan-VRAM check counts them (no more false "teardown leak" for an in-progress load). The embeddings OOM-retry waits out active reservations (bounded 300s) before evicting and retrying. Co-Authored-By:
-
- Jul 03, 2026
-
-
Stefy Lanza (nextime / spora ) authored
evict_cosited_siblings() frees the co-located sibling's VRAM ONCE at training start, but training runs for minutes as an in-engine background job that the front swap-gate can't cover (its POST returns a job_id immediately). So a concurrent LLM request reloaded the gguf text model mid-training and OOM'd the trainer (one fighter LoRA failed while others succeeded). Add codai/models/gpu_lock.py: a cross-engine GPU reservation. Training reserves the card for its whole duration — locally AND on co-located siblings via new /internal/gpu-reserve + /internal/gpu-release endpoints — and every ordinary model-load path (manager.request_model, video _load_video_pipeline, image _load_diffusers_pipeline) calls wait_until_free() first. A request that needs a load during training now BLOCKS until training releases, then loads and serves — the same queue-behind-the-owner behaviour the swap-gate gives request-level work, now extended to cover training. The training thread is exempt from its own reservation, so loading the base model never self-deadlocks; waits are bounded (900s) so a stuck reservation can't hang forever. Validated: sibling reservation blocks a loader thread until release; the reserving thread never waits on itself. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
diffusers' Model.set_attention_backend() doesn't just set per-processor backends — it ALSO flips a process-wide active backend (attention_dispatch's _active_backend). The video path sets that to flash-attn for the Wan transformer; image and video share the nvidia-engine process, so the global stayed flash and leaked to the next image model. Z-Image's transformer sets no backend of its own (passes backend=None → uses the global) and its attention is masked, so it crashed with "`attn_mask` is not supported for flash-attn 2" → image/environment generation 400. reset_attention_backend() clears per-processor backends but NOT the global, so it didn't help. Fix: restore the diffusers global backend to the env default (native/SDPA) (a) before every image generation — bulletproof against a leaked flash backend — and (b) in the video pipeline teardown (_free_pipeline_vram), so it can't persist after a video pipe is freed. Masked image attention (SDPA) now always works; the video transformer keeps its own per-processor backend. Co-Authored-By:
-
- Jun 30, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Two fixes for the township video render failing with CUDA OOM and a ~17 GB "untracked teardown leak" that survived gc + empty_cache. 1. Resident-experts regression. video_resident_experts now defaults to OFF. A dual-expert 14B model (Wan2.2-VACE-Fun: transformer + transformer_2, ~10 GB each at 4-bit) cannot hold both experts + text encoder + VAE + the activation peak in 24 GB; the resident load left transformer_2 on the CPU yet reported success, so the denoise loop (which needs both experts) OOM'd at step 0. 'model' CPU offload keeps only the active ~7 GB expert resident and swaps, so it fits. Also: when resident leaves ANY component off-GPU it is now treated as a failed load — the partial pipe is torn down and it falls through to model offload, instead of returning a half-loaded pipe that pins ~10 GB. 2. Teardown leak. _free_pipeline_vram now breaks the references that outlived a plain component-null: reset any non-default attention backend, unload LoRA/PEFT adapters, run the pipe's own maybe_free_model_hooks()/reset_device_map() (frees accelerate offload hooks + their staging buffers), and drop the _coderai_* stamped attrs, before nulling components + gc + empty_cache. Verified live (image 0.1.33): 31 video units rendered, 0 OOM, 0 leak diagnostics, idle GPU back to ~6 GB. Co-Authored-By:
-
