- 29 Oct, 2026 17 commits
-
-
Stefy Lanza (nextime / spora ) authoredtorchaudio has no cp313 build before 2.6.0: No matching distribution found for torchaudio==2.5.1+cu124 (from versions: 2.6.0, 2.6.0+cu124, 2.7.0, …) so the pair the CrisperWhisper venv uses cannot be reproduced here. Both move to 2.6.0+cu124 together, which is what matters: a torch and a torchaudio that disagree is the failure this pin exists to prevent. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
-
Stefy Lanza (nextime / spora ) authored
The pinned install succeeded — torch 2.5.1+cu124, parler_tts 0.2.2, transformers 4.46.1 — and `import parler_tts` still failed, with the worker reporting only parler-tts install did not yield an importable package which names neither the package nor the reason. It now reports the import error itself. The reason pip left behind: parler-tts pulls in descript-audiotools, which requires an unpinned torchaudio, and pip resolved 2.11.0 against torch 2.5.1. torchaudio is pinned with torch now, in the file and in the inline pins — a pair of versions that must match is not something to leave to a resolver. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The GPU probe shipped a moment ago did its job on the first live run: it caught the cudnn failure and asked for a rebuild. The rebuild then stopped with Parler requirements file missing: /opt/coderai/app/requirements-parler.txt The container mounts `codai/`, `tools/` and `third_party/` over the image, so a repo-ROOT file added since the running image was built is simply not there — and a worker that heals itself must not stop for a reason that has nothing to do with the venv. The same mistake was already on record as ACE-Step reporting a missing `requirements-acestep.txt` for a venv that was fine. The file stays the source of truth and wins whenever it is present; the same pins also live in the worker for when it is not, with a test that parses the file and compares, because two copies of a pin drift. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
With the environment fixed, Parler got further: the venv built, the service started, the model loaded — and the first synthesis died in the DAC audio decoder's conv1d. RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed ptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED Every cudnn sublibrary present in the venv, and the stack unable to load them: a torch 2.15 with the CUDA-13 wheels that `pip install git+…parler-tts` had resolved on its own, because nothing pinned it. requirements-parler.txt names the build now, the way every other isolated worker's file does. The health check was the deeper problem. `import parler_tts, soundfile` answered True for that venv for ever, so nothing ever rebuilt it. A venv's check has to exercise what the worker actually does: the decoder is convolutional, so the probe is one conv1d on the GPU, and a venv that imports but cannot run it is rebuilt from the pinned file. A CPU-only host passes — there is no GPU stack there to be broken. The service also latches cudnn off if it still fails there. torch has native kernels for those convolutions, the decoder is a fraction of the work, and the alternative is no speech at all. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two production failures from the same render, both fixed at the level they belong to. **Speech engines "listed but not installed".** XTTS answered 501 "no venv containing it was found" against a venv that was there and populated; parler answered 500 "python -m venv returned non-zero exit status 1"; a worker's torch died with CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED. Three different failures, one cause: the engine processes export PYTHONHOME=/opt/coderai/python, every worker was launched with env=dict(os.environ), and a venv interpreter started under a foreign PYTHONHOME takes its stdlib and site-packages from THAT installation — so the venv's own packages are not on sys.path at all. The import probe was the first casualty, which is why it reported the venv as missing. codai/api/venv_env.py — which already existed for exactly this, and which the TTS/STT workers were not using — is now the one place a worker's environment and its venv are decided. Every call INTO a venv goes through it: the probe, `python -m venv`, each pip install and the service itself (a clean service with a poisoned probe still reports the venv as broken; a poisoned pip installs into the wrong tree). LD_LIBRARY_PATH is decided per venv, because a foreign python must not see coderai's library directories while a venv built from the server's own interpreter needs its libpython — and sys.prefix no longer counts as "ours" when it IS /usr, which on a pip install would strip the driver path from every worker. venv_usable() replaces four private copies of "can this venv be used". It catches what none of them did: a venv whose python runs perfectly and imports nothing, because its 875 MB of packages are installed for a different minor version (an xtts_venv built by the host's 3.13 pointing at the container's 3.12). `pyvenv.cfg` spells the version two ways; both are read. An intentional overlay (--system-site-packages, for the H3 venv that inherits the server's torch) is respected in both directions. And `GET /v1/audio/speech/capabilities` now reports needs_venv / env_ready / env_detail, from the filesystem alone — never importing into the venv, never loading a model — so a client knows before it offers a voice whether the first request will spend minutes building the environment. **"CrisperWhisper out of GPU memory during compose", every scene.** word timestamps failed (500: … CUDA out of memory. Tried to allocate 86.00 MiB …) The speech model was loaded and idle between scenes while the TTS and visual models of the same job loaded around it and filled the card. 86 MiB is what a resident model needs to run one window, and nothing in the system was keeping it: every reservation is taken at LOAD time, and this model was already loaded. Every caption in that render lost its word timings. Now three numbers per speech model, all measured: the weights, what one pass needs on top of what was already allocated, and their sum. The per-pass number is ensured free before each pass (manager.ensure_free_vram) and is a DELTA, not the process peak — the peak would charge a speech model for a 16 GB image model resident beside it, and every later load of it would evict the world. An OOM that still happens frees the card and retries once, on both the in-process and the isolated-worker path. The in-process reserve was also a third of the truth: 3 GB hard-coded for a whisper-family model, where the isolated worker serving the same checkpoint used 8 GB and said in its own comment that 3 GB was about a third of the real peak. Both paths now call one function, so eviction frees the same amount whichever way the model is running. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A model listed as able to read images has to read them, or say it cannot. `Qwen3.5-9B` was listed with `image_to_text` and answered every photograph with "a blank white image with no visible content" — nine real pictures of one video, indexed as blank. Two causes, one symptom: AutoModelForCausalLM -> Qwen3_5ForCausalLM (text only) AutoModelForImageTextToText -> Qwen3_5ForConditionalGeneration The engine asked for the first, so the tower was never built, and with no projector the chat path flattened the picture to the literal text "[image_url content]". The model was answering the prompt it was given. Now the class comes from the checkpoint's own config, every load path in the CUDA backend goes through that resolver, a vision model also loads the processor that makes a picture promptable, and an image request takes its own generation path (no KV prefix reuse — image tokens come from the pixels). Where a model genuinely cannot see, the request is a 400 rather than a confident wrong answer, and a GGUF with no mmproj stops advertising the capability it cannot serve. Two things found while measuring the image path on a live fleet: * every image load printed "Proactive eviction skipped: name 'request' is not defined". `_load_diffusers_pipeline` has no `request`; the NameError was swallowed as a skip, so the minimum-eviction block never ran and each load fell through to the blunt eviction that unloads models which would have fitted alongside. * `Z-Image-Turbo`, distilled for 8 steps at CFG 1, was served at 30 steps and CFG 7.5 because its card carried no `defaults` — four times the work for an over-guided picture, ~72 s a still. A card that says nothing now falls back to what the model's own name says before the generic floor. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Six findings from one 21h30m production log (548,687 lines, 51 loads of 12 models). They share a cause: nothing measured a model directly, so every decision downstream of a measurement was made on a number that was wrong. 1. The footprint was a process-RSS delta --------------------------------------- `measured_ram = self._get_own_ram_gb() - ram_before`, where _get_own_ram_gb sums RSS over the engine and every child. RSS counts mmap'd weight-file pages the loader just walked; summed over a tree it counts pages SHARED between those processes once per process; and as a delta it absorbs anything else that moved in the window. Against ground truth: gemma-2-9b-it.Q4_K_M 5.37 GB on disk -> reported 30.81 GB (5.7x) LTX-2.5, six loads, one unchanged config: 22.11, 22.11, 23.28, 4.50, 19.02, 29.80 GB (6.6x) Every other model sat at 1.0-1.2x, so this was not noise. Wan2.2's 3.4x spread was the real quantization fix and is the one that belonged there. codai/models/footprint.py measures the object instead. Quantization needs no correction factor, which is the point: a bitsandbytes Params4bit IS a uint8 tensor, so numel x element_size already gives packed bytes. What those formats hide is counted explicitly -- bnb's quant_state (absmax, code book, nested double-quant state) lives outside parameters(), and GPTQ/AWQ keep qweight/qzeros/scales in BUFFERS, so a parameters-only walk reports a 4-bit model as nearly empty. Tied weights are deduped by storage pointer; accelerate meta placeholders are sized from their dtype and flagged as declared, keeping the placement-independence the field already documents; a diffusers pipeline is walked through its components, since it is not an nn.Module. Context is costed properly: KV from the KEY/VALUE head count, not the attention head count (4x on a GQA model), with sliding-window layers sized to their window. But a computed KV is only a fallback and says so -- this codebase already learned that "predicting KV from GGUF metadata is architecture-specific ... and was off by 5x in BOTH directions", so measured_kv_gb for the context in question always wins. A GGUF, which no walk can reach, falls back to its file size; it is loaded as stored, so the file IS the weights. 2. A failed load overwrote the good measurement ----------------------------------------------- Device map: cpu: 1 layers Measured footprint: 4.50 GB total Saved measured_vram_gb=4.505 (force_vram_update) generation FAILED: StopIteration interacts badly with generators Full-GPU load looks viable (4.5 GB peak need) <- 4 seconds later A load that placed one layer and raised still reached the recorder, and force_vram_update wrote it to disk as truth. Now a measurement that is implausible on its own (~0 GB) or that collapses to under half of what the SAME quantization signature previously measured is treated as a partial load and discarded. Wan2.2's genuine 40.04 -> 11.73 GB drop is let through because it arrived WITH a signature change, which is exactly what tells the two apart. The rejection names which of the two checks fired. The ratio guard alone was not enough. Walking a pipeline and finding that NO component holds any weights is a failed load outright, and there is nothing to fall back to -- a size taken from process memory measures everything except the model in question. That case is now refused before any estimate is considered, and the refusal is recorded in the history with its reason rather than left looking like a successful load. The hole showed up as one of the new tests passing alone and failing in the full suite: the fallback delta grows with whatever else the process has loaded, so the figure it invented depended on test order. 3. The teardown-leak breaker never looked at host RAM ----------------------------------------------------- It registers under `if _tens.is_cuda` and sweeps under `_obj.is_cuda`. A pipeline loaded with enable_model_cpu_offload keeps its weights in HOST RAM, and 17 of 21 video loads took that path, so none of those weights were tracked and nothing looked for them. With nothing loaded and 482 MiB of VRAM in use: coderai-nvidia 33.14 GB PSS_Anon 30.85 GB anon in 6588 mappings; 6565 of them under 100 MB (20.25 GB) Thousands of mid-sized allocations is retained tensors, not allocator slack -- which is why gc and malloc_trim ran every 15 s for hours with the figure never moving, and why the ladder then evicted healthy models to chase it: RAM eviction: unloading 'video:Lightricks/LTX-2.5-Diffusers' to free host RAM (RSS 40.0 GB > cap target 36.5 GB) codai/models/teardown.py adds the host arm, by weakref rather than by storage pointer: a freed host address is readily handed back out by malloc, so a pointer match could null a live tensor that by then belongs to another model. A referent still alive after teardown IS this pipeline's leak, identity-exact. It runs before the allocator trim, since trimming while the weights are still referenced returns nothing. 4. The RAM cap counted shared pages and page cache -------------------------------------------------- codai/models/proc_mem.py reads Pss_Anon + Pss_Shmem from smaps_rollup, which has already divided shared pages by their sharers and excludes reclaimable page cache. Measured correction on a 5-engine host: 40.06 GB (RSS) -> 37.91 GB (PSS). Modest -- the leak above was the real problem -- but a cap should chase memory that freeing an allocation can actually release. Falls back to the old reading rather than to 0.0, which would read as "no memory in use". 5. A model nobody used stayed loaded ------------------------------------ Ondemand mode - keeping 'audio_gen:ACE-Step' in VRAM alongside new model It held 7.9 GB of a 3090 and 2.1 GB of host RAM for hours after its last request. offload.idle_unload_seconds (default 1800) unloads a model untouched that long. Deliberately generous: reloads are the expensive event -- that day spent 2h04m of 21h30m inside loads -- so a short timeout would trade a VRAM saving nobody asked for against a reload everybody waits on. A key that was never stamped is left alone rather than assumed idle. It runs before the leak-watch gate, so turning that off does not silently disable this. 6. The log left no room for the history anyone needed ----------------------------------------------------- 32,633 of 33,009 requests were embeddings, and one engine produced 426,682 of the 548,687 lines -- 78% -- filling the 2,000-line view in about four minutes. The per-item endpoints are now summarised to one line (model, count, shape) instead of dumped, which also stops a caller's content being written to a shared log. Prompts are deliberately NOT truncated: a 4,000-character script prompt is what identified the music defect below. GET /v1/models/history replaces inferring swaps from a log that rotates: time, event, model, engine, plus seconds, vram_gb, offloaded_gb, strategy for a load and reason, freed_gb for an eviction. Eviction reasons are passed at the call sites. The front merges across engines, because a swap is a cross-engine event -- one engine evicts so a co-located one can load -- so one engine's copy is half the story. Also: ACE-Step was asked for a song with no words ------------------------------------------------- Motus sends a music_prompt and no lyrics, and empty lyrics do not mean instrumental: the pipeline then conditions on a single padding token (lyric_token_idx = [0], lyric_mask = [0]) and the model still tries to sing, which comes out as babbling over the backing track. ACE-Step's own UI documents the way to ask -- "Use [instrumental] or [inst] to generate instrumental music" -- and every genre preset it ships ends in either "male vocals, raw vocals" or "instrumental". Both are now sent when no lyrics are given. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`_last_used` is stamped in exactly one place: the `active_in_vram` setter, whose comment calls it the "single chokepoint for 'this model was just used'". `add_model()` goes through it. Seven sites in video.py and images.py wrote `multi_model_manager.models[key] = obj` directly and did not, so for every video pipeline and four image models: - `_lru_order()` sorts by `_last_used.get(k, 0.0)`, so the key sorted FIRST and was evicted ahead of genuinely older models -- permanently, however recently it had been used; - the first eviction pass skips `key == self.active_in_vram`, which an unstamped key never matches, so that protection never applied either. Both ladders iterate `_lru_order()`: the VRAM one and the cross-engine `cosite_evict`. Observed on a production run -- a co-located engine needed room and released the freshly-loaded video pipeline before a TTS model that was older but had gone through `add_model`: [cosite-evict] releasing 'video:Wan-AI/Wan2.2-TI2V-5B-Diffusers' ... [cosite-evict] releasing 'tts:Kokoro-82M' ... Also disambiguate the evict-debug line that led here. It read `getattr(model_obj, 'backend', '∅')`, which returns the default both when the object has no such attribute and when `model_obj` is None -- so a healthy diffusers pipeline (which never carries `.backend`) and a key that had already vanished from `self.models` printed an identical `backend=missing`. Only the latter is a fault. Name the two apart, report the object type, and compare the sentinel by identity rather than `==`, which a tensor-bearing object need not answer with a bool. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
8303b38e passed `resolved_name` to _quant_signature from record_vram_delta, which has no such variable. The NameError was swallowed by an enclosing try, so measured_vram_gb persisted and the signature beside it never did. Verified live 2026-10-29: after a real load, measured_quant_sig was None and the invalidation fired zero times. Scope was checked at the READING site and assumed at the writing one. They are different functions. The deeper fault was hashing the model name at all. The two sites do not have the same name to hand -- the reader has a resolved path, the writer only a config key -- so even bound, they could hash different values and every comparison would be a mismatch, discarding every measurement. The signature is stored on one model's own entry, so identity is already implied: it now hashes only precision, load_in_4bit/8bit and component_quantization. That also drops pipeline_cache's cache version from the hash, which would have invalidated every measurement on an unrelated cache bump. A test now walks both call sites and asserts each binds what it passes. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A measurement describes the model as it was BUILT. Change component_quantization and the stored number describes something that no longer exists -- and because force_vram_update makes the measurement win over the configured size, the stale figure goes on deciding placement. Wan2.2-TI2V-5B is the worked example, and the numbers are worth keeping. Its components are 18.63 GB transformer (~5B params at ~3.7 bytes each, i.e. fp32), 10.58 GB text encoder and 2.63 GB VAE. Its map named text_encoder and vae; vae is conv-only so bitsandbytes skips it, which left exactly ONE component quantized and an fp32 18.63 GB transformer loading whole. Hence measured_vram_gb = 40.0 for a model whose name says 5B. With the transformer actually quantized the same pipeline is roughly 10 GB, but 40.0 would have kept forcing CPU offload on a 25.3 GB card until some unlucky load happened to re-measure it. The measurement is now stamped with pipeline_cache._signature -- which already hashes model id, precision, load_in_4bit/8bit and component_quantization, the exact set that decides how big the built weights are -- and is ignored when that signature no longer matches. Offload strategy and used_vram_gb are not in it, so a runtime placement change does not throw away a good measurement. An entry with no recorded signature is still trusted, so upgrading does not invalidate every existing measurement at once. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
component_quantization replaced the global flag instead of refining it, so naming ONE component switched load_in_4bit off for every other one. Measured 2026-10-29 on LTX-2.5 (load_in_4bit=True, map naming transformer and text_encoder): ten of its twelve components loaded at full precision, among them connectors at 11.82 GB and prompt_enhancer at 9.51 GB -- 21.3 GB of a 25.3 GB card before the quantized transformer arrived, which is why the resident attempt OOMed on exactly those components. The candidate list was the second half of it. Written for unet/SD-era pipelines, it matched only transformer*/unet/text_encoder*, so connectors and prompt_enhancer could not be reached by the global flag either. A modern video pipeline carries several multi-GB Linear-heavy parts under other names. Both builders are fixed together -- build_pipeline_quant_config and the incremental component cache -- because they had the same shape, and a cached build that quantizes differently from a direct load is its own bug. An explicit map entry still wins, including "none"/"full", which is how a component is opted out. Wan2.2-TI2V-5B shows the size of this: load_in_4bit=True with a map naming only text_encoder and vae, so its transformer was never quantized at all -- a 5B model measuring 40.0 GB on a 25.3 GB card. Separately, the message printed when eviction cannot make room claimed "Remaining models are busy or VRAM is held elsewhere" without having established either, and it was usually false. Over one production day, 10 of 12 of these were a model asking for more than the whole card (25.4 to 40.0 GB against 25.3 GB), four of them printing "freed 0.0 GB (now 25.3 GB free)" -- the card was already empty. "Skipping eviction of ... still busy" never appeared once: eviction was never blocked by a busy model. That wording sent an investigation at the scheduler when the causes were a model larger than the GPU and components that were never quantized. It now names which of the three cases applies, and for the common one points at the two things that actually help. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
record_resident_fits called save_models(), which writes the config manager's own view of models.json -- a view that never saw this change. Verified live 2026-10-29: after a run that learned the model does not fit, the flag was set in memory (the next load in the same process correctly skipped the resident attempt, 110.6s instead of 865.3s) while models.json still read measured_resident_fits = None so every restart would have paid the ~16 minute discovery again. persist_model_field is the right call and is what record_vram_delta already uses for measured_vram_gb: it re-reads models.json from disk, sets one field and writes back, so a stale in-memory copy cannot revert a UI edit made meanwhile. It also targets the entry by config_id, so two configs sharing one checkpoint path do not overwrite each other's measurement. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The release added in 9715d200 ran after _free_pipeline_vram, which is too late: that function ends in the teardown-leak breaker, which walks live CUDA tensors and nulls whatever still references them. By the time the release ran the components had already been stripped, so it found nothing on the GPU and printed nothing, while the breaker reported the full 16.9 GB of this pipeline's tensors survived teardown; nulled 7581 external reference(s) exactly as before the fix. Verified live 2026-10-29: the new line never appeared and the figure was unchanged. The ordering is not cosmetic, because what the breaker nulls is the modules' own _parameters. It frees the VRAM and leaves the injected components gutted -- and those components are reused by the very next attempt. accelerate's offload hook then does current_device = next(module.parameters()).device on a module with no parameters left, which raises StopIteration, which in a worker thread becomes "StopIteration interacts badly with generators" and fails the generation. Measured: the run whose resident attempt failed died that way at encode_prompt, while the run that skipped the resident attempt (and so never ran the breaker) generated normally on the same pipeline. So the leak, the gutted components and that crash are one bug seen at three removes. Releasing first keeps the components intact, leaves the breaker with nothing to strip, and the retry gets working weights. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two bugs that only bite when something else has already gone wrong, so they turned a recoverable load into a failed one. A fully quantized module could not say what dtype it was. transformers has next(param.dtype for param in self.parameters() if param.is_floating_point()) and a bnb 4-bit module has NO floating-point parameters -- Params4bit are uint8 -- so the generator is empty and next() raises StopIteration. diffusers' DiffusionPipeline.to() reads module.dtype unguarded while deciding whether to warn about fp16-on-cpu, so it escapes enable_model_cpu_offload(); and because loads run in a worker thread, Python turns it into RuntimeError: StopIteration interacts badly with generators and cannot be raised into a Future failing the whole load. Measured on LTX-2.5, whose text encoder is 4-bit: the resident attempt did not fit, the fallback to model CPU offload hit this, and the load restarted from scratch. The recovery path broke exactly when it was needed, and it would do so for any pipeline with an all-4-bit component. The property asks what dtype a module is, and a quantized module has a good answer: the dtype of its storage. The shim falls back to the first parameter of any kind, then to float32, and only when the original raises -- a module with floating-point parameters is answered exactly as before. Guarded at every enable_model_cpu_offload() call site, with a test that fails if one is added without a guard. The injected components kept the GPU between attempts. They are pre-built objects held in a local dict and handed to each attempt through _with_quant, so freeing the pipeline frees nothing: the dict is still a strong reference and whatever the failed attempt moved onto the card stays there. [video] teardown-leak breaker: 16.9 GB of this pipeline's tensors survived teardown; nulled 7581 external reference(s); holders: ["2.01 GB dict['weight', 'bias']", ...] The holders are the modules' own _parameters dicts -- the modules outlived the pipeline. The breaker got the VRAM back by stripping them field by field, which works, but it guts components the next attempt is about to reuse. They are now moved to the CPU at both resident-failure exits: moved, not dropped, because the retry needs these same components, which is the whole point of injecting them. The breaker stays as the backstop for a component that refuses .to('cpu'). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Letting a sized-to-fit cache hit try the GPU (1f42d08c) is right for a model that fits and expensive for one that does not. Measured on LTX-2.5: 04:55:07 RESIDENT all-GPU load (sized to fit) 04:56:53 [resident] components left off-GPU: transformer(CUDA OOM), vocoder(CUDA OOM), prompt_enhancer(CUDA OOM) 05:11:10 ✓ loaded — strategy: model CPU offload 05:11:12 23.28 GB total, of which 21.32 GB offloaded to host RAM Sixteen minutes to reach exactly the placement that would have been chosen immediately, and it would have paid that on every load. The sizing was not wrong so much as answering a different question: it knows what the weights weigh (22.1 GB against 25.0 GB free), while resident needs every component -- transformer, vocoder, VAE, text encoder, prompt_enhancer -- plus the activation peak, all at once. The real footprint turned out to be 23.28 GB. Only an attempt can answer that, so the answer is recorded as a system-owned learned field beside measured_n_gpu_layers, which exists for the same reason: stop repeating an expensive discovery every load. Both outcomes are recorded, not just the failure. Writing only failures would re-prove a model that fits on every load; writing only successes would never stop the expensive retry. And it is not a one-way door -- a later success clears an earlier failure, because a bigger card, a smaller quantisation or fewer co-resident models all change the answer. Unknown still means try. A model never attempted, or a card that has since changed, gets its chance at the GPU rather than being written off by default. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
XTTS-v2 and Parler were advertised in /v1/models and failed on every request. Three separate causes, none of them the one the report assumed. A venv built by a DIFFERENT interpreter bricks the engine permanently. A worker venv keeps bin/python as a symlink to the python that created it, so one built on the host and used from the container (or built before the image moved its interpreter) leaves a dangling link. The failure is not a clean "missing": Path.exists() follows symlinks and says False, so the code decides the venv is absent and runs `python -m venv` over the top, which trips on that same dangling bin/python3: Error: [Errno 2] No such file or directory: '.../parler_venv/bin/python3' reported for weeks as "the venv can't be created", with a fully populated venv and 7 GB of packages sitting right there. Both workers now check that the interpreter RUNS, not that the path exists, and clear a foreign venv before recreating. TRANSFORMERS_CACHE pointed at a second cache. It is deprecated but transformers 4.x still honours it, and honours it as the cache_dir handed to hf_hub_download -- a hub-layout cache, not a different kind of store. Setting it to <dir>/transformers therefore gave the machine TWO hub caches: measured, <dir>/hub held 122 models and <dir>/transformers held 2. A worker in an isolated venv read only the second, found nothing, and -- forced offline so a missing model fails fast -- reported "We couldn't connect to https://huggingface.co" for a model that was present the whole time. It is also why workers re-downloaded models the server already had. All four variables now name the hub directory. XTTS never built its venv at all. parler bootstraps on first use; XTTS only looked for a venv and raised a 501 telling the operator to go and make one. Same situation, two behaviours. It bootstraps now, with CODERAI_XTTS_AUTOINSTALL=0 for a host that manages its own venvs. Also the voice listing the report asks for, but NOT at /v1/audio/speakers: that path is already enrolled speaker voiceprints for recognition, and serving TTS voices there would conflate two unrelated things. The voices join the other "what can this model do" answers at /v1/audio/speech/capabilities, which already takes the same model= parameter. It never loads a model to answer -- an already-loaded backend is asked directly, otherwise the list is whatever was remembered the last time that model loaded, and voices_source says which. And it does not invent metadata: Kokoro ids genuinely encode accent and gender (af_sarah -> en-us, female) so those are derived, while an XTTS speaker name carries neither and reports None. A wrong gender against a voice is worse than no gender. voice_selection is reported too, because parler has no speaker names at all -- it is steered by a free-text description, and a client reading only an empty voices list would conclude it cannot be directed. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A production run put 93% of a video model on the CPU beside an idle GPU, nine times. The ladder was not at fault -- it evicted an active model, then a keep-resident one, reached 24.9 GB free and concluded: Full-GPU load looks viable (22.1 GB peak need, 24.9 GB free) — using full GPU and two minutes later the same load reported: [pipeline-cache] injected cached components → forcing hook-based 'model' CPU offload Video load strategy: model CPU offload [after load] VRAM 0.4/25.3 GB (24.9 GB free) footprint 22.11 GB total, of which 20.47 GB was offloaded to host RAM The sizing expressed "use the whole card" by leaving `offload` as None, and None also means "nobody chose". The cached-components path could not tell the two apart, so it fell through to its own default -- 'model' CPU offload, which is there for a dual-expert Wan2.2 whose two ~10 GB experts genuinely do not fit in 24 GB, and which was then applied to every cached pipeline whatever its size. It did not heal either: a pipeline loaded under accelerate offload is never re-cached ("cache must be built from a pre-offload pipeline"), so once a model had been cached, every later load was forced onto the CPU and no clean copy could replace it. The verdict now travels with the request. A cache hit that was sized to fit loads resident; the dual-expert default still applies when it does not fit, and video_resident_experts still forces it on. _full_gpu_viable is assigned unconditionally. Written where the sizing happens it would have lived inside `if _offload is None:` → `try:` → `if cuda.is_available():`, so a per-model strategy, a CPU-only host or any exception there would raise NameError and break EVERY video request -- a much worse failure than the one being fixed. The test asserts the assignment DOMINATES the read, not merely that it precedes it, and fails if it is nested back inside a branch. Co-Authored-By:
-
- 28 Oct, 2026 23 commits
-
-
Stefy Lanza (nextime / spora ) authored
Production, 2026-10-28: Krea-2-Turbo answered one request and then 500ed. RuntimeError: Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu The offload moved the encoder to the CPU and the pipeline is CACHED, so the next request met an encoder that was no longer where encode_prompt indexes. _free's docstring even said the encoder "waits in host RAM ... the next request needs it again" -- and nothing ever brought it back. What turned that into a 500 rather than a skipped optimisation: the side effect outlives this function. The bail-out path returns the original kwargs and lets the pipeline generate "the ordinary way", but the ordinary way calls the SAME encoder, so it hit the same mismatch and the request failed anyway. A fallback that leaves the thing it fell back to broken is not a fallback. So the offload is a per-request cycle -- bring the encoder on, encode, move it off -- and every bail-out restores the encoder first, because each one hands the request to a path that needs it on the GPU. The regression test drives a pipeline that really refuses to encode off-device and asserts the SECOND request still works; without the restore it reproduces the production message exactly. A second test pins the weaker promise that any bail-out leaves the encoder where the ordinary path can use it. Found by the operator mid-render, not by the suite: every check so far ran one request per model against a freshly started engine, which is the one case that cannot fail. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Scaleway, Exoscale, OVHcloud and Hyperstack all rent GPUs by the hour or the second, and nothing about the fleet logic is really RunPod-shaped. Written down now so the seam does not have to be re-derived later; none of it is built. The point worth keeping: the provider-specific part is small. Demand-based growth, the reaper, budgets, the ledger, spillover, shared pools, LoRA fan-out and region selection all sit above the provider and ask it for about eight things, which RunpodClient already is -- ping, a catalogue with VRAM and price, create, get/list, wait_ready, stop/terminate, logs, volumes. A second provider is a second client plus a per-pool choice, not a second orchestrator. The section also lists what will NOT port, because the §10 notes were all found live and most are categories rather than one provider's quirk: an undocumented image size ceiling, a boot budget that has to cover the image pull, how the port is reached (proxy vs direct TCP vs a load balancer with its own idle timeout), rate limits in their own units, terminal states that do not look terminal, and region-scoped storage. Indexed from docs/README.md so it is findable without reading to the end of an 890-line file. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The sequential text-encoder offload reported itself "not enabled" on a card that plainly had `sequential_text_encoder = True`. Four defects, stacked so each one hid the next; the first is much wider than the feature that exposed it. _resolve_config_key could not resolve ANY non-text model by alias. Typed entries are keyed "<kind>:<name>", so the exact-id and alias-map lookups both miss, and the basename fallback misses too: "video:LTX-2.3-PinkCherry" has no slash, so basename() hands back the whole prefixed string. Every per-model image / video / audio / tts / embedding setting was therefore replaced by the global default, silently, for any model addressed by its alias -- which is the only name that distinguishes two configs sharing one checkpoint path. The resolver now tries the typed keys, ordered by the types the model is actually registered as. images.py (6 sites) and tts.py looked their config up with a raw config.get(model_key) and had the same bug; they resolve now, with a test that fails if a raw one reappears. The flag itself lived only in _raw_cfg, where a consumer reading config['sequential_text_encoder'] never saw it. Promoted to a top-level runtime kwarg, as gpu_split beside it already documents. With the gate finally firing, the render failed: "Must provide `prompt_attention_mask` when specifying `prompt_embeds`". encode_prompt's return order had been guessed. Read off the installed sources, LTX-2 returns (embeds, mask, neg_embeds, neg_mask) and Krea 2 returns (embeds, mask) -- the mask comes SECOND, and the old code read it third, filing negative embeddings as the attention mask. Naming is now done by classifying the values (a mask is an integer tensor, embeddings are floating) instead of by position. That rewrite then dropped Krea 2's negative prompt, which would have been worse than the failure it replaced: Krea 2's __call__ encodes "" itself when handed no negative, against an encoder this function has just moved to the CPU. The negative is always encoded up front for those families now. Hardened for families nobody has tried: an SDXL-shaped pipeline returns four float tensors including the pooled pair, and naming only two does not degrade quietly -- the pipeline rejects incomplete embeddings and the request fails. If everything encode_prompt returned cannot be named, the optimisation is abandoned for that request and says so. Measured on LTX-2.3-PinkCherry (480x480, 25 frames, 8 steps, RTX 3090): without 369.4 s peak 23718 MiB 91% util with 345.3 s peak 24098 MiB 92% util 7.3 GiB freed Read as headroom, not speed: the card is 24 GiB and the denoiser fills what it is given, so peak sits at the ceiling either way and the 24 s is one sample. The 7.3 GiB is what stops these renders being pushed to CPU offload, where the same job took tens of minutes. Also here: a clip-audio scene no longer needs an explicit duration (B1 said the duration comes from the clip, and validation refused exactly the request the feature exists for), and _free now reports per component whether it moved, was already off the GPU, or REFUSED the move -- a bnb-quantized encoder can refuse, and all three used to arrive as "nothing was freed". test_sequential_text_encoder's fakes returned bare strings in the wrong tuple order, which is why the bug shipped: the test encoded the mistaken belief rather than the pipeline's contract. They carry dtypes and the real orders now. test_compose's engine-selection test patches shutil.which as well, because the production image ships wav2lip and sadtalker shims on PATH and the test was asserting the absence of a binary the real deployment has. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A text encoder runs ONCE per request, before the first denoising step, and is dead weight for every step after -- but it stays resident and keeps its VRAM. Measured on the cached snapshots, that is not a rounding error: Z-Image's encoder is 7.5 GiB of a 19.2 GiB pipeline, and LTX-2.3's 4-bit Gemma-3-12B is ~6.5 GiB in front of a 15 GiB transformer. PinkCherry declaring exactly 24.0 GB on a 24 GiB card is that sum, and it is why every LTX render was pushed to CPU offload. Encoding first and freeing the encoder hands that VRAM to the denoiser, so a model that "did not fit" fits. This is the sequential text-encoder offload the requirements asked for; it existed only as a side effect before, because diffusers' enable_model_cpu_offload happens to move each component out after use -- so the OFFLOADED path had it and the resident path did not. The embedding argument names differ per family and there is no common base class to ask, so they are read from the pipeline's own signature: Krea 2 takes prompt_embeds + prompt_embeds_mask and needs a second call for the negative, LTX-2 takes prompt_attention_mask and encodes both at once. A family that takes no embeddings at all is left alone rather than having its encoder freed with no way to pass the prompt back. Opt-in per model and settable from the model page, because freeing the encoder costs a host-RAM copy and only pays where the card is tight. The encoder is moved to CPU rather than deleted: the pipeline is cached and the next request needs it again. Any failure returns the request untouched -- a VRAM optimisation must never be the reason a generation fails, and a test holds that. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two bugs of the same shape: a signal that already existed and was not consulted. Same-path siblings. Two cards can share one weights path and differ only in settings -- Wan2.2-I2V-A14B-Lightning and the plain Wan2.2-I2V-A14B both point at Wan-AI/Wan2.2-I2V-A14B-Diffusers, declaring 22 GB and 151 GB. The manager has disambiguated exactly this since the lisa/lisa-32k case, through `requested_id`, and NEITHER placement caller passed it. So the Lightning card was sized by its 151 GB sibling, read as ~127 GB over free VRAM -- far outside the borderline band -- and went straight to CPU offload without trying the card, where it computed at 0% GPU for 26 minutes holding 37 GB of host RAM. Measured: 97.4 GB without the id, 14.1 GB with it, which is the difference between exile and residency on a 24 GiB card. Quantizing the experts was necessary but not sufficient; the ESTIMATE was still the wrong card's. The leak watcher's load signal. _load_in_progress() consulted only the manager's ready event, which diffusers image/video loads never touch -- they load in codai/api/{images,video}.py. A 12-minute multi-shard load therefore looked idle, its monotonically rising RSS looked exactly like a leak, and the mitigation ladder ran against a load doing nothing wrong ("RSS 19.8/43.0 GB LEAK SUSPECTED", every poll). It now also consults the loading reservations, which cover those paths by construction. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
diffusers added `callback_on_step_end` per pipeline rather than across the board, and audio_gen passed it unconditionally. StableAudioPipeline does not take it, so every request raised StableAudioPipeline.__call__() got an unexpected keyword argument 'callback_on_step_end' AFTER the model had finished loading -- one unsupported keyword cost the whole load and read like a broken model rather than a wrong argument. Asked of the signature, not kept as a list of pipeline names: such a list is wrong again at the next diffusers release. The pre-0.25 `callback` spelling is adapted so those pipelines keep their progress, and a pipeline that supports neither says so, because silently losing progress reporting and mid-run cancellation should not be invisible. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Offload: the default becomes auto-borderline, not auto. `auto` condemns a model on its ESTIMATE -- if used_vram_gb exceeds free VRAM it goes straight to CPU offload without ever trying the card -- and that estimate is a measurement from an earlier load in some cards and a hand-entered round number in others (LTX-2.3-PinkCherry declared exactly 24.0 on a 24 GiB card). So a conservative figure permanently exiled a model that would have fit. The cost of being wrong is asymmetric, which is what decides it: guessing "fits" wrongly costs one failed attempt and a fallback, while guessing "does not fit" wrongly costs a CPU copy of every component and a PCIe trip per request for the life of the install -- 28-37 GB of host RAM per video render on a 54 GB box with no cgroup limit to stop it. The dropdown is relabelled by intent and reordered, because the old labels were the trap: `auto` read as the sensible choice while being the pessimistic one. Tests: three files hardcoded /storage/coderai, so collection FAILED outright inside the production image and every test in them had only ever been exercised against the host's dependency set -- which is diffusers 0.37.0.dev0 with no torchsde and no PyAV, against the image's 0.42.1 with both. The four Krea 2 edit tests were silently skipping for want of Krea2Pipeline and now run (15 passed, 0 skipped in-image). Paths are derived from __file__ instead. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pin to native (SDPA) is also the floor on image generation speed, and that is worth stating where it happens. Measured 2026-10-28 on Z-Image-Turbo-IntoRealism, 768x1344, 10 steps, idle RTX 3090: 21.4 s warm, 2.1 s/step. Two of the three things worth suspecting are ruled out rather than assumed -- the card's step count IS applied (the per-request progress endpoint reports total=10) and there is no bnb repack (the checkpoint is fp8 upcast to bf16) -- and the third was the cause: an unnecessary enable_model_cpu_offload from the model page's default offload_strategy, on a model that fits the card whole. 72 s -> 33 s -> 21.4 s. What remains is SDPA, and for this family that is not a choice: Z-Image and Krea 2 mask their text conditioning, and neither SageAttention nor flash-attn 2 accepts an attn_mask (_ATTN_MASK_FAMILIES in codai/api/video.py, measured on LTX-2.3-10Eros which failed on sage then identically on flash). A few seconds per megapixel is the hardware here, not a misconfiguration. The remaining opportunity is named and NOT taken: this pins every image pipeline, including families that never mask and could use a fast kernel. The video path already decides that per pipeline class; doing the same here needs the masking image families enumerated and each measured, because getting it wrong is a 500 at the first denoising step rather than a slow render. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The diarization worker resolved an HF token from the environment or a protected file in the config dir, and gated pyannote repos worked. The MODEL LOADING path had no such resolution: the host runner forwards HF_TOKEN from the host environment, which is only set if whoever launched the container exported it, and on this install it was not -- so huggingface_hub went out unauthenticated for every from_pretrained. That is invisible until a repo is gated, and then it fails as 401 "must be authenticated", which reads exactly like terms that have not been accepted even when they have. On stabilityai/stable-audio-open-1.0 the account had access and the token on disk could fetch the weights with curl, while the engine loading them could not -- so the obvious reading was a licence problem, and it was a credentials-plumbing problem. The resolution moves into codai/hf_cache.py, which already owns how a process talks to the Hub, and is applied from the same entry point as the cache settings: both have to be in place before huggingface_hub is imported, and an install that configured one has configured the other in the same directory. The file/env list is kept identical to the worker's, and the worker now calls it, because two answers to "where is the token" is how one path authenticates and another does not. Its per-request override still wins. Never logged: it ends up in request headers, and a test holds that. Also torchsde, which Stable Audio Open's CosineDPMSolverMultistepScheduler is built on. diffusers imports it lazily, so without it the model downloads its 3.8 GB in full and only then fails at request time, after the wait. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
--oom-score-adj only decides WHO the kernel reaches for once the host is already out of memory. It does not stop a container getting there, and on 2026-10-28 nothing did: an engine grew to 46.3 GB of a 54.8 GB host, pushed 20.4 GB into swap, and the OOM killer never fired at all (oomkill 0) -- the box thrashed through 108 million page-ins until it was reset by hand. A livelock is worse than a kill: a killed engine restarts, a thrashing host serves nobody and cannot even be logged into. --memory alone still permits swapping, which IS the thrash, so --memory-swap is set to the same value: exceeding the limit becomes a kill rather than a slow death. The default is the host's RAM less a reserve for everything that is not coderai, with an 8 GB floor, because a limit under what one model needs would fail every load -- a worse failure than the one this prevents. It sits ABOVE coderai's own softer ceiling (offload.max_ram_gb, which drops caches and evicts idle models at 90% of it), so the app tidies up first and this is only reached when something is genuinely wrong. On the host that prompted it: evict at 38.7 GB, app ceiling 43 GB, container killed at 48 GB, 6 GB left for the OS. The limit is applied only when the kernel can actually enforce it. A limit it cannot is worse than none: it is discarded with a warning on every start while an operator reading "memory limit: 48g" reasonably believes they are protected. That host has no memory cgroup controller at all (cgroup v1, no `memory` in /sys/fs/cgroup), which is also why its OOM killer stayed silent -- with no accounting there is no limit to reach -- so it now says so, and says what to boot with instead. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
LTX-2.x is the one video family here with an audio VAE and a vocoder, so its output is LTX2PipelineOutput(frames=..., audio=...) -- audio generated jointly with the frames and therefore actually in step with them. Only `frames` was read, so every LTX render came out silent and the only way to get sound was to generate some separately with another model and mux that on top, which does not match the picture. The generated track is now muxed in BEFORE the other post-processing steps, because every one of them re-encodes the frames and the track belongs to these exact ones. `add_audio` becomes tri-state so "say nothing" and "say no" differ: unset keeps what the model generated, false gives a silent video, true runs the older separate-generation path. It defaulted to False, which cannot be told from an explicit false -- so a straight default would have dropped the soundtrack. None is falsy everywhere False was, so existing callers are unaffected. Found while testing it: _has_audio tried `import av` under a bare `except: return False`, and PyAV is in no main requirements file -- only the isolated LongCat venv's. On an install without it EVERY file answered "no audio", and because it gates _mux_original_audio, a clip's own sound was silently dropped from every upscale and interpolate. The audio was there, nothing could see it, and nothing said so. ffprobe cannot be missing the way PyAV can, since ffmpeg is what does the muxing, so it is the fallback, and a failure of both is logged instead of reported as silence. Also: one of my own tests asserted a minimum number of ffmpeg progress reports. `-progress` emits on a time interval, so a small encode can finish having emitted a single block -- it passed alone and failed under the full suite. It now asserts what is guaranteed: frames counted, counted upward. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Krea2Pipeline.__call__ takes a prompt and no image, and the family's only other entry points in diffusers 0.42 are the Krea2Turbo modular blocks. So the card with the identity-edit LoRA had its weights ready and no way to be asked "change THIS picture" -- and because _task_of() finds no img2img mapping for the family, an edit request fell through to text-to-image and silently ignored the image it was given. The edit is done IN CONTEXT, not by denoising from the reference: the reference is VAE-encoded once, kept clean, and appended to hidden_states beside the canvas, so identity survives the whole trajectory instead of being worn away as the canvas resolves. That is what the family's edit LoRAs are trained against, and it is why `strength` has no meaning here -- there is no partial noising to control, which also makes _is_instruction_edit classify it correctly straight off its signature. Telling the token groups apart is the whole trick. position_ids is (seq, 3): the canvas keeps (0, h, w) and the reference repeats its grid at plane 1, so the rotary embedding separates "this pixel of the canvas" from "this pixel of the reference" while keeping them aligned. Identical coordinates would make the two indistinguishable. The builder is module-level and pure so it can be checked without the 12B weights, and on a build whose diffusers has no Krea 2 at all -- importing the module never breaks such a build. Built with from_pipe over the loaded components: a Krea 2 transformer is 12B and a second load would mean an eviction on every alternation between generating and editing. The reference is encoded with mode(), not sample(), so the same edit of the same picture does not differ run to run. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
13 (per-request progress), 13b (encode frames + the job log), A4 (72s -> 21.4s per still), plus models.civitai_cache_dir. B2, B3 and B4 were already implemented and are verified live. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
13. A request may name itself (`request_id` in the body, or an X-Request-Id header) and poll /v1/{images,video,audio}/progress?request_id=... for its OWN steps. One global slot per media type meant two callers could not tell their progress apart, so a client stopped reading them altogether and showed no step progress at all while the server was drawing. The name is used as the TASK id, so this is a VIEW over the existing task registry -- which already tracks step/total/status per task under a lock and prunes finished ones -- rather than a second place progress is written. Claimed once, because a request registers a short "loading" task as well and both adopting the id would serve the client whichever happened to be last. Unknown id = 404, so a client stops polling on a definite answer instead of reading somebody else's progress. 13b. The encode reports frames written (`encoding 1200/2400 frames`) across the band it owns, so the longest step of a render stops looking like a stall; and the job's own log, kept all along but unreachable, is served as a tail in the status document plus GET /v1/video/compose/{id}/log?since=N for the whole thing incrementally. The record holds the live list, not a snapshot. A4. 72s per 768x1344 still -> 21.4s, measured on an idle 3090. The card carried `offload_strategy: "model"`, which is the model page's DEFAULT, so enable_model_cpu_offload() ran on a model that fits a 23.56 GB card whole and paid PCIe for it on every request. `auto` loads on GPU and falls back to offload only on OOM. The doc's other hypothesis is ruled out rather than assumed: the live progress endpoint reports total=10, so the card's step count IS applied, and an explicit steps=10 takes the same 21.2s. Also: `models.civitai_cache_dir`, configured like hf_cache_dir and gguf_cache_dir because it is the same kind of thing. Unset, these multi-gigabyte checkpoints land beside the other caches -- on this host a 140 GB volume, which a 13.5 GB checkpoint took to 99% until the next model could not write even its config fetch ("No space left on device"). That is what was actually stopping Z-Image-Turbo-IntoRealism from loading at all. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The whole suite green: 5507 passed, 0 failed. Two real defects came out of it (a refusal that named neither the capability nor the fix, and a PackageNotFoundError where a missing bitsandbytes should be reported), the rest were expectations left behind by changes to the product. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Every one encoded something that had since changed, or depended on a package this checkout does not have. Each was read against the code before being touched, and two turned out to be real defects: - ggml_rpc.available() returned the bare exception, so a model naming RPC servers refused with "No module named 'llama_cpp'" -- true, but naming neither the capability nor the fix. Every refusal now says both. - building a BitsAndBytesConfig without bitsandbytes installed raised PackageNotFoundError from importlib.metadata. The IMPORT is guarded; only construction consults metadata, so a Vulkan-only or CPU install got a traceback naming neither the model nor the setting. Now reported as loudly as an unrecognised mode, because the consequence is the same: full precision. The 2-bit path already did this for optimum-quanto. The rest were stale expectations: - the broker protocol inverted (the client speaks first: one read, no separate ack), gained keepalive pings and an immediate `pending` frame, and renamed its stream envelopes to chunk/done. Added coverage for the status gate that suppresses `pending`, which nothing tested. - /admin/api/status was removed in def78c18 and the front owns status now, so its on-the-way-out coercion had nothing left to coerce. Retargeted to the invariant that survives: entries are well formed when MADE. - main()'s default boot became the front/engine split, so the app instance reaches uvicorn in the ENGINE path now. And four were brittle rather than wrong, each failing for something it was not testing: a 4000-character source window that stopped covering the raise when a comment was added above it (now the function's own source); a hand-written Config stand-in that broke on every new section the product read (now falls back to the real defaults, per section); a generic `selectedId` used as a proxy for "the pipeline UI is gone" that unrelated code then adopted; and a pool test that needed pypdfium2 to exercise what a BUSY pool does. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Section 14 of the Motus list: 14a (AI music never reached compose), 14c (wrap text instead of shrinking it below readable), 14d (a clip's sound from its own file). 14b was fixed in 0.5.12 by making an image load visible to the eviction ladder. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
14c. A text overlay was one drawtext line sized from the frame height, so a long text ran off a vertical frame, and a pre-wrapped text was drawn as a single line because an explicit newline was never a break. `overlays[].fit` takes none (the default, today's behaviour) | wrap | shrink, and a newline now breaks whatever the fit. Wrapping emits one drawtext per line, placed as a single left-aligned block measured from the widest line, rather than letting each line centre itself ragged. `captions.fit` takes shrink (the default) | wrap, folding a too-wide row's trailing words onto a new row instead of shrinking that event -- on a vertical video shrinking can go below readable. Word order is preserved, so the karaoke highlight still walks the words in the order they are spoken. Both fitters now MEASURE the result instead of scaling the size by the width ratio. Advance is very nearly linear in size, which is why the shortcut looks right, but hinting rounds each glyph's advance to whole pixels and the error compounds over a long string: a 95-character line computed to fit at size 8 measured 440 px against a 300 px budget. A test caught it. _fit_size carried the same assumption and shrink is its default, so it is fixed the same way. 14d. `visuals[].audio_src` names a separate audio-only file -- DASH sites serve picture and sound as separate streams, so the best video of a clip often has no audio track and a caller that cannot mux had to skip it. The picture path is deliberately left alone: stage 3 reuses it as the visual's video, and an audio-only file there would render a scene with no image. Lengths in audio_mix="clip" come from whatever carries the sound. Also: the dialogue tests no longer depend on a TTS model being installed. They test how a dialogue scene is VALIDATED and say so -- their portraits are inline "so the test does not depend on what is saved on this install" -- but three of them failed standalone with 501 "narration needs a TTS model" without reaching the validation they exercise. Only the inventory is faked, not the check. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
audio_generate answers an AudioGenerationResponse, a pydantic model, and _generate_music read it with .get(). Every compose asking for AI music raised "'AudioGenerationResponse' object has no attribute 'get'", warned "rendered without music", and then SUCCEEDED -- silent. That is why it lasted: the render never failed, so only someone watching the finished video would notice, and "AI composed" music has been off in production the whole time. Both shapes are accepted, since the same call is reachable through paths that answer a plain dict, and a non-dict item is coerced rather than assumed. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Four fixes found from one question about two stray ffmpeg processes: - an audio track is built at its length instead of padded up to it, so the mix cannot hang at 100% CPU (4/10) or silently come out a clip short (3/14) - an ffmpeg we start can no longer outlive us (PR_SET_PDEATHSIG) - an image load reserves its VRAM for its load window, so it is visible to the eviction ladder instead of handing out memory it has already taken - the dtype default is chosen from a device that has been detected, so a text model stops loading fp32 on a CUDA card -- twice the VRAM, and no Flash Attention, which is why Qwen3.5-9B answered nothing since 2026-09-30 Plus `precision` is settable from the model page, where the quantization it belongs with already was. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
run() kills its child on cancel and on timeout, but both need this process to be alive to act. SIGKILLed -- an OOM, a container stop, an interrupted test run -- no finally block runs, the child is reparented to init, and nothing owns it any more. That is not a dormant process: a filter graph that never reaches EOF spins at 100% CPU indefinitely. Two orphans from interrupted test runs were found on a dev host having burned 15 h and 11 h of CPU, with no parent to attribute them to. PR_SET_PDEATHSIG has the kernel kill the child as the parent dies, so the outcome stops depending on our cleanup running at all. Best effort, a no-op where prctl is unavailable, and tested by SIGKILLing a real parent rather than by asserting the flag is set. Also bound the audio mix at 20x its own length with a two-minute floor. Mixing a few seconds of PCM takes milliseconds, so the two-hour video backstop meant a wedged mix sat for two hours; it now gives up in two minutes. Plus a test that every name identifying a whisper.cpp runner reaches it -- the runner's id, a shared alias, and the weights path /v1/models advertises, by full path, basename and stem -- and that naming the weights spreads across a shared alias instead of serialising a pool configured for capacity. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`precision` was read by the loader and listed in main.py's _LOAD_AFFECTING, but it was in neither the admin save allowlist nor the model page, so the one setting that decides how much VRAM a card actually needs could not be configured anywhere. The dtype then came from a default derived from GPU detection -- and a card's precision is a property of the card, not of the hardware it happens to land on. The select sits with the quantization boxes because that is where it is read wrongly: 4-bit and 8-bit quantize the WEIGHTS, while the embeddings, lm_head, the norms, the activations and the KV cache all stay in this dtype, so a 4-bit model left in fp32 still pays fp32 for every one of them. The hint says so, and says that fp32 also rules out Flash Attention. "" is stored as null, so "not configured" stays distinguishable from a value and the loader's own default still applies. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
NvidiaBackend.__init__ leaves self.device None. The loader picked the precision default as `'fp16' if self.device == 'cuda' else 'fp32'` at line 897 and only detected the device at line 1029, so the FIRST load in a fresh engine process defaulted to fp32 on a CUDA card -- while a second load in the same process inherited 'cuda' from the first and quietly got fp16. The same model loaded at two precisions depending on load order. fp32 costs twice the VRAM of the dtype asked for, which on a 24 GB card is the whole headroom argument for quantizing in the first place: Qwen3.5-9B is carded load_in_4bit with an estimate of 7.795 GB and measured 14.403 GB, a 1.85x gap that is the fp32 tell. FlashAttention also refuses fp32 outright, so transformers warned twice at load, cast the inputs back to fp32 at inference, and the model answered every request with "FlashAttention only support fp16 and bf16 data type" -- and before that error was surfaced at all, with an empty completion. It had been unusable since 2026-09-30. Also resolve flash_attn by PRESENCE rather than truthiness: `config.get(k) or global` cannot express "off", so a card saying `flash_attention: false` fell through to the global default. Not the cause here (the global is false too), but the same class of bug one layer up. Co-Authored-By:
-