- 30 Oct, 2026 27 commits
-
-
Stefy Lanza (nextime / spora ) authoreddownload_model wrote straight to its final cache path with no try/except and no .partial, and the check guarding it tested existence only. So any interruption left a file that was then returned for ever as a valid cached model, and that URL could never be fetched again. Measured in one cache directory on zeiss: 188 zero-byte `<sha256>_<name>.gguf` entries accruing about 14 a day since 2026-10-16, against 2 non-empty files. And one of those two was a 9.72 GB ltx-2.5-22b-distilled-Q4_K_M.gguf where the real file is 13.58 GB — a truncated download the existence-only check would have served as a complete model. The empty ones break a load loudly; the truncated one loads. Now: the bytes go to a .partial and are renamed only once they are all there, an empty or short transfer raises instead of being kept, and the partial is removed on BaseException — not Exception, because a KeyboardInterrupt or SIGTERM mid-download is exactly how several of these were made. A zero-byte entry found in the cache is treated as a miss and deleted rather than served. The cache LOCATION is deliberately untouched: moving it would be a silent full re-download for every existing install, and a test pins the resolution order. Where it points on a given box is a deployment question — on this one a root symlink chain (~/.cache/coderai -> /AI/coderai -> /storage/coderai/) aims it at the repo's gitignored models/ dir, which is a separate matter from this bug. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
-
Stefy Lanza (nextime / spora ) authored
Our llama-cpp-python 0.3.30 vendors llama.cpp e3a74b299 (2026-06-16). The fix for this exact failure landed five weeks later: b4aa7dd47 mtmd : use align_corners for qwen3vl vision position embedding interpolation (#25781, 2026-07-21) 0.3.36 vendors 0c1e57098 (2026-10-01), which contains it. Wrong position-embedding interpolation makes the vision features spatially meaningless, which is why Qwen3.8-27B described every picture identically rather than merely badly, and why a gemma4v projector -- a different graph, untouched by that bug -- read the same images correctly through the same pipeline. Everything measured fits: the projector pairs correctly (projection_dim 5120 = hidden_size 5120), loads, and injects tokens the model notices, since it says so when no image is sent; two publishers' conversions fail identically because the defect is in the consumer not the artifact; and the image-token floor is irrelevant -- tested at 1024 and at 0, byte-identical answers both times. The bump only takes effect on a rebuild (REBUILD_LLAMA=1), and it must stay a local source build: the CMAKE_ARGS carry -DGGML_CUDA=ON -DGGML_VULKAN=ON -DGGML_RPC=ON together, and a PyPI wheel has at most one backend. Until a rebuild runs, mmproj stays off the qwen38 cards and vision goes to gemma-4-31B-heretic. Also records the method failure, because it cost most of the investigation: three conclusions were drawn from reading headers and code and all three were wrong. The floor was dismissed with "the symptom predates it" -- which was never checked, and could not have been, since the floor is applied automatically whenever the projector type matches. Two projectors were then compared while the one variable this repo introduced was held constant in both. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The previous commit said the capability should go to gemma-4-14b-8k or Qwen3.5-9B and implied more finality than had been earned — neither had been tested at that point. Both now have been, on the same two images and the same prompt: gemma-4-31B-heretic 'Red square.' / 'Blue circle.' gemma-4-14b-8k 'Red square.' / 'Blue circle.' Qwen3.5-9B 'red square' / 'Blue Circle' qwen38 the same sentence for both gemma-4-31B-heretic is the one to reach for: larger than the broken 27B, already on disk with its own projector, exact on both images. That also narrows the fault to the qwen3vl_merger graph alone. gemma4v is correct at two model sizes, and Qwen3.5's own vision tower is correct when transformers runs it instead of llama.cpp — so it is not the model, not the family, and not our pipeline. And a warning that matters more than the finding: the llama-cpp-python here is a LOCAL SOURCE BUILD with -DGGML_CUDA=ON -DGGML_VULKAN=ON -DGGML_RPC=ON, which is the entire reason it exists. I had noted 0.3.36 is current on PyPI in a way that reads as "pip install -U", and a PyPI wheel carries at most one backend: installing one would take out the nvidia engine, the radeon/Vulkan engine and the cluster's GGUF-over-RPC path together, to chase one projector. The upgrade is a version bump in packaging/linux/Dockerfile.update-llama plus update_oci_image.sh REBUILD_LLAMA=1. Written down so nobody reaches for pip. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Qwen3.8-27B served images for months and answered identically for all of them. Four plausible explanations were wrong, two of my own inferences were wrong, and only an experiment settled it — so the whole thing is written down rather than left as a line in a changelog. Cleared with evidence: the projector pairs correctly (projection_dim 5120 = the model's hidden_size), the missing deepstack tensors are missing from unsloth's reference conversion too so this family does not use them, clip_graph_qwen3vl IS implemented in libmtmd, the base model genuinely is a VLM with vision_config and image_token_id 248056, the GGUF tokenizer carries all three vision markers at the ids the base config names, the mmproj does load, coderai's vision guard does fire, and the 1024-token floor is llama.cpp's own requirement. Also wrong, and worth recording as a trap: "no mtmd output at request time, so the handler is bypassed". The llama.cpp log callback is deliberately swapped to a quiet one after the load, so runtime silence proves nothing. What settled it was two experiments. gemma-4-14b-8k with its own projector reads the same two images correctly ('Red square.' / 'Blue circle.') and invents when given none — so the pipeline is sound end to end. And the same Qwen model with a DIFFERENT publisher's projector fails identically, so it is not one bad artifact. The fault is upstream: either llama.cpp's qwen3vl_merger graph is wrong for this model, or both converters mislabel it — the base is qwen3_5 while the projector declares qwen3vl_merger, so the wrong graph may simply be running. mmproj is removed from the qwen38 cards. A model that answers confidently and wrongly is worse than one that declines, and the listing now says text only. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An engine took SIGSEGV mid-sweep and every model resident on it died with it. _is_key_busy() is what stops eviction unloading a model under a live request — its docstring says so — and it reads two sources: the instance pool's ref counts and external_busy. An embedding request appeared in NEITHER. It takes no pool ref, and nothing on that path called external_busy, whose only caller in the whole codebase was the lipsync engine. To the evictor, an embedding model was permanently idle. 12:27:41 Loading Qwen3.6-14B-A3B: need 9.3 GB, have 4.8 GB free — evicting 12:27:41 [req] POST /v1/embeddings model=Qwen3-Embedding-4B 12:27:42 Response status: 200 12:27:42 On-request VRAM eviction: unloading ACTIVE model 'embedding:…Qwen3-Embedding-4B-Q4_K_M.gguf' backend.model=None 12:27:44 engine#1 exited (code -11) code -11 is a segfault, not the -9 OOM-kill everything else this week has been about: the native model was freed while a request was still dereferencing it. The 502s on /v1/embeddings that had been appearing for hours were this. The guard wraps the whole region from having the model to being done with it, not each inference call: there are three such functions across seven call sites and several early returns, and the window that must be protected is the whole of it. The key is request_model's `<model_type>:<resolved_name>`, which is exactly what the evictor logs and _is_key_busy looks up — marking busy under any other spelling would be a no-op that reads like a fix. A test asserts every inference call sits inside the guard. Its first version matched substrings and flagged `_may_embed_images` — a name-and-config capability check that runs before the model is even loaded — so it now matches exact names. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
measured_n_gpu_layers is reused as a cap so a reload jumps straight to the split that fit instead of rediscovering it through OOM retries, and measured_quant_sig is what stops such a measurement outliving its configuration. It covered quantization only. Setting swa_full=False on a gemma card cuts its KV reservation by about 19 GB — its 25 sliding layers stop reserving the full 178k context and hold their 1024-token window instead, at no quality cost. The learned split of 18 of 30 layers was measured with the old full-SWA KV, and nothing in the signature changed, so it would have survived and pinned the model to CPU on a card that can now hold nearly all of it. That is the layer-ratchet bug again, arriving through a setting the signature was not watching. n_ctx, swa_full, cache_type_k/v and kv_offload now form part of it. They are not "how big the weights are", which is what the docstring used to claim the hash was for, but they decide how much of the card is left for the weights and therefore how many layers fit. offload_kqv is read as a spelling of kv_offload because the loader accepts either, and a card written one way must not keep a measurement taken the other. The field list is deliberately narrow: over-invalidating discards every measurement and restores the OOM ladder on every load, which is the whole cost this mechanism exists to avoid. A test pins that both ways. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The MTP repair shipped in 0.5.37 fired on a real model during a capability sweep and made the failure worse. llama.cpp validates per-layer metadata arrays against block_count, and a metadata override can only carry a scalar — bool, int, float or str — so there is no way to shorten the array to match. Correcting the count therefore does not fix the load, it only changes which error it fails with: before missing tensor 'blk.64.attn_norm.weight' after key qwen35.attention.recurrent_layers has wrong array length; expected 64, got 65 The first message names the real problem. The second sends the reader looking for a corrupt array. A repair that only changes which error you get is not a repair, so it now declines when any per-layer array is present and the honest message stands. Declining silently would be its own trap — the shape still looks repairable from outside — so unrepairable_reason() explains it before llama.cpp speaks: which array pins the count, that an override cannot carry a shorter one, and the two ways out (a quant that keeps its NextN block, or a llama.cpp that derives the layer count from nextn_predict_layers). "Per-layer" means length EQUAL to block_count, so a tokenizer's 248,320-entry token list does not veto a repair it has nothing to do with. The test fixture can now write array-valued metadata, which is what it would have taken to catch this before shipping rather than from a live sweep. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
XTTS reported no voices after seventeen successful syntheses, and the reason was that the loaded and remembered paths had never worked for any engine since they were written. Every backend declares `voices` as a METHOD; both callers read it as a sequence: [str(v) for v in (getattr(backend, "voices", None) or [])] A bound method is truthy, so the `or []` never fired, iterating it raised TypeError, and both callers caught that and fell back to "nothing known". No voice was ever recorded or returned, and the cache file had never been written once. Kokoro and bark hid it behind their static lists; XTTS and parler, having none, published zero for ever. The tests passed because their fake sets `voices` as a list attribute — a shape no real backend has. A new test asserts the contract against the real classes through inspect rather than against a fixture, which is the only version of this test that could have failed. With that fixed, 58 XTTS voices were recorded and the listing still showed none: they were written under "XTTS-v2", the id the request used, and looked up under "coqui/XTTS-v2", the id the listing publishes. The lookup now falls back to matching the last path segment — the idiom the coqui speaker lookup already uses — with an exact key still winning and distinct models still distinct. The existing never-breaks-synthesis test then caught a gap in the first fix: a backend may expose `voices` as a PROPERTY, so merely reading the attribute runs its code and can raise. The getattr is inside the guard for that reason. Live after this: XTTS-v2 58 voices, Kokoro 54, bark 130, parler 0 with voice_selection "described", which is the honest answer for an engine steered by a text description. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two models could not be loaded in one afternoon, and establishing why took a diagnosis each. Neither was a memory problem; both failed late and expensively, after other models had been turned out of the card to make room. The second one is the reason for this. A 27B "Flash" quant passed every check a careful operator would make -- right architecture, all 65 blocks present including the MTP one, an mmproj beside it -- and stored 402 of its 866 tensors in ggml type 143, a rotation-folded format from a fork. Nothing in the metadata announces that: the type ids live in the tensor index, which no error message reads back to you. It arrived as a generic load failure after the eviction, six retries and a full read of the weights. gguf_preflight answers it from the header alone and runs before the VRAM need is even computed, so a load that was never going to work costs no other model its residency. The bound is llama.cpp's own GGML_TYPE_COUNT rather than a list kept here, so it tracks the installed library. Only ids at or above that bound are refused, and the conservatism is the design. The Python binding's GGML_TYPE_* constants are an INCOMPLETE mirror of the compiled enum -- 0-3, 6-29, 39-42 -- so 30..38 have no constant, and 30 is BF16, which files we serve today use and which loads perfectly. "Absent from the binding's constants" would have rejected every one of them. An id inside the enum that the build still cannot handle is left to llama.cpp to refuse in its own words. The verdict gets its own exception type because the on-demand path turns a failed VRAM reservation into a printed warning and carries on -- correctly for a reservation, fatally for a verdict, which would then reappear a minute later as the original confusing failure. That path now re-raises GgufUnreadable, the same treatment HostRamExhausted already gets, and the comment three lines below it ("a host-RAM refusal is the decision itself and must reach the caller") is why. It reports 501, not 503: evicting and retrying cannot change the answer, so a client that retries a 503 forever must not be invited to. The type ids ride along in the response, being the actionable part. The architecture is printed before the load too, so llama.cpp's "unknown model architecture" sits beside the file it came from. Deliberately NOT a list of supported architectures: there is no API for it, and guessing from the shared library's strings would invent a second source of truth that can be wrong in the direction that blocks a model which would have worked. No fourth GGUF parser -- the tensor-type histogram rides along on the walk gguf_mtp.read_layout already does, and a test asserts gguf_preflight has no parser of its own. docs/serving-a-model-this-box-cannot-hold.md records the three independent requirements, how to verify each on a new machine, the quant/size table, why an mtp-*.gguf is not a model, and which of the three things called "MTP support" already exist (vLLM's raw --speculative-config passthrough, ds4's --mtp) versus which does not (a GGUF draft model, which llama-cpp-python cannot express). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
cutlet, XTTS's Japanese phonemiser, pulls mojimoji — a C++ extension with no cp313 wheel, so it compiles when the isolated TTS venv is built. The build asked for clang++, because the bundled Python records clang++ as its C++ compiler and these images have no clang. They had no C++ compiler at all: g++, c++ and clang were all absent, /usr/include/c++ did not exist, and only libstdc++.so.6 was there without its dev symlink. So gcc's presence — added for Triton's runtime JIT — looked like it covered this and did not. g++ is added to the same apt lists for the same reason gcc is in them: not to build the image, but because something compiles at RUNTIME inside it. It brings /usr/bin/c++ and the C++ headers with it. Done in Dockerfile.oci-venv (the one build_oci_image.sh actually builds), Dockerfile.oci, and the pod capability-base. The pod XTTS profile had the same undeclared-phonemiser gap as the server venv, which its own comments already describe for click — so pypinyin, cutlet, fugashi and unidic-lite are declared there too, and its deep check now imports both phonemisers and constructs a Cutlet, because the failure they cause is invisible at install and at boot: the pod answers /v1/models with all seventeen languages and then refuses two of them per request. Verified on zeiss after installing these: 17 of 17 XTTS languages return audio, and cutlet romanises Japanese through a working fugashi dictionary. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Probing XTTS's seventeen languages after the truncation fix: fifteen spoke, and the two with non-Latin scripts failed on dependencies coqui-tts does not declare. zh-cn -> ImportError: Chinese requires: pypinyin ja -> ModuleNotFoundError: No module named 'cutlet' So the venv installs cleanly, reports success, advertises all seventeen, and then refuses two of them at synthesis time — the same shape as the torch that was never installed, which is what pinned this file in the first place. They are now declared in requirements-xtts.txt and in the _PINS copy the worker falls back to when the file is not mounted (a test keeps the two in step). cutlet romanises through fugashi, which needs its dictionary at RUNTIME and not only at install, so unidic-lite is declared too — the chain requirements-melotts.txt already notes for MeloTTS. pypinyin is pure Python and is installed on zeiss now, so Chinese works there. Japanese needs a build: cutlet pulls mojimoji, which ships no cp313 wheel, so it compiles — gcc and the Python headers are both in the image, but the install has to run where build deps are available. It is declared here so an image build picks it up. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Probing XTTS-v2's seventeen languages through the running endpoint: sixteen returned audio and zh-cn failed with KeyError: 'zh'. Our own call site did it. `language=lang[:2]` truncated every code to two characters, and zh-cn is the only XTTS code longer than that, so the single language with a regioned code was the single language that could not be spoken. Resolution now goes through XTTS's own declared list, which also fixes the other direction: a client sending a locale ("en-us", "pt-br", "ZH_CN") lands on the code XTTS actually has instead of a near miss. tools/xtts_service.py had the identical truncation. It runs in coqui's own venv and cannot import the server package, so the list is duplicated there — and two tests keep them honest: one compares the declared lists, the other executes the worker's own normaliser and asserts it answers exactly as the server's does. Also updates the bark voice-count test, which pinned the English-only list at ten. That assumption was the bug. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Probing all nine of Kokoro's languages through the running endpoint, one request each: eight returned audio and `fr` returned a 500 — "espeak-ng has no such language". French is regioned in espeak exactly like English (fr-fr, fr-be, fr-ch), so the bare code every OpenAI-compatible client sends was refused. The French voice had never worked. The regionless map's own comment asserted the opposite: that "it", "fr" and "de" are all accepted bare. Two of those are true. A test asserted the same thing for all three, so the belief was pinned in place without ever being measured — exactly the failure the Kokoro-English fix was about, one language further along. "fr" now maps to "fr-fr"; the comment says not to add a code on the strength of the pattern, because the pattern is what got this wrong. Measured bare-OK while I was there, so the next person has data instead of a guess: de, it, nl, pl, tr, ru, ko, ar, cs, hu, sv. The corrected test asserts those, and a new one checks the invariant that broke here — every language the listing advertises for Kokoro must resolve to something espeak was measured to accept. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Languages landed but the voices did not: kokoro reported none until something loaded it, so a client could see nine languages and still have nothing to pick from. The names are now read out of the voices file the backend itself selects from — the same file, so the listing cannot offer a voice that synthesis would then reject — and the accent map already in the module decodes each one's language and gender. 54 voices across all nine languages. Pointedly NOT via _resolve_kokoro_files: that downloads a 28 MB release when nothing local matches, and /v1/models is a polled endpoint, so it must never be a download trigger. A file that is not on disk simply yields no names, and there is a test that fails if the listing path reaches for the network at all. The zip index read is memoised against the file's own mtime (27ms -> 0.1ms). A list a load actually enumerated still outranks the file, and a card's voices_path is honoured. XTTS keeps filling in after its first load, and parler stays empty with selection "described" — it has no speaker names to give. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Every TTS model published `languages: null` on /v1/models while the STT models published theirs correctly. Nothing in the server restricted the languages — it simply never named them, so a client was told nothing about what any voice could say and could only discover a language by trying it. That is how XTTS-v2's seventeen languages got taken for English and Italian: those were the two somebody tried. Each set is read from the engine's own files rather than assumed — XTTS from its config.json, bark from its 13-language speaker bank, parler from its model card, and kokoro derived at call time from the accents already encoded in its voice ids, so adding a voice family cannot leave its language behind. An engine whose languages depend on the checkpoint (melo, the generic transformers fallback) claims nothing: a wrong list is worse than none, because a client that believes it sends text the voice cannot pronounce and gets noise back with no error. A `languages` list on the model's own card still wins. Bark's voice list was the ten English speakers, which is where "bark is English" literally came from; it now covers all 13 languages, English first so a client taking voices[0] as its default still gets an English speaker. The voices are published too, so a voice picker no longer needs to load a model to ask it — and since /v1/models is polled, the voices cache is memoised against its own mtime rather than re-read per model per poll. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Four settings decide whether a GGUF load fits in host RAM at all, and none of them could be set from the model page — only by editing models.json by hand. The placement check, how sliding-window layers reserve, the measured size the check acts on, and the MTP-block repair are now on the page with the rest of the KV cache, gated to GGUF models, and in the savable list that is where a field quietly stops persisting. measured_kv_gb has been written by hand as a {n_ctx: gb} map and is now journalled as a plain number, so the form renders either shape instead of "[object Object]". The MTP override now also sets nextn_predict_layers to 0 rather than correcting block_count alone. Builds disagree on whether block_count includes the NextN blocks: the one this was written against does not subtract them (it asked for blk.64 of a 65/1 file), and a build that does would read a corrected 64 as 63 and quietly leave a real layer out of the model. "64 blocks, none of them NextN" is true under both readings. The journal also takes its directory from platform_paths rather than inventing an environment variable, so a deployment keeps one home. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A model configured cache_type_k/v=q8_0 at n_ctx=128000 got a 47.7 GB KV cache placed in host RAM and the engine was SIGKILLed mid-load, taking every other model resident on it down too. The placement plan that exists to prevent exactly that skipped itself, because any pin — type or location — was read as consent to whatever size resulted. Quantized KV at a long context is the configuration someone writes because they are worried about size; it is the one that most needs checking. A pin is now honoured without switching the check off: a pinned type is never requantized, a pinned placement is never moved, and when the remaining levers cannot make it fit the load is refused with a message naming the pin. That refusal needs a measured size, and the measurement was being taken after a successful load and printed as advice to copy into the config by hand. A KV cache big enough to need the plan is big enough to kill the process before the load ends — so there was no "after", nobody was reading, and the engine died, got respawned, and repeated the identical load forever. It is now journalled the moment llama.cpp names the size, before the allocation is faulted in. measured_kv_gb still wins; the journal only answers when the config was silent. Separately: a quantizer that drops a checkpoint's NextN (multi-token-prediction) blocks without lowering block_count makes llama.cpp ask for a block the file does not have, reported as a bare "missing tensor 'blk.64.attn_norm.weight'" — which reads like a corrupt download. When the file's shape proves that is what happened (blocks contiguous from 0, exactly nextn_predict_layers short), the declared count is corrected via llama.cpp's own kv_overrides. A file missing blocks for any other reason still fails, because a truncated download must not be silently reinterpreted into a model that loads and answers nonsense. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
LongCat had two settings that look independent and are not. A quantised text encoder MUST be resident -- bitsandbytes quantises as weights land on CUDA and cannot shuttle them back, and _place_pipeline returns early for a quantised encoder without consulting the offload mode at all. So "quantise the encoder" and "spill when VRAM runs short" are one decision: asking for both pins 22.1 GB of a 24 GB card (13.6 DiT + 8.0 nf4 encoder + 0.5 VAE) and leaves under 2 GB for the activations of a 13.6B video model, which OOMs at the first long clip rather than at load and so reads as a flaky model. offload_strategy now accepts "auto", which takes that decision once, at load, and logs the arithmetic: * resident -- DiT, VAE and a quantised encoder all on the card, no swapping; * spill -- DiT and VAE resident, encoder on the host, the two swapping for encode_prompt so only one is ever on the card. The decision is made against the card's CAPACITY -- total VRAM less a context reserve -- and deliberately not against the free VRAM at that instant. Free VRAM makes the placement depend on whichever model happened to be resident when the request arrived: a sibling holding 8 GB would park LongCat on the host, and it would stay there for every later request, because nothing re-examines the condition. It is also self-fulfilling -- the worker RESERVES the footprint and the manager evicts what is in the way, so "no room right now" is not a fact about the card but about a decision nobody has taken yet. A model that could be resident is never left on the CPU merely because something else got there first. What is reserved is what the chosen placement OCCUPIES, not what residency would need: on a 24 GB card that is 14.1 GB (DiT + VAE), because reserving the 22.1 GB residency figure would evict siblings to free 8 GB this placement never touches. Both numbers are reported, so the one the decision was made on stays visible. The worker and the service run the SAME pure arithmetic over the same inputs, so they cannot disagree, and the spawn moved inside the acquire loader: it used to run before the reservation, so the service loaded its weights onto a card the manager had not cleared yet and the eviction meant to make room happened afterwards. Fine for a 13.6 GB DiT, wrong for any placement that wants the card. Every load now logs what it actually cost against what the plan predicted, so a constant that drifts shows up as a wrong prediction instead of as an OOM at generation time. And it is selectable: the model page's Offload list had no "auto" entry, which mattered more than it looks, because the page sets that select from the saved value -- a card reading "auto" would have displayed as the first option and the next save would have silently rewritten it to "". The text-encoder labels now say which levels need a card bigger than 24 GB, since that is what decides whether the quantisation is honoured. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
**A seeded request to an offloaded pipeline failed as a load error.** Sequential CPU offload leaves every module on the META device until its own forward pulls it in, and a pipeline whose modules are all on meta reports meta as its device. torch.Generator(device=pipeline.device) then reached torch's accelerator hooks with a meta device: RuntimeError: META device type not an accelerator. The image route reported that as "Failed to load image model 'Tongyi-MAI/Z-Image-Turbo'" -- naming the load, which had in fact succeeded (footprint measured, 1.56 GB offloaded to host RAM) -- so the model read as unloadable. And only a SEEDED request failed, so it looked broken to anyone who passes a seed and perfectly fine to anyone who does not. diffusers already knows the answer: _execution_device is where an offloaded pipeline actually computes. models.utils.generator_device() prefers it, rejects meta and lazy, and falls back to cuda/cpu. Four sites constructed a Generator from a pipeline's own device -- text-to-image, img2img, edit, and audio generation -- and fixing one would have left the same failure reachable by three other routes, so all four go through it. **Four working edit models were reported as broken because of their capability list.** The Qwen-Image-Edit siblings carried image_generation and modes: null, while diffusers maps no text-to-image class for the family and the request path says so outright: 501 QwenImageEditPlusPipeline cannot generate from a prompt alone: diffusers offers no text-to-image pipeline for this family. The runtime was right and the capability list was wrong, which is the wrong way round. The instruction-edit branch of the heuristic lumped two different kinds of model together: FLUX.2 is unified -- one checkpoint that generates from text AND edits -- while Qwen-Image-Edit, FLUX.1 Kontext, Step1X-Edit and the Rapid-AIO fine-tunes are edit-only. They are split now, so only a unified family claims image_generation. And the listing publishes the directions. `modes` is how a client learns a model has no text-to-image path without sending one and reading a 501, and most cards never set it; when a card says nothing, the modes are now inferred from the capabilities the model already publishes (t2i/i2i, t2v/i2v/v2v). Inferred, never invented: every mode is backed by a flag that was already there, and a card that states its own modes always wins. Verified live while testing this: Wan2.2-I2V DOES serve a prompt-only request (it returned an mp4), so that row was collateral from the VRAM contention rather than a missing path. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Publishing several ids for one model is deliberate: an alias can stand for a round-robin group, and a typed id coexists with the checkpoint it names. What was missing is any way for a client to tell which ids are ONE model. Measured on a live install: 206 published ids over 119 card entries -- 136 aliases, 20 absolute paths whose basenames were also listed, and 50 repo ids, with no exact duplicates among them. A capability sweep walking that list reported seventy "new models" it had not tested, all of them names it had already covered, and had no way to know. So each entry now carries: canonical_id the same string on every id belonging to one model, so grouping by it is sufficient; alias_of set ONLY on the non-canonical ids, so a client that skips the entries carrying it tests each model exactly once; aliases on the canonical entry, the other names it answers to. Two cards that share a checkpoint but differ in CONFIGURATION are not the same model. They keep separate canonical ids and stay separate entries, because they are two deliberately different setups and a client must be able to see both -- grouping them would hide one. A bare path only one of them can claim (ids are unique in a listing) groups with the card that claimed it, which is the honest answer: sending that path reaches that card's settings. Nothing is invented. A canonical name that this listing filtered out -- parked as disabled, or assigned to another engine -- is never pointed at; those ids are left standing on their own rather than referred to a name the server will not answer to. _group_ids lives above the class, not between two methods: added there first, it ended the class body, and 126 methods silently became module functions with list_models among them. A test asserts it stays out. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.5.32 shipped the image-token floor and broke the projector load it was meant to fix: mmproj : failed to load projector (name 'ImageTokenFloorHandler' is not defined); continuing text-only handler() referred to the class as a global while the class was produced by a module-level __getattr__, and PEP 562 consults that only for attribute access from OUTSIDE the module -- never for a global name lookup inside it. So the lookup raised NameError, the except around the mmproj block caught it, and the model came up text-only: it then refused images instead of answering them wrongly. Better, and still broken. The factory existed for exactly this; handler() now calls it (cached in _HANDLER_CLASS) instead of naming the class. And the test that let this through was testing the wrong call: it built the class with _make_handler_class() directly, so it passed against a shipped path that raised. It goes through handler() now, with a second test asserting the class is reachable the way the backend reaches it. A probe that does not call what the code calls proves nothing -- the same lesson as the vision-mapping fix in 0.5.27, learned again in the module next door. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A GGUF vision model answered every image question fluently and wrongly. On Qwen3.8-27B, with its own matching projector loaded and llama.cpp reporting "has vision encoder": * a 448x448 image added 16 tokens to the prompt (61 -> 77). The same image through the HF path added 156. * two completely different pictures -- a yellow field with a blue triangle and a green square, and three white circles on black -- produced byte-identical descriptions at temperature 0, neither resembling either image ("the image shows a series of exclamation marks"). * the same prompt with NO image answered differently, so the pictures were reaching the model and only their content was not. llama.cpp had been printing the reason on every single load: load_hparams: projector: qwen3vl_merger load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024 mtmd_context_params carries image_min_tokens and image_max_tokens, and nothing set either, so the count came from the image's own size alone: sixteen tokens, which cannot describe a photograph. The model did the only thing it could and answered from the prompt, with complete confidence. A 200 carrying a fluent wrong answer is the worst failure shape there is, because no caller can tell it from a right one -- and this one survived a full capability sweep, reported as "answers don't match the picture". So the floor is set: from the card's image_min_tokens when it says, and otherwise from the projector's own documented requirement, read from the mmproj GGUF's clip.projector_type rather than guessed from a filename (an mmproj is routinely renamed to sit beside the model it serves). Projectors with no documented floor are left exactly as they were, because forcing an upscale costs prefill on every request. The figure is printed at load, since it changes what every image costs. Upstream builds its context from mtmd_context_params_default() and sets four fields on the result, with no hook in between and the token bounds not among them. Rather than copy that method -- which would silently stop tracking any field upstream later adds -- the handler wraps the default-params factory for the duration of that one call, so the struct upstream receives already carries the bounds and everything upstream does to it still happens. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Every whisper.cpp model answered with text and nothing else -- no segments, no words -- while the HF and Vosk backends on the same endpoint returned both, which reads as whisper being the broken one. Two reasons, in sequence: the server was never asked for a verbose reply, and the route passed a hard-coded empty segment list to _finalize whatever came back. So the server is asked for verbose_json (one extra form field, ignored by a caller that wanted plain json), and the reply is mapped in both shapes it comes in: OpenAI-style "segments" in float seconds, and whisper.cpp's own "transcription" entries whose "offsets" are MILLISECONDS -- read as seconds those would place a 2-second clip 2000 s in. Per-word entries are picked up from the top level or from inside each segment, wherever a DTW build puts them. Nothing is invented: a field the server did not send stays missing. And when word timings were REQUESTED but none came back, that is now said out loud, because whisper.cpp emits them only with DTW enabled and a silent empty list is indistinguishable from a model that found no speech. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two completely different pictures sent to a GGUF model with a configured mmproj produced byte-identical descriptions, while the same prompt with NO image answered differently. So the images reached the model and their content did not -- the embeddings carry nothing -- and the log said nothing either way: projector init, media-marker substitution, bitmap decode and chunk eval all log under names ('mtmd', 'clip', 'bitmap', 'media', 'projector') that the load-phase filter did not have, so every one of them was discarded. Reproduced with two synthetic images (yellow background with a blue triangle and a green square; black background with three white circles) at temperature 0: identical answers, neither resembling either picture, both unlike the no-image control. Keeping these lines is what makes the next request say which step is losing the pixels. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
**Five models "won't load" and llama.cpp had said why.** The load-phase log callback printed only lines matching a keyword list about layers, buffers and devices. llama.cpp writes its diagnosis -- an unknown architecture, a missing tensor -- in none of those words, so it matched nothing and was dropped, while llama-cpp-python raised its one generic sentence: "Failed to load model from file: <path>". That string is all the client got, and all the log had. So every line is now kept in a 60-line ring, lines that look like a diagnosis are printed whether or not they mention a buffer, and the raised error carries llama.cpp's own words. A model this build genuinely cannot read is no longer indistinguishable from one that did not fit, a bad path, or a corrupt download. The retry ladder stops earlier too. It exists to answer one question -- does it fit? -- by moving layers to the host, and an attempt with NO layers on the GPU has already answered it. Three MTP/next-N checkpoints each burned six attempts down to n_gpu_layers=0, every one failing in a fraction of a second with the same message: 18 loads whose only effect was to push the reason further up the log. (What those three have in common is visible in their metadata -- qwen35.nextn_predict_layers=1, blk.N.layer_output_scale.weight -- and with the reason no longer discarded, the next attempt will say so outright instead of leaving it to be inferred.) **XTTS and parler were handed an alias instead of a repo id.** _load_backend passed `model_path or model_name` to Kokoro and the bare `model_name` to everyone else, and for a card registered under an alias that name is the alias. Each isolated worker then failed in its own vocabulary, neither mentioning the cause: XTTS: "XTTS venv could not load 'XTTS-v2': ValueError: not enough values to unpack (expected 4, got 1)" -- coqui splits a model name into type/lang/dataset/model, and the bare alias has no separator at all. parler: "parler-tts-mini-multilingual is not the path to a directory containing a file named config.json" -- the org is missing, so the real repo is never looked up, and the worker reports a model that IS cached as one to download first. Every backend that resolves a checkpoint now gets the path. The XTTS worker's own translation table is fixed to match on the last path segment as well, since it is reached by other callers too: it required a "/" before, which is precisely the case that does not apply. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Four unrelated rows from an end-to-end pass over every capability, with one thing in common: in each case the information needed to get it right was already in the server and something declined to read it. **An image card with no precision loaded at f32.** The default here was f32 and only --no-ram mode coerced it down, so whether a model occupied 2 bytes per parameter or 4 came down to an unrelated flag. Measured on one run: CyberRealisticPony and wai-illustrious (no_ram=True) loaded at 7.49 GB, while PonyRealism and pornmaster-pro -- same SDXL architecture, no_ram=False -- measured 14.98 GB, exactly double, filled a 24 GB card and failed with "CUDA out of memory. Tried to allocate 12.00 MiB". The inflation then spread to the estimate, where the f32 precision factor turned 14.98 GB into a 30.9 GB need and triggered a round of eviction that could not possibly help ("this model needs more than the whole card"). Nothing was gained: these checkpoints are published at half width. The default is now bf16 -- bf16 rather than fp16 because it keeps f32's range, the same reason the --no-ram branch gives. An explicit f32 on a card is still honoured. **LTX refused the width and height.** Every family in video_geometry declares a side_multiple -- 64 for Wan's VAE cell, 32 for H3 -- and nothing anywhere read it; the request path never consulted the table at all. So 848x480 reached LTX-2.5 and came back "`height` and `width` have to be divisible by 32 but are 848 and 480", with the server making the caller do arithmetic it had the numbers for. Sides are now snapped to the nearest legal cell, and LTX has a family of its own: it had none, so it inherited Wan's 16, under which 848 was already "legal" -- snapping to the wrong multiple is indistinguishable from not snapping. The new entry differs from the Wan fallback in exactly the one field a refusal proved. **Kokoro could not speak English.** espeak-ng names English by region and refuses the bare code, while every OpenAI-compatible client sends "en" -- so the Italian voices worked and the English ones raised `RuntimeError: language "en" is not supported by the espeak backend`, as a 500, which reads as a broken model. Regionless codes espeak rejects are mapped ("en" -> "en-us"); codes it accepts bare ("it", "fr", "de") pass through untouched; anything it still does not know is now a 4xx naming the language and what to send. **A pinned model landed on the wrong engine.** by_name() matched an engine by NAME or by BACKEND in one pass and returned the first healthy hit. A GGUF-isolation split runs two engines on one card and both declare the backend "nvidia", so a transformers model pinned to "nvidia" could resolve to the GGUF sibling and be refused for a capability the engine it actually named has: "model 'longcat-avatar' is pinned to 'nvidia' but engine 'nvidia' (backend 'nvidia') can't serve a 'transformers' model (capabilities: ['gguf'])". The sibling fallback could not recover it either -- it appends "-gguf" to the PIN, so it tried the same engine twice and never the one named "nvidia". An exact name is unambiguous, so it now wins, in its own pass. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A load is admitted to VRAM by estimate: ensure_vram_for sizes the model, evicts what is in the way, and failing that a CUDA OOM is caught and reported. Host RAM had no equivalent, and the kernel supplies its own: four engines SIGKILLed in five hours on a 54 GB host, each one while a model staged its weights. Oct 29 23:03 Killed coderai-nvidia-gguf anon-rss 46.3 GB Oct 30 03:10 Killed coderai-nvidia anon-rss 47.7 GB Oct 30 03:36 Killed coderai-nvidia anon-rss 31.6 GB Oct 30 03:49 Killed coderai-nvidia anon-rss 32.7 GB front: engine#0 exited (code -9); respawning (1/5 in 120s) The RAM watcher was awake for all of it and logged "RSS 50.6/43.0 GB (118%) -- mitigation [gc+trim+evict_idle] -> 53.7 GB": every rung of its ladder releases memory already held, and none of them can decline the load doing the allocating. SIGKILL then takes the engine rather than the request, so the client that asked for the oversized model saw a dropped connection -- and so did every unrelated request in flight, plus everything arriving during the 120-second respawn. One bad load reads as the whole server being unreliable. So offload.ram_admission (default on) adds the missing half. codai/models/ host_ram.py sizes a pending load -- measured host residency if the model has loaded before, the whole footprint for an offload strategy that asks for it, the spill past free VRAM otherwise, plus a staging allowance -- against the tighter of MemAvailable and the headroom under max_ram_gb, less a floor kept for the OS and the sibling engines. ensure_host_ram_for() runs the existing pressure ladder FIRST and re-measures, so refusal is the last step; when it comes, it is a 503 naming the model, the estimate, the budget, the shortfall and what to change on the card. A model whose footprint cannot be determined is never refused on a number nobody has. And the proximate cause of the biggest of those kills was a sizing step that reported itself as skipped: (auto-offload sizing skipped: cannot access local variable '_ck' where it is not associated with a value) The GGUF auto-offload branch reads the configured KV-cache types to size its headroom, from a binding made ~110 lines further down where llama_kwargs is assembled. Every load that reached the branch raised UnboundLocalError, the handler called it "skipped", and nothing reduced n_gpu_layers -- so llama.cpp was asked to keep every layer of a 35B on a card that could not hold them, and answered with a 43,562 MiB CPU KV buffer. The read is hoisted above its use (tested on source order, not a line number), and the handler now says the load is UNSIZED, because "skipped" invites the reader to assume nothing needed doing. Separately, and found while checking which cards were really misconfigured: a repo whose model_index.json declares a pipeline class diffusers does not have is now detected and loaded with its own code when allow-listed, or refused by name when not. The allow-list and the mechanism both already existed, but only fired when a card named a custom_pipeline -- nothing ever did, so MOSS-Image and Kromoss failed with "module diffusers has no attribute MOSSImagePipeline", which names neither the cause nor the remedy and reads as a broken model. Co-Authored-By:
-
- 29 Oct, 2026 13 commits
-
-
Stefy Lanza (nextime / spora ) authored
Reported from production within the hour: `Qwen3.5-9B` now REFUSES pictures, "served without a vision tower" — the exact opposite of what the vision loading was for. transformers' `_LazyAutoMapping.get` is not `dict.get`. Its signature is `get(self, key, default)` and `default` is REQUIRED, so AutoModelForImageTextToText._model_mapping.get(type(cfg)) raises TypeError. `is_vision_checkpoint` caught it and answered "no vision class" — for every checkpoint, always. Every vision model therefore loaded as text, and then had its pictures refused by the guard added for exactly that case. The probe I verified it with passed two arguments, so it tested a call the code does not make. Two tests now call what the code calls: one with a stub whose `get` mirrors the required `default`, and one through the real mapping with a config class this transformers actually ships. The silence was the other half. `is_vision_checkpoint` returned False without a word, so the engine log said nothing at all and the only symptom was a 400 that blamed the model's configuration — which was fine. `vision_report` now carries the reason at every step, a checkpoint that looks like a vision model and is not loaded as one says so in the log, and the refusal quotes what was actually found rather than listing three things it might be. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Adding `defaults` to the Z-Image-Turbo entry showed they would have been ignored anyway. The manager keeps runtime kwargs FLAT and the models.json entry under `_raw_cfg`, so only keys that `build_runtime_kwargs` promotes exist at the top level — and `defaults` is not one of them. `model_defaults()` read the flat dict, so on the image path a card's steps and guidance were silently dropped: `/v1/models` published them (it reads the entry) while generation used the generic 30 steps at CFG 7.5. A client that read the listing and trusted the server to apply it was wrong twice over. Which means `Z-Image-Turbo-IntoRealism` (10 steps) and `Krea-2-RealDream` (8) had never had their cards applied, and `call_kwargs` sits in the same place — MOSS-Image's `uncond_skip_layers` was never passed either. `card_view()` merges the entry under the flat kwargs, with the flat ones winning a collision because they are what the loader was given, and both readers go through it. The video path had already hit this and merged by hand at its call site; `images.py` never did, and one of the two being right is what made it invisible. Same lesson as the `gpu_split` and `sequential_text_encoder` promotions, three lines apart in that same function. The new test walks the installed catalogue and asserts, for every card carrying `defaults`, that what the listing publishes is what generation would use. Also: a TTS engine's venv no longer reports itself ready on the strength of the engine package alone. `pip install coqui-tts` brought ninety dependencies and no torch, the venv looked complete, `/v1/audio/speech/capabilities` said `env_ready: true`, and the first synthesis failed on `import TTS`. `torch` is in every family's list now. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pre-load check went through AutoConfig, which reaches the network — 20 s on the first call against 0.8 s once local — and with trust_remote_code EXECUTES the repo own config code, another 2.7-5.6 s, to read two plain JSON fields. A validation that can hang is not a validation. It now reads config.json from what is already on disk (a local directory, or the hub cache via try_to_load_from_cache) and judges from vision_config, model_type and architectures: 0.1 ms, no network, no remote code. A model whose config is not here yet is still not judged — the load fetches it and the check on the loaded object decides. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three faults, two of them reported from a production render on 2026-10-29. **A local GGUF embedder was fetched from the Hub.** `gme-Qwen2-VL-2B-Instruct-Q4_K_M` and `dinov2-large-Q8_0` answered 500 Failed to load embedding model: 404 Client Error … huggingface.co/sentence-transformers/<id>/resolve/main/ for files sitting in /AI/guffcache. The loader dispatches on the NAME — `.gguf` goes to llama.cpp, or for a ViT to the dinov2-embed server, and anything else to sentence-transformers, which prepends `sentence-transformers/` to a name with no slash and asks the Hub. These entries are registered under three ids (path, basename, basename without extension) and the request used the last. The config lookup missed for the same reason, which is worse than it looks: it carries `mmproj`, so the vision GGUF would have lost its image tower even with the dispatch right. Both now resolve through the entry. **An impossible request paid for a model load before being refused.** Four texts and four images to a text-only 8 GB embedder: forty seconds of loading, then the 400 it could have had immediately — and a caller cannot tell that from a hang. `_may_embed_images` answers from the config and the HF config before anything loads, and only when something positively says "no image tower"; what it cannot determine still goes to the load, where the exact check on the loaded object decides as before. **XTTS installed no torch.** `pip install coqui-tts soundfile requests` brought coqui-tts 0.28.0 and its ninety dependencies and no torch at all, so the install reported success and `import TTS` failed. It has a pinned requirements-xtts.txt now, with the same torch/torchaudio pair as parler. Both TTS engines also stopped hiding the reason: venv_env.import_ok and venv_env.gpu_conv_ok carry the real message out, /v1/audio/speech/capabilities serves it as env_detail, and the XTTS 501 no longer claims no venv was found over a venv that is there and populated with the wrong stack. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
torchaudio has no cp313 build before 2.6.0: No matching distribution found for torchaudio==2.5.1+cu124 (from versions: 2.6.0, 2.6.0+cu124, 2.7.0, …) so the pair the CrisperWhisper venv uses cannot be reproduced here. Both move to 2.6.0+cu124 together, which is what matters: a torch and a torchaudio that disagree is the failure this pin exists to prevent. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The pinned install succeeded — torch 2.5.1+cu124, parler_tts 0.2.2, transformers 4.46.1 — and `import parler_tts` still failed, with the worker reporting only parler-tts install did not yield an importable package which names neither the package nor the reason. It now reports the import error itself. The reason pip left behind: parler-tts pulls in descript-audiotools, which requires an unpinned torchaudio, and pip resolved 2.11.0 against torch 2.5.1. torchaudio is pinned with torch now, in the file and in the inline pins — a pair of versions that must match is not something to leave to a resolver. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The GPU probe shipped a moment ago did its job on the first live run: it caught the cudnn failure and asked for a rebuild. The rebuild then stopped with Parler requirements file missing: /opt/coderai/app/requirements-parler.txt The container mounts `codai/`, `tools/` and `third_party/` over the image, so a repo-ROOT file added since the running image was built is simply not there — and a worker that heals itself must not stop for a reason that has nothing to do with the venv. The same mistake was already on record as ACE-Step reporting a missing `requirements-acestep.txt` for a venv that was fine. The file stays the source of truth and wins whenever it is present; the same pins also live in the worker for when it is not, with a test that parses the file and compares, because two copies of a pin drift. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
With the environment fixed, Parler got further: the venv built, the service started, the model loaded — and the first synthesis died in the DAC audio decoder's conv1d. RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed ptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED Every cudnn sublibrary present in the venv, and the stack unable to load them: a torch 2.15 with the CUDA-13 wheels that `pip install git+…parler-tts` had resolved on its own, because nothing pinned it. requirements-parler.txt names the build now, the way every other isolated worker's file does. The health check was the deeper problem. `import parler_tts, soundfile` answered True for that venv for ever, so nothing ever rebuilt it. A venv's check has to exercise what the worker actually does: the decoder is convolutional, so the probe is one conv1d on the GPU, and a venv that imports but cannot run it is rebuilt from the pinned file. A CPU-only host passes — there is no GPU stack there to be broken. The service also latches cudnn off if it still fails there. torch has native kernels for those convolutions, the decoder is a fraction of the work, and the alternative is no speech at all. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two production failures from the same render, both fixed at the level they belong to. **Speech engines "listed but not installed".** XTTS answered 501 "no venv containing it was found" against a venv that was there and populated; parler answered 500 "python -m venv returned non-zero exit status 1"; a worker's torch died with CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED. Three different failures, one cause: the engine processes export PYTHONHOME=/opt/coderai/python, every worker was launched with env=dict(os.environ), and a venv interpreter started under a foreign PYTHONHOME takes its stdlib and site-packages from THAT installation — so the venv's own packages are not on sys.path at all. The import probe was the first casualty, which is why it reported the venv as missing. codai/api/venv_env.py — which already existed for exactly this, and which the TTS/STT workers were not using — is now the one place a worker's environment and its venv are decided. Every call INTO a venv goes through it: the probe, `python -m venv`, each pip install and the service itself (a clean service with a poisoned probe still reports the venv as broken; a poisoned pip installs into the wrong tree). LD_LIBRARY_PATH is decided per venv, because a foreign python must not see coderai's library directories while a venv built from the server's own interpreter needs its libpython — and sys.prefix no longer counts as "ours" when it IS /usr, which on a pip install would strip the driver path from every worker. venv_usable() replaces four private copies of "can this venv be used". It catches what none of them did: a venv whose python runs perfectly and imports nothing, because its 875 MB of packages are installed for a different minor version (an xtts_venv built by the host's 3.13 pointing at the container's 3.12). `pyvenv.cfg` spells the version two ways; both are read. An intentional overlay (--system-site-packages, for the H3 venv that inherits the server's torch) is respected in both directions. And `GET /v1/audio/speech/capabilities` now reports needs_venv / env_ready / env_detail, from the filesystem alone — never importing into the venv, never loading a model — so a client knows before it offers a voice whether the first request will spend minutes building the environment. **"CrisperWhisper out of GPU memory during compose", every scene.** word timestamps failed (500: … CUDA out of memory. Tried to allocate 86.00 MiB …) The speech model was loaded and idle between scenes while the TTS and visual models of the same job loaded around it and filled the card. 86 MiB is what a resident model needs to run one window, and nothing in the system was keeping it: every reservation is taken at LOAD time, and this model was already loaded. Every caption in that render lost its word timings. Now three numbers per speech model, all measured: the weights, what one pass needs on top of what was already allocated, and their sum. The per-pass number is ensured free before each pass (manager.ensure_free_vram) and is a DELTA, not the process peak — the peak would charge a speech model for a 16 GB image model resident beside it, and every later load of it would evict the world. An OOM that still happens frees the card and retries once, on both the in-process and the isolated-worker path. The in-process reserve was also a third of the truth: 3 GB hard-coded for a whisper-family model, where the isolated worker serving the same checkpoint used 8 GB and said in its own comment that 3 GB was about a third of the real peak. Both paths now call one function, so eviction frees the same amount whichever way the model is running. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A model listed as able to read images has to read them, or say it cannot. `Qwen3.5-9B` was listed with `image_to_text` and answered every photograph with "a blank white image with no visible content" — nine real pictures of one video, indexed as blank. Two causes, one symptom: AutoModelForCausalLM -> Qwen3_5ForCausalLM (text only) AutoModelForImageTextToText -> Qwen3_5ForConditionalGeneration The engine asked for the first, so the tower was never built, and with no projector the chat path flattened the picture to the literal text "[image_url content]". The model was answering the prompt it was given. Now the class comes from the checkpoint's own config, every load path in the CUDA backend goes through that resolver, a vision model also loads the processor that makes a picture promptable, and an image request takes its own generation path (no KV prefix reuse — image tokens come from the pixels). Where a model genuinely cannot see, the request is a 400 rather than a confident wrong answer, and a GGUF with no mmproj stops advertising the capability it cannot serve. Two things found while measuring the image path on a live fleet: * every image load printed "Proactive eviction skipped: name 'request' is not defined". `_load_diffusers_pipeline` has no `request`; the NameError was swallowed as a skip, so the minimum-eviction block never ran and each load fell through to the blunt eviction that unloads models which would have fitted alongside. * `Z-Image-Turbo`, distilled for 8 steps at CFG 1, was served at 30 steps and CFG 7.5 because its card carried no `defaults` — four times the work for an over-guided picture, ~72 s a still. A card that says nothing now falls back to what the model's own name says before the generic floor. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Six findings from one 21h30m production log (548,687 lines, 51 loads of 12 models). They share a cause: nothing measured a model directly, so every decision downstream of a measurement was made on a number that was wrong. 1. The footprint was a process-RSS delta --------------------------------------- `measured_ram = self._get_own_ram_gb() - ram_before`, where _get_own_ram_gb sums RSS over the engine and every child. RSS counts mmap'd weight-file pages the loader just walked; summed over a tree it counts pages SHARED between those processes once per process; and as a delta it absorbs anything else that moved in the window. Against ground truth: gemma-2-9b-it.Q4_K_M 5.37 GB on disk -> reported 30.81 GB (5.7x) LTX-2.5, six loads, one unchanged config: 22.11, 22.11, 23.28, 4.50, 19.02, 29.80 GB (6.6x) Every other model sat at 1.0-1.2x, so this was not noise. Wan2.2's 3.4x spread was the real quantization fix and is the one that belonged there. codai/models/footprint.py measures the object instead. Quantization needs no correction factor, which is the point: a bitsandbytes Params4bit IS a uint8 tensor, so numel x element_size already gives packed bytes. What those formats hide is counted explicitly -- bnb's quant_state (absmax, code book, nested double-quant state) lives outside parameters(), and GPTQ/AWQ keep qweight/qzeros/scales in BUFFERS, so a parameters-only walk reports a 4-bit model as nearly empty. Tied weights are deduped by storage pointer; accelerate meta placeholders are sized from their dtype and flagged as declared, keeping the placement-independence the field already documents; a diffusers pipeline is walked through its components, since it is not an nn.Module. Context is costed properly: KV from the KEY/VALUE head count, not the attention head count (4x on a GQA model), with sliding-window layers sized to their window. But a computed KV is only a fallback and says so -- this codebase already learned that "predicting KV from GGUF metadata is architecture-specific ... and was off by 5x in BOTH directions", so measured_kv_gb for the context in question always wins. A GGUF, which no walk can reach, falls back to its file size; it is loaded as stored, so the file IS the weights. 2. A failed load overwrote the good measurement ----------------------------------------------- Device map: cpu: 1 layers Measured footprint: 4.50 GB total Saved measured_vram_gb=4.505 (force_vram_update) generation FAILED: StopIteration interacts badly with generators Full-GPU load looks viable (4.5 GB peak need) <- 4 seconds later A load that placed one layer and raised still reached the recorder, and force_vram_update wrote it to disk as truth. Now a measurement that is implausible on its own (~0 GB) or that collapses to under half of what the SAME quantization signature previously measured is treated as a partial load and discarded. Wan2.2's genuine 40.04 -> 11.73 GB drop is let through because it arrived WITH a signature change, which is exactly what tells the two apart. The rejection names which of the two checks fired. The ratio guard alone was not enough. Walking a pipeline and finding that NO component holds any weights is a failed load outright, and there is nothing to fall back to -- a size taken from process memory measures everything except the model in question. That case is now refused before any estimate is considered, and the refusal is recorded in the history with its reason rather than left looking like a successful load. The hole showed up as one of the new tests passing alone and failing in the full suite: the fallback delta grows with whatever else the process has loaded, so the figure it invented depended on test order. 3. The teardown-leak breaker never looked at host RAM ----------------------------------------------------- It registers under `if _tens.is_cuda` and sweeps under `_obj.is_cuda`. A pipeline loaded with enable_model_cpu_offload keeps its weights in HOST RAM, and 17 of 21 video loads took that path, so none of those weights were tracked and nothing looked for them. With nothing loaded and 482 MiB of VRAM in use: coderai-nvidia 33.14 GB PSS_Anon 30.85 GB anon in 6588 mappings; 6565 of them under 100 MB (20.25 GB) Thousands of mid-sized allocations is retained tensors, not allocator slack -- which is why gc and malloc_trim ran every 15 s for hours with the figure never moving, and why the ladder then evicted healthy models to chase it: RAM eviction: unloading 'video:Lightricks/LTX-2.5-Diffusers' to free host RAM (RSS 40.0 GB > cap target 36.5 GB) codai/models/teardown.py adds the host arm, by weakref rather than by storage pointer: a freed host address is readily handed back out by malloc, so a pointer match could null a live tensor that by then belongs to another model. A referent still alive after teardown IS this pipeline's leak, identity-exact. It runs before the allocator trim, since trimming while the weights are still referenced returns nothing. 4. The RAM cap counted shared pages and page cache -------------------------------------------------- codai/models/proc_mem.py reads Pss_Anon + Pss_Shmem from smaps_rollup, which has already divided shared pages by their sharers and excludes reclaimable page cache. Measured correction on a 5-engine host: 40.06 GB (RSS) -> 37.91 GB (PSS). Modest -- the leak above was the real problem -- but a cap should chase memory that freeing an allocation can actually release. Falls back to the old reading rather than to 0.0, which would read as "no memory in use". 5. A model nobody used stayed loaded ------------------------------------ Ondemand mode - keeping 'audio_gen:ACE-Step' in VRAM alongside new model It held 7.9 GB of a 3090 and 2.1 GB of host RAM for hours after its last request. offload.idle_unload_seconds (default 1800) unloads a model untouched that long. Deliberately generous: reloads are the expensive event -- that day spent 2h04m of 21h30m inside loads -- so a short timeout would trade a VRAM saving nobody asked for against a reload everybody waits on. A key that was never stamped is left alone rather than assumed idle. It runs before the leak-watch gate, so turning that off does not silently disable this. 6. The log left no room for the history anyone needed ----------------------------------------------------- 32,633 of 33,009 requests were embeddings, and one engine produced 426,682 of the 548,687 lines -- 78% -- filling the 2,000-line view in about four minutes. The per-item endpoints are now summarised to one line (model, count, shape) instead of dumped, which also stops a caller's content being written to a shared log. Prompts are deliberately NOT truncated: a 4,000-character script prompt is what identified the music defect below. GET /v1/models/history replaces inferring swaps from a log that rotates: time, event, model, engine, plus seconds, vram_gb, offloaded_gb, strategy for a load and reason, freed_gb for an eviction. Eviction reasons are passed at the call sites. The front merges across engines, because a swap is a cross-engine event -- one engine evicts so a co-located one can load -- so one engine's copy is half the story. Also: ACE-Step was asked for a song with no words ------------------------------------------------- Motus sends a music_prompt and no lyrics, and empty lyrics do not mean instrumental: the pipeline then conditions on a single padding token (lyric_token_idx = [0], lyric_mask = [0]) and the model still tries to sing, which comes out as babbling over the backing track. ACE-Step's own UI documents the way to ask -- "Use [instrumental] or [inst] to generate instrumental music" -- and every genre preset it ships ends in either "male vocals, raw vocals" or "instrumental". Both are now sent when no lyrics are given. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`_last_used` is stamped in exactly one place: the `active_in_vram` setter, whose comment calls it the "single chokepoint for 'this model was just used'". `add_model()` goes through it. Seven sites in video.py and images.py wrote `multi_model_manager.models[key] = obj` directly and did not, so for every video pipeline and four image models: - `_lru_order()` sorts by `_last_used.get(k, 0.0)`, so the key sorted FIRST and was evicted ahead of genuinely older models -- permanently, however recently it had been used; - the first eviction pass skips `key == self.active_in_vram`, which an unstamped key never matches, so that protection never applied either. Both ladders iterate `_lru_order()`: the VRAM one and the cross-engine `cosite_evict`. Observed on a production run -- a co-located engine needed room and released the freshly-loaded video pipeline before a TTS model that was older but had gone through `add_model`: [cosite-evict] releasing 'video:Wan-AI/Wan2.2-TI2V-5B-Diffusers' ... [cosite-evict] releasing 'tts:Kokoro-82M' ... Also disambiguate the evict-debug line that led here. It read `getattr(model_obj, 'backend', '∅')`, which returns the default both when the object has no such attribute and when `model_obj` is None -- so a healthy diffusers pipeline (which never carries `.backend`) and a key that had already vanished from `self.models` printed an identical `backend=missing`. Only the latter is a fault. Name the two apart, report the object type, and compare the sentinel by identity rather than `==`, which a tensor-bearing object need not answer with a bool. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
8303b38e passed `resolved_name` to _quant_signature from record_vram_delta, which has no such variable. The NameError was swallowed by an enclosing try, so measured_vram_gb persisted and the signature beside it never did. Verified live 2026-10-29: after a real load, measured_quant_sig was None and the invalidation fired zero times. Scope was checked at the READING site and assumed at the writing one. They are different functions. The deeper fault was hashing the model name at all. The two sites do not have the same name to hand -- the reader has a resolved path, the writer only a config key -- so even bound, they could hash different values and every comparison would be a mismatch, discarding every measurement. The signature is stored on one model's own entry, so identity is already implied: it now hashes only precision, load_in_4bit/8bit and component_quantization. That also drops pipeline_cache's cache version from the hash, which would have invalidated every measurement on an unrelated cache bump. A test now walks both call sites and asserts each binds what it passes. Co-Authored-By:
-