- 31 Oct, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authoredThe footage checker scores a photo against a text description, and of everything configured only `gme-Qwen2-VL-2B` could do it: the rest are text-only (Qwen3-Embedding, bge-m3), image-only (dinov2, salad, vpr) or coordinates (geoclip-location). Its Q4_K_M sibling crashed the engine, which is a separate bug in the GGUF unload path. Three of the four need no new embedding code, which is why their documented API was read before any was written: gme-Qwen2-VL-7B-Instruct Qwen2-VL weights -> the existing `qwenvl` backend, detected on model_type rather than repo name royokong/e5-v SentenceTransformer.encode(texts | images), 4096-d BAAI/BGE-VL-MLLM-S2 SentenceTransformer.encode({'text':…, 'image':…}) `_embed_images` already routes a sentence_transformers model through `model.encode(pil_images)` with a `[{'image': im}]` fallback — exactly the two shapes those models document. nvidia/MM-Embed does need its own backend, because its API resembles nothing else here: model.encode([{'txt': …, 'img': …}], max_length=4096, is_query=True, instruction=…)['hidden_states'] `trust_remote_code` is passed by the loader rather than taken from the model entry: the checkpoint's encode() lives in its repo, so without it the model cannot load at all, and a missing operator flag would be a silent failure. An `instruction` switches it to query mode — without one it embeds the item as a PASSAGE, which is the right default for "embed this" and the wrong one for a query, so the distinction is kept rather than hardcoded. NOT claimed: coderai embeds texts and images as separate items, so MM-Embed's combined text+image QUERY form is not reachable through /v1/embeddings. Text-vs-image similarity in one space — what the footage check needs — works regardless. One mistake caught in passing: 'geoclip' was briefly added to the image-capable backend list. Its two modes SHARE that backend name, so it would have claimed that geoclip-LOCATION accepts images when it takes coordinates. Removed, with the reason recorded beside the list. timm reaches the image at last. It has been declared in requirements.txt since the depth/segment endpoints were added and absent from every incrementally updated image — the same overlay gap as geoclip and vosk, found the same way, by listing the built image's packages rather than reading the source tree. jina-clip-v2 cannot load without it: its EVA vision tower IS a timm model. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
-
Stefy Lanza (nextime / spora ) authored
**Eviction headroom.** Every size reaching eviction is an estimate, and one landing a few hundred MB short does not fail honestly: the load is admitted, starts allocating, and dies on a tiny buffer. Measured on zeiss across 2026-10-30/31, each on a load eviction had just reported as satisfied: Tried to allocate 20.00 MiB … 8.69 MiB is free Tried to allocate 190.00 MiB … 18.69 MiB is free Tried to allocate 12.00 MiB … 8.69 MiB is free Eviction now targets `needed_gb * 1.10`. The headroom is what it TARGETS, never what it reports: the shortfall warning and the "needs more than the whole card" verdict both still measure the caller's actual request, so freeing enough for the load but not the whole margin stays silent. A warning on the padded figure would fire on every successful eviction and bury the one real warning. 10% of the request costs at most one more idle model, and scales — a flat margin would be too small for a 30 GB model and absurd for a 0.1 GB one. **JSON mode returned JSON that would not parse.** 21 script generations ended `finish_reason=length` after writing 6000 tokens of well-formed-but-unfinished JSON for a job needing ~800; one ran 8.5 minutes. Nothing was misconfigured: max_tokens was the caller's own 6000, the prompt 2414 tokens, the model's n_ctx 32000. `json_object` IS honoured, but its grammar constrains syntax only and never requires the root object to close, so an array of scenes grows until the budget ends mid-token. Two changes, neither touching the caller's max_tokens: * `json_schema` is now accepted, which bounds the SHAPE so the object must close. llama-cpp-python takes only `type: json_object` and reads a sibling `schema` key, while OpenAI nests it under `json_schema` — so the shapes are translated. Forwarding OpenAI's form verbatim would have silently DISABLED the grammar (`_grammar_for_response_format` returns None for other types), which is worse than not supporting it at all. * When the budget is hit anyway, the open structures are closed so the caller gets valid JSON holding the scenes that did arrive. It drops a half-written element rather than closing around it, never invents content, parses its own output before returning it (an unverifiable repair is not applied), leaves prose and already-valid JSON alone, and does not rewrite `finish_reason` — which stays `length`, so a caller that checks still learns it was cut short. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An eviction freed 16.3 GB and the load that triggered it saw 0.0 GB. 23:58:05 Loading 'image:…pornmaster…': need 30.9 GB VRAM, have 2.9 GB free — evicting models to make room 23:58:05 unloading active model 'FLUX.2-klein-4B' (measured 17.24 GB) 23:59:23⚠ Eviction freed 13.4 GB (now 16.3 GB free, needed 30.9 GB) 23:59:25 needs ~32.9 GB host RAM (30.9 GB footprint against 0.0 GB free VRAM spills 30.9 GB to the host, plus 2.0 GB to stage it), and only 27.8 GB is available — short by 5.1 GB. Refusing the load. Two seconds after eviction reported 16.3 GB free, admission read 0.0, and no other load intervened — the next one starts a second later. `_get_free_vram_gb` subtracted every in-flight reservation, including the one belonging to the load being judged: `max(0, 16.3 - 30.9)`. A reservation exists so OTHER loads do not count VRAM already spoken for. Subtracted from the reserving load's own view it says the room it just reserved does not exist, so the whole footprint looks like a host spill, the host-RAM budget refuses it, and the eviction that preceded it cost another model its residency and bought nothing. That is where 17.24 GB of resident FLUX pipeline went, twice over, in the hour that ended in the first OOM kill. `exclude_key` leaves one model's own reservation out of the subtraction, and the admission path passes its own key. Every other caller — eviction, sizing, placement — keeps the conservative figure, which is correct for them: they are deciding on behalf of a load that has not reserved anything yet. Verified by reproducing the arithmetic: with the old view the need is 32.9 GB and the shortfall 5.1 GB, matching the server's own message to the decimal; with the load's reservation excluded the same load needs 18.6 GB and is admitted. NOT the fix that was proposed first. Checking admission BEFORE evicting would have been worse: at that point 2.9 GB were free, so it would have refused sooner. Eviction HELPS the host-RAM verdict by shrinking the spill — more free VRAM, less spill, smaller host requirement, monotonically. It only has to be visible to the load that paid for it. A test pins that ordering claim so the earlier idea cannot be reintroduced as an improvement. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Fixing a regression I shipped hours earlier in d83075a4. Found in production by the floor misfiring, on the same night: [vram-est] key='image:black-forest-labs/FLUX.2-klein-4B' source=used_vram_gb(disk-floor) -> 25.61 GB | used_vram_gb=17.0 Loading 'image:black-forest-labs/FLUX.2-klein-4B': need 25.6 GB VRAM, have 0.0 GB free — evicting models to make room FLUX.2-klein-4B declares `used_vram_gb = 17.0` and really occupies 17.24 GB -- the eviction log prints the measured figure when it unloads it -- while its cached repo is about 23.7 GB. A diffusers cache holds every component (transformer, two text encoders, VAE) and routinely duplicate formats and variants, .bin beside .safetensors and fp32 beside fp16, of which one subset loads. So the repo on disk is not a lower bound on VRAM, and the floor raised an already-accurate figure past the 24 GB card. Each load then announced "this model needs more than the whole card", evicted the resident fleet, and gained nothing. The two errors are not symmetric, which is the whole lesson. The under-estimate the floor prevents made ONE text model attempt a load that could not fit. An over-estimate on an image pipeline costs OTHER models their residency, on every load, while the model it was protecting was already sized correctly. So the floor now applies only where it was measured to help: single-checkpoint text and vision models, never image or video pipelines. It reached a running engine because the engine was OOM-killed and respawned against the bind-mounted tree, so a commit made minutes earlier went live without a deliberate restart. Worth remembering when reading a log from a box whose code is bind-mounted: a crash is also a deploy. Also: the size test wrote two real gigabytes and failed with OSError on a host whose root filesystem was 98% full. It truncates a sparse file now -- the helper reads st_size and never the contents. Co-Authored-By:
-
- 30 Oct, 2026 36 commits
-
-
Stefy Lanza (nextime / spora ) authored
Auto: Qwen/Qwen3.5-9B is quantized — trying full-GPU regardless of the 3.1 GB estimate (5.6 GB free) Error loading model Qwen/Qwen3.5-9B: … needs about 3.1 GB of VRAM to itself; 5.6 GB is free after evicting what could be evicted. That reads like a bug in eviction -- 5.6 is more than 3.1 -- and is not. The estimate was wrong in the dangerous direction: too SMALL, so the loader believed the model fitted three times over, attempted a full-GPU load of something needing ~6.5 GB, and failed. The refusal itself is correct and stays: a quantized model cannot be served from CPU offload, and would answer every request with an empty completion. The chain, from the debug line: source=used_vram_gb -> 3.13 GB | used_vram_gb=7.795 measured_cfg=8.67 load_in_4bit=True prequantized=False quant_mult=0.300 `measured_vram_gb=8.67` -- the real 4-bit footprint -- was correctly discarded, recorded under a different quantization signature; that guard earns its keep elsewhere and is untouched. The fallback `used_vram_gb=7.795` is a STORAGE-size baseline that the quant factors apply on top of, so it became 7.795 x 0.300 + reserve = 3.13 GB. The multiplier is right: the checkpoint is 19.31 GB of bf16 with `quantization_config: None`, verified on disk, so `prequantized=False` is also right. 7.795 was not the size of anything. The post-download backfill writes `cached_size * 1.2` and runs from the download thread, so an interrupted or still-running fetch leaves a figure for a fraction of the checkpoint: 6.5 GB downloaded x 1.2 = 7.8. So the field can hold a number nobody checked, and the factors then scale it down again. The checkpoint on disk is measurable fact available at that moment, and is now a floor -- UPWARD only, with a 1.25x margin, so an operator's larger value stays theirs and an honest small difference is tolerated. The factors still apply to the floor, because the disk size is a storage size: 19.31 GB at 4-bit is ~5.8 GB, not 19. On the incident model that is 5.8 + reserve against 5.6 GB free, so the load makes room or is refused honestly instead of being attempted on a figure three times under the truth. `source=used_vram_gb(disk-floor)` in the debug line, because a floor reporting itself as an ordinary estimate would be invisible to whoever reads these. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Every image sent to a GGUF vision embedder answered 500. Caught on live traffic by the 5xx logging added hours earlier, which named it in one line: [radeon] mtmd_tokenize: error: number of media markers in text (0) does not match number of bitmaps (1) [radeon] [http] 500 on POST /v1/embeddings: 'Embedding failed: mtmd_tokenize failed (2)' `mtmd_input_text` carries a `text_len` beside its `c_char_p`, and the struct was built with the constructor form, which leaves it at 0. Measured in the container: that form gives text_len == 0 for a 38-byte prompt, the field-by-field form gives 38. llama.cpp read a zero-length prompt, found no media marker in it, and refused the bitmap it had been handed. The marker was never the problem -- mtmd_default_marker() is b'<__media__>' and _prompt() interpolates it, both verified against the running engine, and both were suspected first. Built field by field now, exactly as llama_cpp's own chat handler does it, which also keeps a NAMED reference to the encoded bytes. That second half matters on its own: `text` is a c_char_p, so a temporary is free to be collected while the struct still points at it -- a latent defect in the same two lines that would have been intermittent rather than total. Verified against the running server afterwards: image-only, text-only and text+image to gme-Qwen2-VL-2B-Instruct-Q4_K_M all return 1536-d embeddings. Each of the two image cases was a 500 before. The request log now reports the image count. Its absence is why the search started in the wrong place: `model=gme-… inputs=4 chars=84` reads as a text-only request, because _summarise logged `input` and never `image`, so a failure on the IMAGE path looked impossible. Counts only, no content, like the rest of that line. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Follow-up: the footprint fix earlier today addressed one of two sources of the same wrong attribution, and the live log showed the other one an hour later. footprint: 8.48 GB total (weights 7.71 GB (0.00 on GPU, 7.71 on host), …) Measured footprint for 'Gemma-4-14B-A4B-Heretic.i1-IQ4_XS': 8.48 GB total (… of which 7.71 GB was offloaded to host RAM this load; n_gpu_layers=-1) That model was loaded with `n_gpu_layers: all` and was holding 10.1 GB of VRAM. `measure()` has no torch object to walk for a GGUF, so it takes the weights total from the file size -- correctly, a GGUF is loaded as-is -- and banked it in `weights_cpu_gb`, which keeps the TOTAL right and asserts a placement nobody measured. "0.00 on GPU" is the exact opposite of the truth for a fully resident model, and that line is what an operator reads while chasing a CPU-spill problem. I read it that way myself this evening and called the earlier mismatch cosmetic. `weights_split_known` now records that the split is unknown, `describe()` says "placement not measured here" instead of inventing one, and the manager distinguishes the three different things that reach `measured_ram`: a per-module measurement (genuinely on the host), a GGUF's unsplit total, and the process's RSS growth. Only the first is reported as an offload, and only the first puts a size on the Tasks page -- which had been labelling fully-GPU-resident GGUFs "CPU offload 7.71 GB". The arithmetic is untouched in both paths. The total is what sizing and eviction consume, it was right, and under-reporting a footprint is how eviction stops freeing room and the forward OOMs. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Five faults, found by one client asking where the OCR boxes had gone. **The boxes.** `_page_from_pod` rebuilt a pod's page key by key -- index, text, width, height, and a named list of diagnostics -- and never copied `lines`, `regions` or `tables`. Every page read on a pod therefore arrived with no geometry at all, for every engine, and no request parameter could change it: the pod puts all of it on the wire (it runs the same `OcrPage.to_dict`) and this boundary dropped it. `confidence` survived only by being one of the names. From outside it looked exactly like a missing engine feature, which is the expensive part: the client saw `lines=0` with a populated `conf`, concluded the engines were not emitting boxes, and went hunting for the parameter that turns line detection on. There is none -- `detect` is stamp/signature detection (off|layout|detector|both) and has nothing to do with text lines. The reassembly now lives once, in `codai.ocr.base.page_from_dict`, shared with the worker-subprocess boundary that had the same job and the better implementation of it. Diagnostics ride by exclusion rather than by name, which is what that boundary already learned after losing `accents_repaired` and then `model_attempts` the same way. A test asserted that "confidence" and "confidence_measured" appeared in the SOURCE of the pod reassembly. They did -- in the key-by-key list that was discarding the geometry -- so it passed throughout. It now runs the boundary. **geoclip answered 500 for text.** Reproduced: "una frase" loses its space, and "unafrase" is eight characters of the base64 alphabet, so it decodes to six bytes and only PIL can tell it is not a picture. PIL raises UnidentifiedImageError, which is NOT a ValueError, so it escaped the caller's `except ValueError -> 400` and became "the server broke" for bad input. Fixed at both ends: `_decode_image` raises ValueError quoting the input, and geoclip's IMAGE half joins vpr/salad in the pre-load refusal, because its contract puts the image in `input` -- so a text `input` is answerable from the name, before 1.9 GB is loaded. Its LOCATION half is deliberately excluded: it takes coordinates there and already answers a clean 400 naming the format. **The 5xx logging I added this morning caught nothing.** Starlette lifts a handler registered for status `500` out of the status table and gives it to ServerErrorMiddleware, which fires only for an UNHANDLED exception -- so `raise HTTPException(status_code=500, ...)`, which is how nearly every 500 here is raised, went straight past it. Confirmed live: two geoclip 500s, no `[http]` line, and the cause had to be found by replaying the request. Now registered on the exception class, logging at >= 500, with the status-keyed 400 handler still winning for its own richer line. Exercised with a TestClient rather than asserted from the source, because the first version passed every source-level check and caught nothing. **A model whose package is absent is no longer advertised as fine.** vosk and geoclip were both listed for as long as they existed while answering every request with an import error, because nothing asked. `codai/models/deps.py` answers it with a memoised `find_spec` -- never an import, never a model load -- and is used in two places: the listing annotates `unavailable_reason` (the model stays listed; hiding one that works is the worse mistake) and the embeddings path refuses with a 503 naming the package and the two files it has to be in. Only MAIN-venv dependencies are declared, because a package that lives in an isolated worker venv is absent from this one by design. **A fully-GPU-resident GGUF reported itself as offloaded.** "21.47 GB total ... of which 19.52 GB was offloaded to host RAM; n_gpu_layers=46", for a model with 46 of 60 layers on the card and ~15 GB resident there. That number is the process's RSS growth, and llama.cpp mmaps the whole file, so a GGUF grows RSS by about its file size whatever the split. The arithmetic is untouched -- the TOTAL is load-bearing for sizing, it was right, and under-reporting a footprint is how eviction stops freeing room and the forward OOMs -- but the line now says which of the two numbers it is holding, and the Tasks page no longer prints a delta as an offload size. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The engines run with LD_LIBRARY_PATH=/opt/coderai/local-libs first, and that beats the wheel's DT_RUNPATH. So llama-cpp-python dlopens its own libllama.so by absolute path, and that library then resolves libggml-base.so.0 against local-libs -- which still held the PREVIOUS build, because replacing the wheel never touched that directory. A 0.3.36 wheel (libggml 0.25.3, libllama 0.5.0) against local-libs' 0.9.7/0.0.8351 produced, in all five engines at once: Failed to load shared library '.../llama_cpp/lib/libllama.so': undefined symbol: ggml_rope_set_offset That is no GGUF serving at all. It hid because `docker restart` keeps the container's original image, so the image built this morning was never executed until the container was recreated this evening -- the rebuild was reported as deployed and the running process was still 0.3.30. The libs this build produces are now copied over the stale ones, so both search paths resolve to the same version in whatever order they are found. Orphaned older sonames are left alone: harmless once the soname every binary asks for points at the new file, and removing them would break anything else in the bundle still linked against them. The check that passed on the broken image is replaced. `ldd` answers "can every needed library be found" and is blind to a library that IS found and is the wrong version; `ldd -r` relocates too and names the symbol exactly as the runtime does. Two details decide whether it proves anything, and the first version of it got both wrong: - it must run with LD_LIBRARY_PATH set the way the engines set it, because the search ORDER is the entire bug; - it must check the WHEEL's libllama.so, the file actually dlopened, not local-libs' own copy. Old libllama against old libggml is a self-consistent pair that reports nothing: checked that way, this image passed with zero findings. Checked correctly it reported ten. Verified in a throwaway container against the broken image: 10 undefined symbols before, including ggml_rope_set_offset, and 0 after. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
One request, five minutes, fourteen cores, the GPU at 6%, on a card with room for the whole model. Eviction was never the problem -- it freed 14.2 GB and handed the loader 22.9 GB. Two defects then threw that away. The journal key was `<path>@<n_ctx>`, so a 7.5 GB KV measurement taken before swa_full=False stayed reachable after it. The real cache was 1.0 GB. That inflated the requirement from ~16.2 GB to 22.7 GB. The key now carries a hash of the settings that decide the cache size (swa_full, cache_type_k/v, kv_offload), so a measurement taken under one configuration is a miss under another; an unreadable config still yields a signature, because orphaning old entries is the safe direction and over-invalidating only costs one measurement. The sizing then never asked for more room. It sampled free VRAM once, found itself 0.2 GB inside a 95% trigger, and gave layers away while another engine still held VRAM it would have released on request. It now asks the manager to evict what a full-GPU placement needs, re-samples, and skips the spill entirely if it then fits; a failed eviction is not fatal and leaves the old decision. This is only safe because _evict_models_for_vram consults _is_key_busy and waits for idle -- and that check became trustworthy for embedding models when external_busy was wired into the embeddings path earlier today. Separately, geoclip never worked. `geoclip` and `geoclip-location` were advertised by /v1/models and both answered 500 in half a second, because they share _load_geoclip and the package was absent from the image: declared in requirements.txt, which the incremental overlay never installs from. Exactly the fault vosk had, in the same file, for the same reason, and fixed the same way -- including `--target`, since the venv is not writable by the container user and a plain install lands in ~/.local where the engine will not find it. Found by listing the built image's packages; it is invisible from the source tree. The install was verified in a throwaway container: all three imports resolve. And the reason that took so long to find: there is a 400 handler that logs why a request was rejected, and no 500 handler. The detail existed ("No module named 'geoclip'") and went only to the client, which in a capability sweep is a row in somebody else's report; the engine log held the thermal check, the VRAM estimate, then `Response status: 500`. A 400 is a statement about the request and a 500 is a statement about us, and this side logged the first and not the second. Behaviour unchanged; the class is now diagnosable. Four KV tests asserted exact argument lists and broke on the signature change. They now match the call through collapsed whitespace, via one shared helper, so the next reformat does not break them again. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
SALAD answered four sentences with nothing at all: no response in 180s, recorded by the sweep as a 500 with no error logged anywhere, while dinov2 given the identical request returned a clean 400 immediately. The refusal for the other image-only models keys on the backend name (`vision`, `dinov2cpp`). SALAD is `vpr-server`, and its contract — like EigenPlaces' `vpr` — puts the image in `input` rather than `image`, so the refusal cannot key on the field that was used, only on what is in it. `vpr` decodes locally and so fails cleanly; `vpr-server` handed the strings to an isolated subprocess that could not answer and did not reply. A hang is worse than a refusal: it holds a request slot and tells the caller nothing. Both backends now check their inputs look like image references — the same four shapes _decode_image accepts, tested without decoding so a real image is not decoded twice. A short base64-valid token is treated as text, because no picture is eight bytes and the alternative is sending "hello" to the subprocess again. The first version of this guard sat inside the embed path, after the load: it stopped the hang but answered 400 in 42 seconds, because SALAD had to spawn its subprocess first. The call site of the mirror check records that exact lesson eleven lines away — # Refuse an impossible request BEFORE loading anything. The same refusal # used to come after the load: images sent to a text-only 8 GB embedder # spent forty seconds loading it and then answered 400. — so repeating it in the other direction was avoidable. _takes_image_in_input answers from the model NAME, reading the existing _VPR_IDS/_SALAD_IDS sets rather than a second list that would drift from the ones the loaders dispatch on. The post-load check stays as a backstop for a model reached by an alias those sets do not know. Measured after: text 400 in 0.13s, a real image still returns its 8448-dim embedding. Also repairs a pre-existing test whose `[:4000]` source window this change pushed `_load_embedding_locked` past, so it failed with "substring not found" on correct code. It now bounds `_run_embeddings` by structure rather than a byte count — the third time a fixed source window has broken in this repo. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Qwen3.8-27B is built on the Qwen3.5 architecture — the model version and the architecture name are different numbers, and both our GGUFs and Qwen's own config say qwen35 / qwen3_5. llama.cpp's qwen35 support was contributed without vision and states it three times: #19435 "Qwen3.5 dense and MoE support (no vision)", merged; #19456 "Qwen3.5 support w/o vision"; and #22954, open, which says outright that the text-only qwen35.cpp does not load the vision encoder tensors and whose fix merely relaxes the tensor-count check so multimodal GGUFs can load as text-only. That accounts for everything measured. The mmproj loads because clip loading is architecture-independent, mtmd injects the image tokens, and the tokenizer carries the three vision markers at the ids the base config names — so the model notices whether an image was sent and says so when none is. There is simply no code in qwen35.cpp to consume the embeddings, so the content contributes nothing and the answer never varies. Eight theories died to reach a conclusion that was in the title of a search result seen at the start: dimensional mismatch, missing deepstack, a bad artifact, our own image-token floor, a stale llama.cpp, CUDA flash attention, the abliterated fine-tune, and finally the base model — which fails identically, so it is the architecture and not the weights. Two of those were written into these docs as settled before being tested, and the doc now records that as a method failure: everything was argued from reading rather than running, and the upstream tracker should have been the first stop rather than the eighth. mmproj stays off the qwen38 cards; vision goes to gemma-4-31B-heretic, which reads the same images exactly. The base Qwen3.8-27B download is kept as a non-abliterated text model with no projector. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
I wrote into the docs that our llama.cpp predating b4aa7dd47 ("mtmd : use align_corners for qwen3vl vision position embedding interpolation") was "the actual cause" of Qwen3.8-27B describing every image identically. The argument was a plausible commit plus date arithmetic — 0.3.30 vendors llama.cpp from 2026-06-16, the fix landed 2026-07-21, 0.3.36 vendors 2026-10-01 — and every observation fitted it. I had not run it. Tested now: 0.3.36 built from source with the full -DGGML_CUDA=ON -DGGML_VULKAN=ON -DGGML_RPC=ON set and put into service. The same two images still return byte-identical answers. The fourth theory joins the other three, and the doc says so. What the test buys is that the last candidate is eliminated by experiment rather than argument. The remaining reading is that the qwen3vl graph does not fit this model at all — the base is qwen3_5 while the projector declares qwen3vl_merger — and no version bump addresses that. mmproj stays off the qwen38 cards; vision goes to gemma-4-31B-heretic, which reads the same images correctly. The rebuild was worth doing for everything else, and it needed three fixes to complete at all: * build-rpc-server.sh asked for the `rpc-server` CMake target, renamed upstream to `ggml-rpc-server`. It now asks the generator which exists, so it spans both. * my first version of that detection used `grep -q` under `set -euo pipefail`: grep exits on first match, closing the pipe, cmake dies on SIGPIPE, and pipefail reports the pipeline as failed — a SUCCESSFUL match tested as a failure. Matched in pure bash now. It passes in a toy test because printf finishes before the pipe closes, so the race only appears with a slow producer. * Dockerfile.update-llama EXECUTED `rpc-server --help` at build time, which needs libcuda.so.1 — the driver, absent from any build by definition. The file's own comment eleven lines above states the rule ("the check is ldd, not import"); the wheel check obeyed it and this one did not. It had never been reachable before, so the first build to get that far could not pass. Now checks linkage and tolerates only libcuda being unresolved. And Dockerfile.update now carries vosk and g++, because the incremental overlay never touches the venv: an updated image shipped without vosk while /v1/models advertised vosk-en and vosk-it, and without any C++ compiler at all, which is what XTTS's Japanese phonemiser needs to build. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
download_model wrote straight to its final cache path with no try/except and no .partial, and the check guarding it tested existence only. So any interruption left a file that was then returned for ever as a valid cached model, and that URL could never be fetched again. Measured in one cache directory on zeiss: 188 zero-byte `<sha256>_<name>.gguf` entries accruing about 14 a day since 2026-10-16, against 2 non-empty files. And one of those two was a 9.72 GB ltx-2.5-22b-distilled-Q4_K_M.gguf where the real file is 13.58 GB — a truncated download the existence-only check would have served as a complete model. The empty ones break a load loudly; the truncated one loads. Now: the bytes go to a .partial and are renamed only once they are all there, an empty or short transfer raises instead of being kept, and the partial is removed on BaseException — not Exception, because a KeyboardInterrupt or SIGTERM mid-download is exactly how several of these were made. A zero-byte entry found in the cache is treated as a miss and deleted rather than served. The cache LOCATION is deliberately untouched: moving it would be a silent full re-download for every existing install, and a test pins the resolution order. Where it points on a given box is a deployment question — on this one a root symlink chain (~/.cache/coderai -> /AI/coderai -> /storage/coderai/) aims it at the repo's gitignored models/ dir, which is a separate matter from this bug. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Our llama-cpp-python 0.3.30 vendors llama.cpp e3a74b299 (2026-06-16). The fix for this exact failure landed five weeks later: b4aa7dd47 mtmd : use align_corners for qwen3vl vision position embedding interpolation (#25781, 2026-07-21) 0.3.36 vendors 0c1e57098 (2026-10-01), which contains it. Wrong position-embedding interpolation makes the vision features spatially meaningless, which is why Qwen3.8-27B described every picture identically rather than merely badly, and why a gemma4v projector -- a different graph, untouched by that bug -- read the same images correctly through the same pipeline. Everything measured fits: the projector pairs correctly (projection_dim 5120 = hidden_size 5120), loads, and injects tokens the model notices, since it says so when no image is sent; two publishers' conversions fail identically because the defect is in the consumer not the artifact; and the image-token floor is irrelevant -- tested at 1024 and at 0, byte-identical answers both times. The bump only takes effect on a rebuild (REBUILD_LLAMA=1), and it must stay a local source build: the CMAKE_ARGS carry -DGGML_CUDA=ON -DGGML_VULKAN=ON -DGGML_RPC=ON together, and a PyPI wheel has at most one backend. Until a rebuild runs, mmproj stays off the qwen38 cards and vision goes to gemma-4-31B-heretic. Also records the method failure, because it cost most of the investigation: three conclusions were drawn from reading headers and code and all three were wrong. The floor was dismissed with "the symptom predates it" -- which was never checked, and could not have been, since the floor is applied automatically whenever the projector type matches. Two projectors were then compared while the one variable this repo introduced was held constant in both. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The previous commit said the capability should go to gemma-4-14b-8k or Qwen3.5-9B and implied more finality than had been earned — neither had been tested at that point. Both now have been, on the same two images and the same prompt: gemma-4-31B-heretic 'Red square.' / 'Blue circle.' gemma-4-14b-8k 'Red square.' / 'Blue circle.' Qwen3.5-9B 'red square' / 'Blue Circle' qwen38 the same sentence for both gemma-4-31B-heretic is the one to reach for: larger than the broken 27B, already on disk with its own projector, exact on both images. That also narrows the fault to the qwen3vl_merger graph alone. gemma4v is correct at two model sizes, and Qwen3.5's own vision tower is correct when transformers runs it instead of llama.cpp — so it is not the model, not the family, and not our pipeline. And a warning that matters more than the finding: the llama-cpp-python here is a LOCAL SOURCE BUILD with -DGGML_CUDA=ON -DGGML_VULKAN=ON -DGGML_RPC=ON, which is the entire reason it exists. I had noted 0.3.36 is current on PyPI in a way that reads as "pip install -U", and a PyPI wheel carries at most one backend: installing one would take out the nvidia engine, the radeon/Vulkan engine and the cluster's GGUF-over-RPC path together, to chase one projector. The upgrade is a version bump in packaging/linux/Dockerfile.update-llama plus update_oci_image.sh REBUILD_LLAMA=1. Written down so nobody reaches for pip. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Qwen3.8-27B served images for months and answered identically for all of them. Four plausible explanations were wrong, two of my own inferences were wrong, and only an experiment settled it — so the whole thing is written down rather than left as a line in a changelog. Cleared with evidence: the projector pairs correctly (projection_dim 5120 = the model's hidden_size), the missing deepstack tensors are missing from unsloth's reference conversion too so this family does not use them, clip_graph_qwen3vl IS implemented in libmtmd, the base model genuinely is a VLM with vision_config and image_token_id 248056, the GGUF tokenizer carries all three vision markers at the ids the base config names, the mmproj does load, coderai's vision guard does fire, and the 1024-token floor is llama.cpp's own requirement. Also wrong, and worth recording as a trap: "no mtmd output at request time, so the handler is bypassed". The llama.cpp log callback is deliberately swapped to a quiet one after the load, so runtime silence proves nothing. What settled it was two experiments. gemma-4-14b-8k with its own projector reads the same two images correctly ('Red square.' / 'Blue circle.') and invents when given none — so the pipeline is sound end to end. And the same Qwen model with a DIFFERENT publisher's projector fails identically, so it is not one bad artifact. The fault is upstream: either llama.cpp's qwen3vl_merger graph is wrong for this model, or both converters mislabel it — the base is qwen3_5 while the projector declares qwen3vl_merger, so the wrong graph may simply be running. mmproj is removed from the qwen38 cards. A model that answers confidently and wrongly is worse than one that declines, and the listing now says text only. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An engine took SIGSEGV mid-sweep and every model resident on it died with it. _is_key_busy() is what stops eviction unloading a model under a live request — its docstring says so — and it reads two sources: the instance pool's ref counts and external_busy. An embedding request appeared in NEITHER. It takes no pool ref, and nothing on that path called external_busy, whose only caller in the whole codebase was the lipsync engine. To the evictor, an embedding model was permanently idle. 12:27:41 Loading Qwen3.6-14B-A3B: need 9.3 GB, have 4.8 GB free — evicting 12:27:41 [req] POST /v1/embeddings model=Qwen3-Embedding-4B 12:27:42 Response status: 200 12:27:42 On-request VRAM eviction: unloading ACTIVE model 'embedding:…Qwen3-Embedding-4B-Q4_K_M.gguf' backend.model=None 12:27:44 engine#1 exited (code -11) code -11 is a segfault, not the -9 OOM-kill everything else this week has been about: the native model was freed while a request was still dereferencing it. The 502s on /v1/embeddings that had been appearing for hours were this. The guard wraps the whole region from having the model to being done with it, not each inference call: there are three such functions across seven call sites and several early returns, and the window that must be protected is the whole of it. The key is request_model's `<model_type>:<resolved_name>`, which is exactly what the evictor logs and _is_key_busy looks up — marking busy under any other spelling would be a no-op that reads like a fix. A test asserts every inference call sits inside the guard. Its first version matched substrings and flagged `_may_embed_images` — a name-and-config capability check that runs before the model is even loaded — so it now matches exact names. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
measured_n_gpu_layers is reused as a cap so a reload jumps straight to the split that fit instead of rediscovering it through OOM retries, and measured_quant_sig is what stops such a measurement outliving its configuration. It covered quantization only. Setting swa_full=False on a gemma card cuts its KV reservation by about 19 GB — its 25 sliding layers stop reserving the full 178k context and hold their 1024-token window instead, at no quality cost. The learned split of 18 of 30 layers was measured with the old full-SWA KV, and nothing in the signature changed, so it would have survived and pinned the model to CPU on a card that can now hold nearly all of it. That is the layer-ratchet bug again, arriving through a setting the signature was not watching. n_ctx, swa_full, cache_type_k/v and kv_offload now form part of it. They are not "how big the weights are", which is what the docstring used to claim the hash was for, but they decide how much of the card is left for the weights and therefore how many layers fit. offload_kqv is read as a spelling of kv_offload because the loader accepts either, and a card written one way must not keep a measurement taken the other. The field list is deliberately narrow: over-invalidating discards every measurement and restores the OOM ladder on every load, which is the whole cost this mechanism exists to avoid. A test pins that both ways. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The MTP repair shipped in 0.5.37 fired on a real model during a capability sweep and made the failure worse. llama.cpp validates per-layer metadata arrays against block_count, and a metadata override can only carry a scalar — bool, int, float or str — so there is no way to shorten the array to match. Correcting the count therefore does not fix the load, it only changes which error it fails with: before missing tensor 'blk.64.attn_norm.weight' after key qwen35.attention.recurrent_layers has wrong array length; expected 64, got 65 The first message names the real problem. The second sends the reader looking for a corrupt array. A repair that only changes which error you get is not a repair, so it now declines when any per-layer array is present and the honest message stands. Declining silently would be its own trap — the shape still looks repairable from outside — so unrepairable_reason() explains it before llama.cpp speaks: which array pins the count, that an override cannot carry a shorter one, and the two ways out (a quant that keeps its NextN block, or a llama.cpp that derives the layer count from nextn_predict_layers). "Per-layer" means length EQUAL to block_count, so a tokenizer's 248,320-entry token list does not veto a repair it has nothing to do with. The test fixture can now write array-valued metadata, which is what it would have taken to catch this before shipping rather than from a live sweep. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
XTTS reported no voices after seventeen successful syntheses, and the reason was that the loaded and remembered paths had never worked for any engine since they were written. Every backend declares `voices` as a METHOD; both callers read it as a sequence: [str(v) for v in (getattr(backend, "voices", None) or [])] A bound method is truthy, so the `or []` never fired, iterating it raised TypeError, and both callers caught that and fell back to "nothing known". No voice was ever recorded or returned, and the cache file had never been written once. Kokoro and bark hid it behind their static lists; XTTS and parler, having none, published zero for ever. The tests passed because their fake sets `voices` as a list attribute — a shape no real backend has. A new test asserts the contract against the real classes through inspect rather than against a fixture, which is the only version of this test that could have failed. With that fixed, 58 XTTS voices were recorded and the listing still showed none: they were written under "XTTS-v2", the id the request used, and looked up under "coqui/XTTS-v2", the id the listing publishes. The lookup now falls back to matching the last path segment — the idiom the coqui speaker lookup already uses — with an exact key still winning and distinct models still distinct. The existing never-breaks-synthesis test then caught a gap in the first fix: a backend may expose `voices` as a PROPERTY, so merely reading the attribute runs its code and can raise. The getattr is inside the guard for that reason. Live after this: XTTS-v2 58 voices, Kokoro 54, bark 130, parler 0 with voice_selection "described", which is the honest answer for an engine steered by a text description. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two models could not be loaded in one afternoon, and establishing why took a diagnosis each. Neither was a memory problem; both failed late and expensively, after other models had been turned out of the card to make room. The second one is the reason for this. A 27B "Flash" quant passed every check a careful operator would make -- right architecture, all 65 blocks present including the MTP one, an mmproj beside it -- and stored 402 of its 866 tensors in ggml type 143, a rotation-folded format from a fork. Nothing in the metadata announces that: the type ids live in the tensor index, which no error message reads back to you. It arrived as a generic load failure after the eviction, six retries and a full read of the weights. gguf_preflight answers it from the header alone and runs before the VRAM need is even computed, so a load that was never going to work costs no other model its residency. The bound is llama.cpp's own GGML_TYPE_COUNT rather than a list kept here, so it tracks the installed library. Only ids at or above that bound are refused, and the conservatism is the design. The Python binding's GGML_TYPE_* constants are an INCOMPLETE mirror of the compiled enum -- 0-3, 6-29, 39-42 -- so 30..38 have no constant, and 30 is BF16, which files we serve today use and which loads perfectly. "Absent from the binding's constants" would have rejected every one of them. An id inside the enum that the build still cannot handle is left to llama.cpp to refuse in its own words. The verdict gets its own exception type because the on-demand path turns a failed VRAM reservation into a printed warning and carries on -- correctly for a reservation, fatally for a verdict, which would then reappear a minute later as the original confusing failure. That path now re-raises GgufUnreadable, the same treatment HostRamExhausted already gets, and the comment three lines below it ("a host-RAM refusal is the decision itself and must reach the caller") is why. It reports 501, not 503: evicting and retrying cannot change the answer, so a client that retries a 503 forever must not be invited to. The type ids ride along in the response, being the actionable part. The architecture is printed before the load too, so llama.cpp's "unknown model architecture" sits beside the file it came from. Deliberately NOT a list of supported architectures: there is no API for it, and guessing from the shared library's strings would invent a second source of truth that can be wrong in the direction that blocks a model which would have worked. No fourth GGUF parser -- the tensor-type histogram rides along on the walk gguf_mtp.read_layout already does, and a test asserts gguf_preflight has no parser of its own. docs/serving-a-model-this-box-cannot-hold.md records the three independent requirements, how to verify each on a new machine, the quant/size table, why an mtp-*.gguf is not a model, and which of the three things called "MTP support" already exist (vLLM's raw --speculative-config passthrough, ds4's --mtp) versus which does not (a GGUF draft model, which llama-cpp-python cannot express). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
cutlet, XTTS's Japanese phonemiser, pulls mojimoji — a C++ extension with no cp313 wheel, so it compiles when the isolated TTS venv is built. The build asked for clang++, because the bundled Python records clang++ as its C++ compiler and these images have no clang. They had no C++ compiler at all: g++, c++ and clang were all absent, /usr/include/c++ did not exist, and only libstdc++.so.6 was there without its dev symlink. So gcc's presence — added for Triton's runtime JIT — looked like it covered this and did not. g++ is added to the same apt lists for the same reason gcc is in them: not to build the image, but because something compiles at RUNTIME inside it. It brings /usr/bin/c++ and the C++ headers with it. Done in Dockerfile.oci-venv (the one build_oci_image.sh actually builds), Dockerfile.oci, and the pod capability-base. The pod XTTS profile had the same undeclared-phonemiser gap as the server venv, which its own comments already describe for click — so pypinyin, cutlet, fugashi and unidic-lite are declared there too, and its deep check now imports both phonemisers and constructs a Cutlet, because the failure they cause is invisible at install and at boot: the pod answers /v1/models with all seventeen languages and then refuses two of them per request. Verified on zeiss after installing these: 17 of 17 XTTS languages return audio, and cutlet romanises Japanese through a working fugashi dictionary. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Probing XTTS's seventeen languages after the truncation fix: fifteen spoke, and the two with non-Latin scripts failed on dependencies coqui-tts does not declare. zh-cn -> ImportError: Chinese requires: pypinyin ja -> ModuleNotFoundError: No module named 'cutlet' So the venv installs cleanly, reports success, advertises all seventeen, and then refuses two of them at synthesis time — the same shape as the torch that was never installed, which is what pinned this file in the first place. They are now declared in requirements-xtts.txt and in the _PINS copy the worker falls back to when the file is not mounted (a test keeps the two in step). cutlet romanises through fugashi, which needs its dictionary at RUNTIME and not only at install, so unidic-lite is declared too — the chain requirements-melotts.txt already notes for MeloTTS. pypinyin is pure Python and is installed on zeiss now, so Chinese works there. Japanese needs a build: cutlet pulls mojimoji, which ships no cp313 wheel, so it compiles — gcc and the Python headers are both in the image, but the install has to run where build deps are available. It is declared here so an image build picks it up. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Probing XTTS-v2's seventeen languages through the running endpoint: sixteen returned audio and zh-cn failed with KeyError: 'zh'. Our own call site did it. `language=lang[:2]` truncated every code to two characters, and zh-cn is the only XTTS code longer than that, so the single language with a regioned code was the single language that could not be spoken. Resolution now goes through XTTS's own declared list, which also fixes the other direction: a client sending a locale ("en-us", "pt-br", "ZH_CN") lands on the code XTTS actually has instead of a near miss. tools/xtts_service.py had the identical truncation. It runs in coqui's own venv and cannot import the server package, so the list is duplicated there — and two tests keep them honest: one compares the declared lists, the other executes the worker's own normaliser and asserts it answers exactly as the server's does. Also updates the bark voice-count test, which pinned the English-only list at ten. That assumption was the bug. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Probing all nine of Kokoro's languages through the running endpoint, one request each: eight returned audio and `fr` returned a 500 — "espeak-ng has no such language". French is regioned in espeak exactly like English (fr-fr, fr-be, fr-ch), so the bare code every OpenAI-compatible client sends was refused. The French voice had never worked. The regionless map's own comment asserted the opposite: that "it", "fr" and "de" are all accepted bare. Two of those are true. A test asserted the same thing for all three, so the belief was pinned in place without ever being measured — exactly the failure the Kokoro-English fix was about, one language further along. "fr" now maps to "fr-fr"; the comment says not to add a code on the strength of the pattern, because the pattern is what got this wrong. Measured bare-OK while I was there, so the next person has data instead of a guess: de, it, nl, pl, tr, ru, ko, ar, cs, hu, sv. The corrected test asserts those, and a new one checks the invariant that broke here — every language the listing advertises for Kokoro must resolve to something espeak was measured to accept. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Languages landed but the voices did not: kokoro reported none until something loaded it, so a client could see nine languages and still have nothing to pick from. The names are now read out of the voices file the backend itself selects from — the same file, so the listing cannot offer a voice that synthesis would then reject — and the accent map already in the module decodes each one's language and gender. 54 voices across all nine languages. Pointedly NOT via _resolve_kokoro_files: that downloads a 28 MB release when nothing local matches, and /v1/models is a polled endpoint, so it must never be a download trigger. A file that is not on disk simply yields no names, and there is a test that fails if the listing path reaches for the network at all. The zip index read is memoised against the file's own mtime (27ms -> 0.1ms). A list a load actually enumerated still outranks the file, and a card's voices_path is honoured. XTTS keeps filling in after its first load, and parler stays empty with selection "described" — it has no speaker names to give. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Every TTS model published `languages: null` on /v1/models while the STT models published theirs correctly. Nothing in the server restricted the languages — it simply never named them, so a client was told nothing about what any voice could say and could only discover a language by trying it. That is how XTTS-v2's seventeen languages got taken for English and Italian: those were the two somebody tried. Each set is read from the engine's own files rather than assumed — XTTS from its config.json, bark from its 13-language speaker bank, parler from its model card, and kokoro derived at call time from the accents already encoded in its voice ids, so adding a voice family cannot leave its language behind. An engine whose languages depend on the checkpoint (melo, the generic transformers fallback) claims nothing: a wrong list is worse than none, because a client that believes it sends text the voice cannot pronounce and gets noise back with no error. A `languages` list on the model's own card still wins. Bark's voice list was the ten English speakers, which is where "bark is English" literally came from; it now covers all 13 languages, English first so a client taking voices[0] as its default still gets an English speaker. The voices are published too, so a voice picker no longer needs to load a model to ask it — and since /v1/models is polled, the voices cache is memoised against its own mtime rather than re-read per model per poll. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Four settings decide whether a GGUF load fits in host RAM at all, and none of them could be set from the model page — only by editing models.json by hand. The placement check, how sliding-window layers reserve, the measured size the check acts on, and the MTP-block repair are now on the page with the rest of the KV cache, gated to GGUF models, and in the savable list that is where a field quietly stops persisting. measured_kv_gb has been written by hand as a {n_ctx: gb} map and is now journalled as a plain number, so the form renders either shape instead of "[object Object]". The MTP override now also sets nextn_predict_layers to 0 rather than correcting block_count alone. Builds disagree on whether block_count includes the NextN blocks: the one this was written against does not subtract them (it asked for blk.64 of a 65/1 file), and a build that does would read a corrected 64 as 63 and quietly leave a real layer out of the model. "64 blocks, none of them NextN" is true under both readings. The journal also takes its directory from platform_paths rather than inventing an environment variable, so a deployment keeps one home. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A model configured cache_type_k/v=q8_0 at n_ctx=128000 got a 47.7 GB KV cache placed in host RAM and the engine was SIGKILLed mid-load, taking every other model resident on it down too. The placement plan that exists to prevent exactly that skipped itself, because any pin — type or location — was read as consent to whatever size resulted. Quantized KV at a long context is the configuration someone writes because they are worried about size; it is the one that most needs checking. A pin is now honoured without switching the check off: a pinned type is never requantized, a pinned placement is never moved, and when the remaining levers cannot make it fit the load is refused with a message naming the pin. That refusal needs a measured size, and the measurement was being taken after a successful load and printed as advice to copy into the config by hand. A KV cache big enough to need the plan is big enough to kill the process before the load ends — so there was no "after", nobody was reading, and the engine died, got respawned, and repeated the identical load forever. It is now journalled the moment llama.cpp names the size, before the allocation is faulted in. measured_kv_gb still wins; the journal only answers when the config was silent. Separately: a quantizer that drops a checkpoint's NextN (multi-token-prediction) blocks without lowering block_count makes llama.cpp ask for a block the file does not have, reported as a bare "missing tensor 'blk.64.attn_norm.weight'" — which reads like a corrupt download. When the file's shape proves that is what happened (blocks contiguous from 0, exactly nextn_predict_layers short), the declared count is corrected via llama.cpp's own kv_overrides. A file missing blocks for any other reason still fails, because a truncated download must not be silently reinterpreted into a model that loads and answers nonsense. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
LongCat had two settings that look independent and are not. A quantised text encoder MUST be resident -- bitsandbytes quantises as weights land on CUDA and cannot shuttle them back, and _place_pipeline returns early for a quantised encoder without consulting the offload mode at all. So "quantise the encoder" and "spill when VRAM runs short" are one decision: asking for both pins 22.1 GB of a 24 GB card (13.6 DiT + 8.0 nf4 encoder + 0.5 VAE) and leaves under 2 GB for the activations of a 13.6B video model, which OOMs at the first long clip rather than at load and so reads as a flaky model. offload_strategy now accepts "auto", which takes that decision once, at load, and logs the arithmetic: * resident -- DiT, VAE and a quantised encoder all on the card, no swapping; * spill -- DiT and VAE resident, encoder on the host, the two swapping for encode_prompt so only one is ever on the card. The decision is made against the card's CAPACITY -- total VRAM less a context reserve -- and deliberately not against the free VRAM at that instant. Free VRAM makes the placement depend on whichever model happened to be resident when the request arrived: a sibling holding 8 GB would park LongCat on the host, and it would stay there for every later request, because nothing re-examines the condition. It is also self-fulfilling -- the worker RESERVES the footprint and the manager evicts what is in the way, so "no room right now" is not a fact about the card but about a decision nobody has taken yet. A model that could be resident is never left on the CPU merely because something else got there first. What is reserved is what the chosen placement OCCUPIES, not what residency would need: on a 24 GB card that is 14.1 GB (DiT + VAE), because reserving the 22.1 GB residency figure would evict siblings to free 8 GB this placement never touches. Both numbers are reported, so the one the decision was made on stays visible. The worker and the service run the SAME pure arithmetic over the same inputs, so they cannot disagree, and the spawn moved inside the acquire loader: it used to run before the reservation, so the service loaded its weights onto a card the manager had not cleared yet and the eviction meant to make room happened afterwards. Fine for a 13.6 GB DiT, wrong for any placement that wants the card. Every load now logs what it actually cost against what the plan predicted, so a constant that drifts shows up as a wrong prediction instead of as an OOM at generation time. And it is selectable: the model page's Offload list had no "auto" entry, which mattered more than it looks, because the page sets that select from the saved value -- a card reading "auto" would have displayed as the first option and the next save would have silently rewritten it to "". The text-encoder labels now say which levels need a card bigger than 24 GB, since that is what decides whether the quantisation is honoured. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
**A seeded request to an offloaded pipeline failed as a load error.** Sequential CPU offload leaves every module on the META device until its own forward pulls it in, and a pipeline whose modules are all on meta reports meta as its device. torch.Generator(device=pipeline.device) then reached torch's accelerator hooks with a meta device: RuntimeError: META device type not an accelerator. The image route reported that as "Failed to load image model 'Tongyi-MAI/Z-Image-Turbo'" -- naming the load, which had in fact succeeded (footprint measured, 1.56 GB offloaded to host RAM) -- so the model read as unloadable. And only a SEEDED request failed, so it looked broken to anyone who passes a seed and perfectly fine to anyone who does not. diffusers already knows the answer: _execution_device is where an offloaded pipeline actually computes. models.utils.generator_device() prefers it, rejects meta and lazy, and falls back to cuda/cpu. Four sites constructed a Generator from a pipeline's own device -- text-to-image, img2img, edit, and audio generation -- and fixing one would have left the same failure reachable by three other routes, so all four go through it. **Four working edit models were reported as broken because of their capability list.** The Qwen-Image-Edit siblings carried image_generation and modes: null, while diffusers maps no text-to-image class for the family and the request path says so outright: 501 QwenImageEditPlusPipeline cannot generate from a prompt alone: diffusers offers no text-to-image pipeline for this family. The runtime was right and the capability list was wrong, which is the wrong way round. The instruction-edit branch of the heuristic lumped two different kinds of model together: FLUX.2 is unified -- one checkpoint that generates from text AND edits -- while Qwen-Image-Edit, FLUX.1 Kontext, Step1X-Edit and the Rapid-AIO fine-tunes are edit-only. They are split now, so only a unified family claims image_generation. And the listing publishes the directions. `modes` is how a client learns a model has no text-to-image path without sending one and reading a 501, and most cards never set it; when a card says nothing, the modes are now inferred from the capabilities the model already publishes (t2i/i2i, t2v/i2v/v2v). Inferred, never invented: every mode is backed by a flag that was already there, and a card that states its own modes always wins. Verified live while testing this: Wan2.2-I2V DOES serve a prompt-only request (it returned an mp4), so that row was collateral from the VRAM contention rather than a missing path. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Publishing several ids for one model is deliberate: an alias can stand for a round-robin group, and a typed id coexists with the checkpoint it names. What was missing is any way for a client to tell which ids are ONE model. Measured on a live install: 206 published ids over 119 card entries -- 136 aliases, 20 absolute paths whose basenames were also listed, and 50 repo ids, with no exact duplicates among them. A capability sweep walking that list reported seventy "new models" it had not tested, all of them names it had already covered, and had no way to know. So each entry now carries: canonical_id the same string on every id belonging to one model, so grouping by it is sufficient; alias_of set ONLY on the non-canonical ids, so a client that skips the entries carrying it tests each model exactly once; aliases on the canonical entry, the other names it answers to. Two cards that share a checkpoint but differ in CONFIGURATION are not the same model. They keep separate canonical ids and stay separate entries, because they are two deliberately different setups and a client must be able to see both -- grouping them would hide one. A bare path only one of them can claim (ids are unique in a listing) groups with the card that claimed it, which is the honest answer: sending that path reaches that card's settings. Nothing is invented. A canonical name that this listing filtered out -- parked as disabled, or assigned to another engine -- is never pointed at; those ids are left standing on their own rather than referred to a name the server will not answer to. _group_ids lives above the class, not between two methods: added there first, it ended the class body, and 126 methods silently became module functions with list_models among them. A test asserts it stays out. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.5.32 shipped the image-token floor and broke the projector load it was meant to fix: mmproj : failed to load projector (name 'ImageTokenFloorHandler' is not defined); continuing text-only handler() referred to the class as a global while the class was produced by a module-level __getattr__, and PEP 562 consults that only for attribute access from OUTSIDE the module -- never for a global name lookup inside it. So the lookup raised NameError, the except around the mmproj block caught it, and the model came up text-only: it then refused images instead of answering them wrongly. Better, and still broken. The factory existed for exactly this; handler() now calls it (cached in _HANDLER_CLASS) instead of naming the class. And the test that let this through was testing the wrong call: it built the class with _make_handler_class() directly, so it passed against a shipped path that raised. It goes through handler() now, with a second test asserting the class is reachable the way the backend reaches it. A probe that does not call what the code calls proves nothing -- the same lesson as the vision-mapping fix in 0.5.27, learned again in the module next door. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A GGUF vision model answered every image question fluently and wrongly. On Qwen3.8-27B, with its own matching projector loaded and llama.cpp reporting "has vision encoder": * a 448x448 image added 16 tokens to the prompt (61 -> 77). The same image through the HF path added 156. * two completely different pictures -- a yellow field with a blue triangle and a green square, and three white circles on black -- produced byte-identical descriptions at temperature 0, neither resembling either image ("the image shows a series of exclamation marks"). * the same prompt with NO image answered differently, so the pictures were reaching the model and only their content was not. llama.cpp had been printing the reason on every single load: load_hparams: projector: qwen3vl_merger load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024 mtmd_context_params carries image_min_tokens and image_max_tokens, and nothing set either, so the count came from the image's own size alone: sixteen tokens, which cannot describe a photograph. The model did the only thing it could and answered from the prompt, with complete confidence. A 200 carrying a fluent wrong answer is the worst failure shape there is, because no caller can tell it from a right one -- and this one survived a full capability sweep, reported as "answers don't match the picture". So the floor is set: from the card's image_min_tokens when it says, and otherwise from the projector's own documented requirement, read from the mmproj GGUF's clip.projector_type rather than guessed from a filename (an mmproj is routinely renamed to sit beside the model it serves). Projectors with no documented floor are left exactly as they were, because forcing an upscale costs prefill on every request. The figure is printed at load, since it changes what every image costs. Upstream builds its context from mtmd_context_params_default() and sets four fields on the result, with no hook in between and the token bounds not among them. Rather than copy that method -- which would silently stop tracking any field upstream later adds -- the handler wraps the default-params factory for the duration of that one call, so the struct upstream receives already carries the bounds and everything upstream does to it still happens. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Every whisper.cpp model answered with text and nothing else -- no segments, no words -- while the HF and Vosk backends on the same endpoint returned both, which reads as whisper being the broken one. Two reasons, in sequence: the server was never asked for a verbose reply, and the route passed a hard-coded empty segment list to _finalize whatever came back. So the server is asked for verbose_json (one extra form field, ignored by a caller that wanted plain json), and the reply is mapped in both shapes it comes in: OpenAI-style "segments" in float seconds, and whisper.cpp's own "transcription" entries whose "offsets" are MILLISECONDS -- read as seconds those would place a 2-second clip 2000 s in. Per-word entries are picked up from the top level or from inside each segment, wherever a DTW build puts them. Nothing is invented: a field the server did not send stays missing. And when word timings were REQUESTED but none came back, that is now said out loud, because whisper.cpp emits them only with DTW enabled and a silent empty list is indistinguishable from a model that found no speech. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two completely different pictures sent to a GGUF model with a configured mmproj produced byte-identical descriptions, while the same prompt with NO image answered differently. So the images reached the model and their content did not -- the embeddings carry nothing -- and the log said nothing either way: projector init, media-marker substitution, bitmap decode and chunk eval all log under names ('mtmd', 'clip', 'bitmap', 'media', 'projector') that the load-phase filter did not have, so every one of them was discarded. Reproduced with two synthetic images (yellow background with a blue triangle and a green square; black background with three white circles) at temperature 0: identical answers, neither resembling either picture, both unlike the no-image control. Keeping these lines is what makes the next request say which step is losing the pixels. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
**Five models "won't load" and llama.cpp had said why.** The load-phase log callback printed only lines matching a keyword list about layers, buffers and devices. llama.cpp writes its diagnosis -- an unknown architecture, a missing tensor -- in none of those words, so it matched nothing and was dropped, while llama-cpp-python raised its one generic sentence: "Failed to load model from file: <path>". That string is all the client got, and all the log had. So every line is now kept in a 60-line ring, lines that look like a diagnosis are printed whether or not they mention a buffer, and the raised error carries llama.cpp's own words. A model this build genuinely cannot read is no longer indistinguishable from one that did not fit, a bad path, or a corrupt download. The retry ladder stops earlier too. It exists to answer one question -- does it fit? -- by moving layers to the host, and an attempt with NO layers on the GPU has already answered it. Three MTP/next-N checkpoints each burned six attempts down to n_gpu_layers=0, every one failing in a fraction of a second with the same message: 18 loads whose only effect was to push the reason further up the log. (What those three have in common is visible in their metadata -- qwen35.nextn_predict_layers=1, blk.N.layer_output_scale.weight -- and with the reason no longer discarded, the next attempt will say so outright instead of leaving it to be inferred.) **XTTS and parler were handed an alias instead of a repo id.** _load_backend passed `model_path or model_name` to Kokoro and the bare `model_name` to everyone else, and for a card registered under an alias that name is the alias. Each isolated worker then failed in its own vocabulary, neither mentioning the cause: XTTS: "XTTS venv could not load 'XTTS-v2': ValueError: not enough values to unpack (expected 4, got 1)" -- coqui splits a model name into type/lang/dataset/model, and the bare alias has no separator at all. parler: "parler-tts-mini-multilingual is not the path to a directory containing a file named config.json" -- the org is missing, so the real repo is never looked up, and the worker reports a model that IS cached as one to download first. Every backend that resolves a checkpoint now gets the path. The XTTS worker's own translation table is fixed to match on the last path segment as well, since it is reached by other callers too: it required a "/" before, which is precisely the case that does not apply. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Four unrelated rows from an end-to-end pass over every capability, with one thing in common: in each case the information needed to get it right was already in the server and something declined to read it. **An image card with no precision loaded at f32.** The default here was f32 and only --no-ram mode coerced it down, so whether a model occupied 2 bytes per parameter or 4 came down to an unrelated flag. Measured on one run: CyberRealisticPony and wai-illustrious (no_ram=True) loaded at 7.49 GB, while PonyRealism and pornmaster-pro -- same SDXL architecture, no_ram=False -- measured 14.98 GB, exactly double, filled a 24 GB card and failed with "CUDA out of memory. Tried to allocate 12.00 MiB". The inflation then spread to the estimate, where the f32 precision factor turned 14.98 GB into a 30.9 GB need and triggered a round of eviction that could not possibly help ("this model needs more than the whole card"). Nothing was gained: these checkpoints are published at half width. The default is now bf16 -- bf16 rather than fp16 because it keeps f32's range, the same reason the --no-ram branch gives. An explicit f32 on a card is still honoured. **LTX refused the width and height.** Every family in video_geometry declares a side_multiple -- 64 for Wan's VAE cell, 32 for H3 -- and nothing anywhere read it; the request path never consulted the table at all. So 848x480 reached LTX-2.5 and came back "`height` and `width` have to be divisible by 32 but are 848 and 480", with the server making the caller do arithmetic it had the numbers for. Sides are now snapped to the nearest legal cell, and LTX has a family of its own: it had none, so it inherited Wan's 16, under which 848 was already "legal" -- snapping to the wrong multiple is indistinguishable from not snapping. The new entry differs from the Wan fallback in exactly the one field a refusal proved. **Kokoro could not speak English.** espeak-ng names English by region and refuses the bare code, while every OpenAI-compatible client sends "en" -- so the Italian voices worked and the English ones raised `RuntimeError: language "en" is not supported by the espeak backend`, as a 500, which reads as a broken model. Regionless codes espeak rejects are mapped ("en" -> "en-us"); codes it accepts bare ("it", "fr", "de") pass through untouched; anything it still does not know is now a 4xx naming the language and what to send. **A pinned model landed on the wrong engine.** by_name() matched an engine by NAME or by BACKEND in one pass and returned the first healthy hit. A GGUF-isolation split runs two engines on one card and both declare the backend "nvidia", so a transformers model pinned to "nvidia" could resolve to the GGUF sibling and be refused for a capability the engine it actually named has: "model 'longcat-avatar' is pinned to 'nvidia' but engine 'nvidia' (backend 'nvidia') can't serve a 'transformers' model (capabilities: ['gguf'])". The sibling fallback could not recover it either -- it appends "-gguf" to the PIN, so it tried the same engine twice and never the one named "nvidia". An exact name is unambiguous, so it now wins, in its own pass. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A load is admitted to VRAM by estimate: ensure_vram_for sizes the model, evicts what is in the way, and failing that a CUDA OOM is caught and reported. Host RAM had no equivalent, and the kernel supplies its own: four engines SIGKILLed in five hours on a 54 GB host, each one while a model staged its weights. Oct 29 23:03 Killed coderai-nvidia-gguf anon-rss 46.3 GB Oct 30 03:10 Killed coderai-nvidia anon-rss 47.7 GB Oct 30 03:36 Killed coderai-nvidia anon-rss 31.6 GB Oct 30 03:49 Killed coderai-nvidia anon-rss 32.7 GB front: engine#0 exited (code -9); respawning (1/5 in 120s) The RAM watcher was awake for all of it and logged "RSS 50.6/43.0 GB (118%) -- mitigation [gc+trim+evict_idle] -> 53.7 GB": every rung of its ladder releases memory already held, and none of them can decline the load doing the allocating. SIGKILL then takes the engine rather than the request, so the client that asked for the oversized model saw a dropped connection -- and so did every unrelated request in flight, plus everything arriving during the 120-second respawn. One bad load reads as the whole server being unreliable. So offload.ram_admission (default on) adds the missing half. codai/models/ host_ram.py sizes a pending load -- measured host residency if the model has loaded before, the whole footprint for an offload strategy that asks for it, the spill past free VRAM otherwise, plus a staging allowance -- against the tighter of MemAvailable and the headroom under max_ram_gb, less a floor kept for the OS and the sibling engines. ensure_host_ram_for() runs the existing pressure ladder FIRST and re-measures, so refusal is the last step; when it comes, it is a 503 naming the model, the estimate, the budget, the shortfall and what to change on the card. A model whose footprint cannot be determined is never refused on a number nobody has. And the proximate cause of the biggest of those kills was a sizing step that reported itself as skipped: (auto-offload sizing skipped: cannot access local variable '_ck' where it is not associated with a value) The GGUF auto-offload branch reads the configured KV-cache types to size its headroom, from a binding made ~110 lines further down where llama_kwargs is assembled. Every load that reached the branch raised UnboundLocalError, the handler called it "skipped", and nothing reduced n_gpu_layers -- so llama.cpp was asked to keep every layer of a 35B on a card that could not hold them, and answered with a 43,562 MiB CPU KV buffer. The read is hoisted above its use (tested on source order, not a line number), and the handler now says the load is UNSIZED, because "skipped" invites the reader to assume nothing needed doing. Separately, and found while checking which cards were really misconfigured: a repo whose model_index.json declares a pipeline class diffusers does not have is now detected and loaded with its own code when allow-listed, or refused by name when not. The allow-list and the mechanism both already existed, but only fired when a card named a custom_pipeline -- nothing ever did, so MOSS-Image and Kromoss failed with "module diffusers has no attribute MOSSImagePipeline", which names neither the cause nor the remedy and reads as a broken model. Co-Authored-By:
-