- Nov 01, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
-
Stefy Lanza (nextime / spora ) authored
Z-Image-Turbo, measured 2026-11-01: Cannot generate a cpu tensor from a generator of type cuda. generator_device() followed _execution_device, which diffusers documents as the device an offloaded pipeline COMPUTES on -- not the one it allocates its initial latents on. randn_tensor is asymmetric about the difference: if gen_device_type != device.type and gen_device_type == "cpu": rand_device = "cpu" # draw on the host, copy over. Fine. elif gen_device_type != device.type and gen_device_type == "cuda": raise ValueError(f"Cannot generate a {device} tensor from a " f"generator of type {gen_device_type}.") So a CPU generator is portable and a CUDA one is a bet that the pipeline wants CUDA tensors. An offloaded pipeline does not, and only SEEDED requests lost the bet -- the same shape of failure this function was written for, one layer down: the first version crashed on a meta device, following _execution_device fixed that and left this. Draw on the host unconditionally. It costs one copy of a few MB per request and buys reproducibility the old behaviour never had: the same seed now gives the same image whether the model is resident, offloaded or sharded, because the RNG stream no longer depends on placement. Seeds from before this change produce different images, which is the price of them working at all. test_the_execution_device_wins_when_diffusers_offers_one still PASSED after the change, by coincidence, while asserting a rule that no longer exists; it is now the offloaded case, and the guarantee that would actually catch a regression -- a CUDA pipeline still gets a host generator -- is tested beside it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
From the model test of 2026-11-01. A mask handed over as a negative. encode_prompt returns a 2-tuple for several families and nothing about its LENGTH says which: SD 1.x means (prompt_embeds, negative_prompt_embeds), Krea 2 and Qwen-Image mean (prompt_embeds, prompt_embeds_mask), MOSS-Image and Kromoss mean (embeds, mask) and name the parameter prompt_attention_mask. The disambiguation looked for one name -- the Krea spelling -- so MOSS fell through to the SD 1.x reading and its boolean [B, S] mask was passed as negative_prompt_embeds. Every MOSS generation then failed with "Prompt embeddings must be [B, S, C] with matching batch size.": true, and about an argument the caller never meant to send. Now the signature is asked for both spellings and, when it is silent, the tensors decide -- a negative matches the positive in rank and width, a mask is boolean or one rank short. The test that pinned the old check was called "told apart by signature not by name" while asserting a single name, which is how the gap survived. An error with nothing after it. str(exc) is empty for any exception raised with no arguments, and interpolating it produced "Image generation failed ... so this is not a load failure." followed by nothing -- measured on Z-Image-Turbo, where the reader could not tell a shape error from a full card. _exc_text always names the type and says plainly when there is no message. A promise the loader cannot keep. component_quantization's docstring said a single-file component could be quantized as it loads. It cannot: from_single_file given a bitsandbytes config takes the PRE-quantized path, so a plain community fine-tune fails after the whole file is read, naming a tensor instead of the cause. Refused up front now -- before the class lookup, which `continue`s when the installed diffusers has no class for the component and so skipped the check entirely -- naming the two routes that do work. Also: the text-only image refusal on /v1/embeddings was gated on `model_obj is None`, so the same request was refused 400 on a cold server and admitted on a warm one, where only the loaded-object check stood between images and a text-only encoder. A correctness check must not read a cache state. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
datalab-to/chandra-ocr-2 is an OCR fine-tune of a Qwen3.5 base, and its chat template still mentions <think>. thinks() therefore said yes, the pod launched with --reasoning-parser qwen3, and that parser -- which assumes the stream starts INSIDE the block and ends at </think> -- filed every page of transcription as reasoning_content and returned content empty. The pod answered 200 OK twenty-eight times and read nothing: no error anywhere, a healthy pod, and $3.34 for six pages that came back blank. The model config had not changed; the :latest pod image had moved to one carrying the reasoning auto-detection, and moving the tag is the deploy. Chandra's generation prompt emits '<think>\n\n</think>\n\n' unconditionally -- no enable_thinking switch anywhere in the template -- so the block is shut before the model emits a token and its output contains no closing tag at all. That is the opposite of the leaking form this detector exists for, which pre-fills an OPEN tag and emits a bare </think>. enable_thinking is what separates them, and it is checked first: a template carrying that switch pre-closes the block only in its thinking-disabled branch (Qwen3.5-9B does exactly this), so the model can still think when asked. Reading the pre-closed pair without checking for the switch would silence every real thinking model, which is far worse than the bug being fixed. Scoped to the generation prompt, because these templates reuse the same tag pair to replay a past assistant turn that had reasoning of its own, and that says nothing about what the model is about to emit. Measured against the four real templates: chandra False, Qwen3.5-9B True, surya-ocr-2 and olmOCR False (no tags at all, unchanged). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Krea-2-RealDream loaded, ran, and died with CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)` which is a 31 GB pipeline on a 24 GB card: 6.07 GB had already gone to host RAM, and cuBLAS could not get the workspace it needs before multiplying anything. _is_cuda_oom matched only the string "out of memory", so the condition was classified 500 -- a server fault, not a busy card -- and the detail named the VAE decode, which was not where it happened. So match the allocator failure whichever library reports it, from a declared list rather than a chain of ifs: out of memory, alloc_failed (cuBLAS and cuDNN both spell it that way), cudaErrorMemoryAllocation, failed to allocate. CUBLAS_STATUS_EXECUTION_FAILED stays out on purpose -- that is a different fault and must not be answered "come back later". The 503 detail now names cuBLAS's workspace alongside the VAE decode, because they are both real and they are not the same place. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
MOSS-Image-SFT, MOSS-Image-NSFW and Kromoss all failed as "Failed to load image model", with a 404 naming the repository -- which read as the model not existing on the Hub. All three repositories are public and complete; the 404 was on a file. They declare _class_name: MOSSImagePipeline, a class that ships in the repo, so kwargs_for_declared_class passed custom_pipeline=<repo id>. diffusers resolves a repo id by one convention only: pipeline.py at the root. These repos call their file pipeline_moss_image.py, so every attempt fetched <repo>/resolve/main/pipeline.py -- a path that had never existed. So read the module name instead of assuming it. model_index.json names it: a component slot holding neither diffusers nor any installed package is naming a module that travels with the weights ("scheduler": ["pipeline_moss_image", "MOSSImageScheduler"]). Given that bare name diffusers does the right thing -- load_pipe_from_hub looks for <module>.py among the repo's files and loads it from the repo. None when the repo does ship pipeline.py, so every community pipeline keeps the resolution that is already correct for it. The log line now names repo and module both, since the gap between them is what a 404 on this path is about. Co-Authored-By:
-
- Oct 31, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two image models "failed to load" on a 24 GB card mid-run. Neither had failed to load: both loaded, and both completed every denoising step before dying on the VAE decode. pornmaster-pro-pony (SDXL, no precision on its entry) loaded at f32 — "Using precision: f32 (torch.float32)" — finished 30/30 steps, then died on "Tried to allocate 1008.00 MiB ... 704.69 MiB is free", with 22.12 GiB of a 23.56 GiB card held by its own weights. The same architecture at half width measures 7.49 GB. Krea-2-Edit, correctly quantized (GGUF Q4_K_M transformer + 4-bit text encoder, bf16), finished 8/8 steps and died the same way on 190 MiB. The bf16-for-an-undeclared-precision default was unreachable: ImageConfig supplied precision "f32" into every per-model config, so by the time the loader asked, nothing was ever undeclared. The default is now "auto" and resolves from the card — a half precision at 32 GB or less, f32 above it, since a card big enough to hold both loses nothing by upcasting. bf16 rather than fp16 for the half type: same two bytes, but fp16 tops out near 65504 and a flow-matching DiT runs past it into NaN and a black image (Z-Image-Turbo, 2026-10-25). An explicit precision, on the card or the model, is still honoured. The reporting was the costlier half. The generate call sat inside the load's `except Exception`, so a decode OOM was filed as a load error, fell through to the sd.cpp backend, and answered "Failed to load image model 'X'. Errors: diffusers: CUDA out of memory". Two separate readers concluded from that wording that VRAM eviction was leaking and that the models never loaded; neither was true, and the real causes went unexamined for it. The same wording had already mislabelled a seeded-request failure once — see models.utils.generator_device. Generation now runs outside that handler and reports itself: 503 and "the model loaded and ran, so this is not a load failure" for an OOM, naming the decode. Also honour per-model `vae_tiling` on the image path. Slicing splits a BATCH and so does nothing for n=1 — which is the case that OOMed; tiling is what bounds the peak of one large decode. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
MM-Embed loads quantized on the card: accelerate is pinned to the generation its transformers expects, because accelerate 1.x moves a quantized model and transformers 4.42 forbids that — which had it running on the CPU at 30x the latency and 33 GB of host RAM. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The last reason MM-Embed ran on the CPU, and the one that only the new diagnosis could have found. Its quantized GPU load died in accelerate's own frame: transformers/modeling_utils.py:3904 in from_pretrained dispatch_model(model, **device_map_kwargs) accelerate/big_modeling.py:514 in dispatch_model model.to(device) transformers/modeling_utils.py:2774 in to ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. accelerate 1.x takes a `model.to(device)` shortcut for a single-device map, and transformers 4.42 refuses that for a bnb model -- so the fallback ran it on the CPU at 30x the latency and 33 GB of host RAM, for a model that fits in 5.7 GB of card. accelerate==0.32.1 now sits in that pin's extras with peft, datasets, pyarrow and bitsandbytes; the shared requirements carry none of them, because only the quantizing pin needs them. Verified live on zeiss after the relaunch: [tf4-embed] nvidia/MM-Embed ready on cuda (4096-d, transformers 4.42.4, quantized) card: 5666 MiB (the planner estimated 5.18 GB) text: dims=4096 finite=True image: dims=4096 finite=True Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An isolated venv goes on the roomiest disk rather than the first one named, a child's stderr reaches the log, a CPU fallback says why it fell back, and a failed GPU load gives the card back. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
It reported "ready on cpu" while holding 9756 MiB of VRAM, 33 GB of host RAM and 1100% CPU. The cause was not the model and not the GPU: /cache had reached 11 MB free of 141 GB -- 15 GB of it pip's download cache from building these venvs, plus two 6 GB venvs put there by this code -- and HF_HOME is on that partition, so the shard download died with OSError: I/O error: IO Error: No space left on device (os error 28) the GPU load aborted part-way, and the fallback loaded the model again on the CPU. Everything about that was invisible: * The child's stderr was read only if it exited silently, so the reason sat in a pipe nobody drained -- which is also a pipe that fills, at which point the child blocks writing to it and looks like a model that hangs. It is now pumped into the engine log for the child's whole life. * "ready on cpu" said nothing about whether the CPU was asked for or fallen into, and the difference is 30x and tens of GB. The fallback now carries its reason (gpu_error) and the parent logs it loudly. * The aborted GPU load kept what it had already allocated for the life of the process: 9756 MiB that the manager's ledger could not see and nothing else could use. The card is now released (gc + empty_cache + ipc_collect) before the CPU retry. And the placement that caused it is fixed: a NEW venv goes on the roomiest candidate root, not the first one named. The engines' own CODERAI_CACHE_DIR is /AI/guffcache with 197 GB free while the default cache root had 15 GB; six GB has to go somewhere, and it should not be the fullest disk. An existing venv still wins over a roomier empty root, so nothing is built twice. One of this file's own tests was also writing a pointer file into the REAL config root, naming a pytest tmpdir, which a later process would then have resolved the venv from. Removed, and a test now fails if any test here calls ensure_installed() without redirecting _pointer_file. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
MM-Embed is served out of a venv pinned to the transformers generation its remote code was written for (4.42.4, on python 3.12), beside jina-clip-v2's 4.49.0 one. The interpreter-provisioning ladder is shared with the lip-sync engines rather than duplicated. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Moving it out of the server process was the only way to serve it at all, and the move turned up what its remote code really needs. Every step here is a measured constraint, in the order it surfaced on zeiss 2026-10-31: * peft, then datasets. Its own modeling file imports peft; the retriever it loads with trust_remote_code (nvidia/NV-Embed-v1, named in its config.json) imports datasets. transformers checks both by SCANNING the files' imports, so each absence surfaced only as the next load failure. * The same retriever is why transformers 5 was never the problem to shim around: it imports MISTRAL_INPUTS_DOCSTRING, deleted in 5, and runs its OWN decoder loop, calling each Mistral layer without `position_embeddings` -- which transformers computed inside the attention layer until the rotary refactor of 4.43. On 4.49 it dies in transformers' own code with "cannot unpack non-iterable NoneType object"; its card says transformers==4.42.4. So the venv is per PIN, not one shared "transformers 4" venv. * 4.42.4 pins tokenizers<0.20, whose newest wheel is cp312, so that venv needs python 3.12 -- and the image's /usr/bin/python3.12 has no `-venv` package, prints "After installing the python3-venv package…" and STILL EXITS 0, leaving a tree with no pip. The check is now `pip --version`, not the exit code, and the ladder falls through to a fetched standalone interpreter. * 4.42.4 also pins numpy<2, which the current peft and datasets cannot live with: peft 0.21 imports EncoderDecoderCache (>= 4.43) and datasets 5 pulls pyarrow 26 ("requires NumPy 2.0 or newer"). Pinning datasets alone left pyarrow 26 in place, because datasets 2.20 accepts pyarrow>=15. Hence PIN_EXTRAS: peft 0.11.1, datasets 2.20.0, pyarrow 16.1.0, beside the transformers version they belong to and in no other venv. Verified end to end through coderai's own worker, child and protocol: [tf4-embed] nvidia/MM-Embed ready on cpu (4096-d, transformers 4.42.4) MM-Embed: text 4096-d image 4096-d finite=True and jina-clip-v2, unmoved on its own 4.49.0 venv, still answers 1024-d with cos(it, image)=+0.2588 and cos(ar, image)=+0.2688. Three defects the bring-up found in the machinery itself, each fixed with tests: * The probe decided whether an EXISTING venv was usable from five imports, so a dependency added to the requirements never reached one: ensure_installed returned early and skipped pip entirely. It now lists everything a pin installs, including its extras. * The venv path was not the same in every process. CODERAI_CACHE_DIR is set per engine (/AI/guffcache there, /cache elsewhere), so the engine built one venv and another process built a second -- 11 GB of the same thing, and the models page would have reported a venv the serving engine was not using. The chosen path is now recorded in the config root, which the whole fleet shares. * A child's load failure arrived without a traceback: one line of "TypeError: cannot unpack non-iterable NoneType object" from another process, in another venv, inside a model's remote code. It now carries one. The interpreter-provisioning ladder the lip-sync engines needed first moved to codai.api.venv_env, because this needed exactly the same thing. The register shim that let MM-Embed overwrite LlavaNextConfig in-process is deleted: out of the process that cannot happen, and a test keeps its remote code from ever being imported here again. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three fixes from a customer test run on a card that was nearly full: a configured used_vram_gb is no longer halved by the checkpoint's storage dtype (which admitted a 7B embedder on 2.75 GB for a 5.6 GB load, and it OOMed on its first batch), the eviction that frees scratch space no longer waits for the very model it is freeing it for, and MM-Embed no longer leaves every LLaVA-NeXT model in the process loading as MM-Embed. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Found by a customer test run on zeiss 2026-10-31 that asked for four embedders in a row. The third, MM-Embed, failed on its gated retriever; the fourth, e5-v, then answered 500 Failed to load embedding model: 'LlavaNextConfig' object has no attribute 'retriever' e5-v had worked all day on its own, and the only thing between them was the order. MM-Embed's `modeling_nvmmembed` calls `AutoModel.register(LlavaNextConfig, NVMMEmbedModel)` at import time, which is process-wide, and e5-v IS a LLaVA-NeXT checkpoint — so once MM-Embed has been touched, every LLaVA-NeXT model afterwards is constructed as MM-Embed and asks its config for MM-Embed's field. A model's own 500 became another model's. coderai has to permit that registration (transformers 5 refuses a duplicate, and it refuses during the import, so loading the class directly does not avoid it). What it must also do is put the mapping back, which the first version of the shim did not: it restored the `register` METHOD and left the entry that had been written through it. Now it records each override it allows -- in the lazy mapping's `_extra_content`, read with `in` rather than a one-argument `.get`, which that mapping does not have -- and restores or removes it afterwards, including when the load it was allowing fails. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Found during a customer test run on zeiss, 2026-10-31, with a 26B GGUF resident in the sibling engine on the same card: [vram-est] key='embedding:…gme-Qwen2-VL-7B-Instruct' source=used_vram_gb -> 2.75 GB | used_vram_gb=17.0 load_in_4bit=True quant_mult=0.300 prec(load2/disk4)=0.50 Measured footprint for '…gme-Qwen2-VL-7B-Instruct': 5.64 GB total [embeddings] encode OOM (Tried to allocate 1.02 GiB … 954 MiB free) Active model '…gme-Qwen2-VL-7B-Instruct' is busy — waiting for it to go idle before eviction… Two defects, in sequence. `used_vram_gb` is a VRAM figure, and by this estimator's own convention an UNQUANTIZED one: 17.0 x 0.300 = 5.1 GB, which is what the loader then measured. The storage->load precision ratio was applied on top because the checkpoint declares torch_dtype float32, halving a number that was already in the right units. That ratio belongs to a baseline derived from the bytes on disk and nowhere else, so _apply_factors takes `from_disk` and the configured path says no. The quantization multiplier still applies — the opposite error, measured earlier the same night, turned a declared 7.0 into 1.13. Then the recovery. The retry sets active_in_vram to keep the eviction pass off the resident model, but the pass's second phase is designed to evict the ACTIVE model as a fallback — so it targeted the model serving the request and waited for it to go idle, which it cannot do while blocked on the memory that pass was called to free. The wait budget went on a model it must not touch, while 17.8 GB sat idle in the sibling engine and the cross-engine releaser that would have asked for it runs only afterwards. _evict_models_for_vram now takes `protect`: never evicted, never waited on, in either phase. Four source-anchored tests moved with the code and are corrected here. One of them — the Wan2.2 Lightning pair — was only passing BECAUSE of the halving: its config omitted the load_in_4bit the real cards carry, so a declared 22 GB read as 11 and "fits a 24 GiB card" was true by accident. With 4-bit as production configures it the figure is ~7.8 GB, and true on purpose. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
jina-clip-v2's isolated venv gets a library path of its own, so its image tower runs instead of failing in the server's cuDNN. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Live on zeiss, through the restarted server: the subprocess came up on cuda, `POST /v1/embeddings` with text returned the same 1024-d vector the offline run measured, and the same request with an image returned RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed ptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED Half the model worked because only the vision tower runs convolutions. The cause is the environment, not the venv: env_for_venv keeps LD_LIBRARY_PATH for a venv built by THIS interpreter -- correct for one that shares the server's native libraries, and wrong for this one, which ships its own torch 2.15+cu132 and its own CUDA/cuDNN wheels and was therefore loading the server's. So env_for_venv/env_for_python take `own_libs`, a third case that is not about the interpreter at all: a venv on our python with a CUDA stack of its own gets our lib directories filtered out, while entries from outside the install (a system CUDA, anything the operator added) are left alone as always. jina-clip passes it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The lip-sync engines can be installed from the image that offers them, the RunPod pools stop paying for pods they are not allowed to keep, jina-clip-v2 embeds images instead of returning NaNs, and the OCR fleet workers read the hub cache the rest of the install uses. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The model's remote code is pinned by its own config to revision 39e6a55 (July 2025) and jinaai's jina-clip-implementation has had no commit since, so it is not transformers-5 code and will not become it. On the server's transformers 5.12 it needs two shims merely to import -- `clip_loss`, deleted from modeling_clip, and the `torch_dtype`/`dtype` rename -- and with both applied it loads and returns non-finite image vectors at fp32, bf16 and fp16, with xattn on and off, inside coderai and outside it. A NaN vector is worse than an error: it scores every comparison equally, silently. Measured the same day, same weights, same images, on transformers 4.49: text : shape=(2, 1024) finite=True nan=0 norm=0.9992 image: shape=(1, 1024) finite=True nan=0 norm=1.0007 cos(text, image): 0.258 / 0.260 on CUDA and CPU, and on python 3.11 and 3.13 alike -- so the venv pins transformers and nothing else, and is built with the SERVER'S OWN interpreter: there is no second python to find, which makes this an install rather than a bring-up. coderai owns it the way it owns pyannote's and NeMo's: baked into the image when it is there, built on demand when it is not, with a Build / Rebuild / Delete card on the models page beside the lip-sync engines so the minutes of pip do not land on somebody's first request. The model runs in it as a subprocess (`jina_clip_server.py`, one JSON object per line, the dinov2/SALAD protocol extended with texts because this model embeds both into one space). Its VRAM is reclaimable -- eviction kills the child, and the next request respawns it -- and calls are serialised, because one stdin pipe shared by two threads reads the other's answer. Three things the bring-up found, each of which looks like something else: * `dtype=` fails on 4.49 ("unexpected keyword argument") exactly as `torch_dtype=` fails on 5.12 ("attribute name must be string"), because the remote code forwards **kwargs into its own __init__. Neither is passed: the checkpoint's config.json already names bfloat16, which is what the verified run loads in and what the EVA-style vision tower needs. * requirements-jinaclip.txt is NOT in a running container -- only codai/, tools/ and third_party/ are mounted over the image -- so the install uses venv_env.pip_install_args and the same pins inline, with a test that keeps the two copies equal. * The in-process path is gone rather than kept as a fallback. There is no stack this process can offer the model that produces a usable vector, and a second code path that silently produces NaNs is worth nothing. Verified on zeiss through coderai's own code: venv built, server ready (1024-d, transformers 4.49.0), and one photograph against Italian, Arabic and Japanese text giving finite vectors at cos 0.2588 / 0.2688 / 0.2247. That run was on CPU -- a throwaway container on this host cannot be given the card -- so the cuda branch is the same torch.cuda.is_available() check every other worker uses and is exercised by the live server, not here. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
download_model refuses a fetch that produced no bytes ("the server sent no data for …"), which is right: a zero-byte GGUF is not a model. The stub in this test returns content-length 0 and no chunks, so it hit that guard instead of checking the one thing it exists for — that a real URL reaches requests.get rather than being resolved as a local path. One byte of body, and it checks that again. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
codai/hf_cache.py stopped putting TRANSFORMERS_CACHE in `<hf>/transformers` and pointed it at `<hf>/hub` like the other three, for a measured reason: transformers 4.x honours that variable as a HUB-layout cache rather than as a store of its own, so a directory beside the hub does not give transformers its own corner -- it gives the machine two hub caches, and which one a process reads depends on the transformers version in its venv. On 2026-10-28 `<hf>/hub` held 122 models and `<hf>/transformers` held 2, and a worker forced offline reported We couldn't connect to https://huggingface.co for a model that was sitting in the other directory all along. codai/ocr_fleet_workers.py carries its own inline copy of that environment -- deliberately, because importing the package into the supervisor costs exactly what the supervisor exists to save -- and the copy was not updated, so every OCR fleet worker was still being handed the second cache. The test that covers it had been failing since, along with two more that the same change left asserting the old path; those are stale assertions and are corrected here. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Measured on Saturday 2026-10-31 against a pool whose warm-pod window is weekdays only, so the warm floor was 0 and the idle timeout is deliberately collapsed to one second ("do not wait out idle_timeout_s before releasing a pod that only existed to be warm"): 09:44:53 pod w9u0tgs1reu5dn created for 'pool:digesta-qwen38' — booting 09:52:06 port open after 433s (image pull + container start) 09:58:39 weights resident after 392s — ready means ready 09:58:39 pod w9u0tgs1reu5dn ready for 'pool:digesta-qwen38' 09:58:54 pod w9u0tgs1reu5dn terminated (idle); billed ~$0.002 and the same cycle at 08:54, 09:05, 09:15, 09:34 and 09:42 — roughly $0.14 of A40 each, four to six times an hour, for pods that never answered a request. The pod came from the ANTICIPATORY trigger: "renting one more for the next burst, not for work already queued". That trigger now asks first whether the pool may hold an idle pod at all (_reaps_on_arrival: warm window closed, or a client has asked for the fleet back) and declines when it may not, saying so once rather than on every pass. What it was renting the pod to do was not something the pool was allowed to let it do. Demand-driven growth is NOT gated this way: that work exists now, and a pod that takes it is not idle. The ledger also stopped the waste being visible. PodHandle.started_at was stamped after the image pull and the warm-up, so every one of those cycles was recorded as "$0.002" when fourteen minutes of A40 had been billed. It is now taken when the pod is CREATED, which is when RunPod starts charging. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two config entries can point at one checkpoint and differ only in settings -- lisa and lisa-32k are the same 26B GGUF at n_ctx 178000 and 32768, measured 51.97 GB and 14.33 GB -- and _get_model_used_vram_gb already prefers the config of the id actually REQUESTED for exactly that reason. The disk floor added last night then threw that away in the next breath: [vram-est] sizing 'Wan2.2-I2V-A14B-Lightning' by its own config (shares a weights file with 'Wan2.2-I2V-A14B-Diffusers') used_vram_gb=22.000 GB for 'Wan-AI/Wan2.2-I2V-A14B-Diffusers' is well under the 126.18 GB of weights on disk — sizing against the disk instead So the floor is skipped when another entry shares the same path and declares a different size: for the smaller member of such a pair the file on disk is not a bound at all. What the floor catches is a value written before a download finished, and nothing declares a second size for a half-downloaded checkpoint. Both source-scraping tests that this moved are also fixed to stop measuring in characters. One split on the FIRST occurrence of "needs more than the whole card", which is now a comment quoting the message while explaining a different decision; the other looked 1800 characters back from the floor for its rationale. Each failed while the thing it guards was there, word for word. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
LatentSync and MuseTalk each run in their own venv on their own interpreter, because each pins a stack the server's 3.13 cannot host. Making that venv had two routes: `python3.11 -m venv`, and uv. The shipped container has neither -- the bundled 3.13, the distribution's 3.12, no uv and no apt-get at run time -- so the admin page's Install button failed at the first step with could not create a python 3.11 venv for latentsync at …: no python3.11 on PATH; uv not found i.e. two engines coderai advertises as installable were uninstallable from the image that advertises them, and the presenter went on ranking them first. An interpreter is now resolved before the venv is built, because the two failures read nothing alike: PATH, then a standalone interpreter the image carries (the bundle builds /opt/coderai/py310 for the wav2lip venv exactly this way) or one an earlier install provisioned under the config root, then -- only when there is no uv either -- one fetched from python-build-standalone, the same publisher and release the image's own pythons come from. A `python3.11` on PATH that will not run no longer consumes the attempt. A venv built on the host and then used from inside the container is also repaired rather than abandoned: its bin/python points at a /usr/bin/python3.11 the container has not got, and `python -m venv` into that directory fails on the dangling symlink ("[Errno 2] No such file or directory: …/bin/python3.11"). The dead links go, the cp311 wheels already installed stay, and only a directory nothing can adopt is rebuilt from scratch -- once. Two downloads also moved out of the render path, which is where this module says weights never come from: * MuseTalk's s3fd detector (86 MB) is fetched by its vendored face_detection at module import, via torch's model zoo, so it cannot be a WeightSet. The install now runs that constructor once (EngineSpec.warm) and TORCH_HOME points at the install root, so the file lands beside the engine's other weights instead of in a container layer that a recreate throws away. * LatentSync's buffalo_l (~280 MB) is downloaded by insightface from inside FaceAnalysis.__init__ unless checkpoints/auxiliary/models/buffalo_l is already there. The two modules its face detector allows are now part of the install and part of its ready markers, so an install without them is not "complete". Verified on zeiss, in a container with the live install's mounts: both engines report available and both entrypoints run (`python -m scripts.inference --help` with the engine's own env), where before the venv was missing in the container while is_installed() said yes. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Four text-and-image embedders were added last commit; making them actually answer took three fixes measured against the live models. Quantization now skips the vision tower and the projector by default (hf_loading.quant_skip_modules). For a vision-language EMBEDDER those are the part being judged: the tower turns pixels into the vector that gets compared, the projector is one small matrix whose error lands on every image, and together they are a fraction of a 7B's weights -- so nf4 there saves almost nothing and spends it in the worst place. An explicit empty quant_skip_modules on the model entry still means "quantize all of it". jina-clip-v2 and MM-Embed each needed one thing transformers 5 took away: * `dtype=`, not `torch_dtype=`. The old name is still accepted for most models, but a model whose remote code forwards **kwargs into the config dies with "TypeError: attribute name must be string, not 'torch.dtype'", which reads like a corrupt checkpoint. * `AutoModel.register(..., exist_ok=True)`. MM-Embed's modeling_nvmmembed registers LlavaNextConfig, which transformers already owns, and 5.x refuses a duplicate instead of ignoring it. * `clip_loss`, deleted from modeling_clip while `contrastive_loss`, which it is two lines of, is still there. Restored from it rather than vendored. Embeddings are also checked for NaN/Inf before they are returned (_require_finite): a vector of NaNs scores every comparison equally and silently, and that is what jina-clip-v2's image tower returns on this stack -- so it now answers with an error instead of nonsense. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The footage checker scores a photo against a text description, and of everything configured only `gme-Qwen2-VL-2B` could do it: the rest are text-only (Qwen3-Embedding, bge-m3), image-only (dinov2, salad, vpr) or coordinates (geoclip-location). Its Q4_K_M sibling crashed the engine, which is a separate bug in the GGUF unload path. Three of the four need no new embedding code, which is why their documented API was read before any was written: gme-Qwen2-VL-7B-Instruct Qwen2-VL weights -> the existing `qwenvl` backend, detected on model_type rather than repo name royokong/e5-v SentenceTransformer.encode(texts | images), 4096-d BAAI/BGE-VL-MLLM-S2 SentenceTransformer.encode({'text':…, 'image':…}) `_embed_images` already routes a sentence_transformers model through `model.encode(pil_images)` with a `[{'image': im}]` fallback — exactly the two shapes those models document. nvidia/MM-Embed does need its own backend, because its API resembles nothing else here: model.encode([{'txt': …, 'img': …}], max_length=4096, is_query=True, instruction=…)['hidden_states'] `trust_remote_code` is passed by the loader rather than taken from the model entry: the checkpoint's encode() lives in its repo, so without it the model cannot load at all, and a missing operator flag would be a silent failure. An `instruction` switches it to query mode — without one it embeds the item as a PASSAGE, which is the right default for "embed this" and the wrong one for a query, so the distinction is kept rather than hardcoded. NOT claimed: coderai embeds texts and images as separate items, so MM-Embed's combined text+image QUERY form is not reachable through /v1/embeddings. Text-vs-image similarity in one space — what the footage check needs — works regardless. One mistake caught in passing: 'geoclip' was briefly added to the image-capable backend list. Its two modes SHARE that backend name, so it would have claimed that geoclip-LOCATION accepts images when it takes coordinates. Removed, with the reason recorded beside the list. timm reaches the image at last. It has been declared in requirements.txt since the depth/segment endpoints were added and absent from every incrementally updated image — the same overlay gap as geoclip and vosk, found the same way, by listing the built image's packages rather than reading the source tree. jina-clip-v2 cannot load without it: its EVA vision tower IS a timm model. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
**Eviction headroom.** Every size reaching eviction is an estimate, and one landing a few hundred MB short does not fail honestly: the load is admitted, starts allocating, and dies on a tiny buffer. Measured on zeiss across 2026-10-30/31, each on a load eviction had just reported as satisfied: Tried to allocate 20.00 MiB … 8.69 MiB is free Tried to allocate 190.00 MiB … 18.69 MiB is free Tried to allocate 12.00 MiB … 8.69 MiB is free Eviction now targets `needed_gb * 1.10`. The headroom is what it TARGETS, never what it reports: the shortfall warning and the "needs more than the whole card" verdict both still measure the caller's actual request, so freeing enough for the load but not the whole margin stays silent. A warning on the padded figure would fire on every successful eviction and bury the one real warning. 10% of the request costs at most one more idle model, and scales — a flat margin would be too small for a 30 GB model and absurd for a 0.1 GB one. **JSON mode returned JSON that would not parse.** 21 script generations ended `finish_reason=length` after writing 6000 tokens of well-formed-but-unfinished JSON for a job needing ~800; one ran 8.5 minutes. Nothing was misconfigured: max_tokens was the caller's own 6000, the prompt 2414 tokens, the model's n_ctx 32000. `json_object` IS honoured, but its grammar constrains syntax only and never requires the root object to close, so an array of scenes grows until the budget ends mid-token. Two changes, neither touching the caller's max_tokens: * `json_schema` is now accepted, which bounds the SHAPE so the object must close. llama-cpp-python takes only `type: json_object` and reads a sibling `schema` key, while OpenAI nests it under `json_schema` — so the shapes are translated. Forwarding OpenAI's form verbatim would have silently DISABLED the grammar (`_grammar_for_response_format` returns None for other types), which is worse than not supporting it at all. * When the budget is hit anyway, the open structures are closed so the caller gets valid JSON holding the scenes that did arrive. It drops a half-written element rather than closing around it, never invents content, parses its own output before returning it (an unverifiable repair is not applied), leaves prose and already-valid JSON alone, and does not rewrite `finish_reason` — which stays `length`, so a caller that checks still learns it was cut short. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An eviction freed 16.3 GB and the load that triggered it saw 0.0 GB. 23:58:05 Loading 'image:…pornmaster…': need 30.9 GB VRAM, have 2.9 GB free — evicting models to make room 23:58:05 unloading active model 'FLUX.2-klein-4B' (measured 17.24 GB) 23:59:23⚠️ Eviction freed 13.4 GB (now 16.3 GB free, needed 30.9 GB) 23:59:25 needs ~32.9 GB host RAM (30.9 GB footprint against 0.0 GB free VRAM spills 30.9 GB to the host, plus 2.0 GB to stage it), and only 27.8 GB is available — short by 5.1 GB. Refusing the load. Two seconds after eviction reported 16.3 GB free, admission read 0.0, and no other load intervened — the next one starts a second later. `_get_free_vram_gb` subtracted every in-flight reservation, including the one belonging to the load being judged: `max(0, 16.3 - 30.9)`. A reservation exists so OTHER loads do not count VRAM already spoken for. Subtracted from the reserving load's own view it says the room it just reserved does not exist, so the whole footprint looks like a host spill, the host-RAM budget refuses it, and the eviction that preceded it cost another model its residency and bought nothing. That is where 17.24 GB of resident FLUX pipeline went, twice over, in the hour that ended in the first OOM kill. `exclude_key` leaves one model's own reservation out of the subtraction, and the admission path passes its own key. Every other caller — eviction, sizing, placement — keeps the conservative figure, which is correct for them: they are deciding on behalf of a load that has not reserved anything yet. Verified by reproducing the arithmetic: with the old view the need is 32.9 GB and the shortfall 5.1 GB, matching the server's own message to the decimal; with the load's reservation excluded the same load needs 18.6 GB and is admitted. NOT the fix that was proposed first. Checking admission BEFORE evicting would have been worse: at that point 2.9 GB were free, so it would have refused sooner. Eviction HELPS the host-RAM verdict by shrinking the spill — more free VRAM, less spill, smaller host requirement, monotonically. It only has to be visible to the load that paid for it. A test pins that ordering claim so the earlier idea cannot be reintroduced as an improvement. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Fixing a regression I shipped hours earlier in d83075a4. Found in production by the floor misfiring, on the same night: [vram-est] key='image:black-forest-labs/FLUX.2-klein-4B' source=used_vram_gb(disk-floor) -> 25.61 GB | used_vram_gb=17.0 Loading 'image:black-forest-labs/FLUX.2-klein-4B': need 25.6 GB VRAM, have 0.0 GB free — evicting models to make room FLUX.2-klein-4B declares `used_vram_gb = 17.0` and really occupies 17.24 GB -- the eviction log prints the measured figure when it unloads it -- while its cached repo is about 23.7 GB. A diffusers cache holds every component (transformer, two text encoders, VAE) and routinely duplicate formats and variants, .bin beside .safetensors and fp32 beside fp16, of which one subset loads. So the repo on disk is not a lower bound on VRAM, and the floor raised an already-accurate figure past the 24 GB card. Each load then announced "this model needs more than the whole card", evicted the resident fleet, and gained nothing. The two errors are not symmetric, which is the whole lesson. The under-estimate the floor prevents made ONE text model attempt a load that could not fit. An over-estimate on an image pipeline costs OTHER models their residency, on every load, while the model it was protecting was already sized correctly. So the floor now applies only where it was measured to help: single-checkpoint text and vision models, never image or video pipelines. It reached a running engine because the engine was OOM-killed and respawned against the bind-mounted tree, so a commit made minutes earlier went live without a deliberate restart. Worth remembering when reading a log from a box whose code is bind-mounted: a crash is also a deploy. Also: the size test wrote two real gigabytes and failed with OSError on a host whose root filesystem was 98% full. It truncates a sparse file now -- the helper reads st_size and never the contents. Co-Authored-By:
-
- Oct 30, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Auto: Qwen/Qwen3.5-9B is quantized — trying full-GPU regardless of the 3.1 GB estimate (5.6 GB free) Error loading model Qwen/Qwen3.5-9B: … needs about 3.1 GB of VRAM to itself; 5.6 GB is free after evicting what could be evicted. That reads like a bug in eviction -- 5.6 is more than 3.1 -- and is not. The estimate was wrong in the dangerous direction: too SMALL, so the loader believed the model fitted three times over, attempted a full-GPU load of something needing ~6.5 GB, and failed. The refusal itself is correct and stays: a quantized model cannot be served from CPU offload, and would answer every request with an empty completion. The chain, from the debug line: source=used_vram_gb -> 3.13 GB | used_vram_gb=7.795 measured_cfg=8.67 load_in_4bit=True prequantized=False quant_mult=0.300 `measured_vram_gb=8.67` -- the real 4-bit footprint -- was correctly discarded, recorded under a different quantization signature; that guard earns its keep elsewhere and is untouched. The fallback `used_vram_gb=7.795` is a STORAGE-size baseline that the quant factors apply on top of, so it became 7.795 x 0.300 + reserve = 3.13 GB. The multiplier is right: the checkpoint is 19.31 GB of bf16 with `quantization_config: None`, verified on disk, so `prequantized=False` is also right. 7.795 was not the size of anything. The post-download backfill writes `cached_size * 1.2` and runs from the download thread, so an interrupted or still-running fetch leaves a figure for a fraction of the checkpoint: 6.5 GB downloaded x 1.2 = 7.8. So the field can hold a number nobody checked, and the factors then scale it down again. The checkpoint on disk is measurable fact available at that moment, and is now a floor -- UPWARD only, with a 1.25x margin, so an operator's larger value stays theirs and an honest small difference is tolerated. The factors still apply to the floor, because the disk size is a storage size: 19.31 GB at 4-bit is ~5.8 GB, not 19. On the incident model that is 5.8 + reserve against 5.6 GB free, so the load makes room or is refused honestly instead of being attempted on a figure three times under the truth. `source=used_vram_gb(disk-floor)` in the debug line, because a floor reporting itself as an ordinary estimate would be invisible to whoever reads these. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Every image sent to a GGUF vision embedder answered 500. Caught on live traffic by the 5xx logging added hours earlier, which named it in one line: [radeon] mtmd_tokenize: error: number of media markers in text (0) does not match number of bitmaps (1) [radeon] [http] 500 on POST /v1/embeddings: 'Embedding failed: mtmd_tokenize failed (2)' `mtmd_input_text` carries a `text_len` beside its `c_char_p`, and the struct was built with the constructor form, which leaves it at 0. Measured in the container: that form gives text_len == 0 for a 38-byte prompt, the field-by-field form gives 38. llama.cpp read a zero-length prompt, found no media marker in it, and refused the bitmap it had been handed. The marker was never the problem -- mtmd_default_marker() is b'<__media__>' and _prompt() interpolates it, both verified against the running engine, and both were suspected first. Built field by field now, exactly as llama_cpp's own chat handler does it, which also keeps a NAMED reference to the encoded bytes. That second half matters on its own: `text` is a c_char_p, so a temporary is free to be collected while the struct still points at it -- a latent defect in the same two lines that would have been intermittent rather than total. Verified against the running server afterwards: image-only, text-only and text+image to gme-Qwen2-VL-2B-Instruct-Q4_K_M all return 1536-d embeddings. Each of the two image cases was a 500 before. The request log now reports the image count. Its absence is why the search started in the wrong place: `model=gme-… inputs=4 chars=84` reads as a text-only request, because _summarise logged `input` and never `image`, so a failure on the IMAGE path looked impossible. Counts only, no content, like the rest of that line. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Follow-up: the footprint fix earlier today addressed one of two sources of the same wrong attribution, and the live log showed the other one an hour later. footprint: 8.48 GB total (weights 7.71 GB (0.00 on GPU, 7.71 on host), …) Measured footprint for 'Gemma-4-14B-A4B-Heretic.i1-IQ4_XS': 8.48 GB total (… of which 7.71 GB was offloaded to host RAM this load; n_gpu_layers=-1) That model was loaded with `n_gpu_layers: all` and was holding 10.1 GB of VRAM. `measure()` has no torch object to walk for a GGUF, so it takes the weights total from the file size -- correctly, a GGUF is loaded as-is -- and banked it in `weights_cpu_gb`, which keeps the TOTAL right and asserts a placement nobody measured. "0.00 on GPU" is the exact opposite of the truth for a fully resident model, and that line is what an operator reads while chasing a CPU-spill problem. I read it that way myself this evening and called the earlier mismatch cosmetic. `weights_split_known` now records that the split is unknown, `describe()` says "placement not measured here" instead of inventing one, and the manager distinguishes the three different things that reach `measured_ram`: a per-module measurement (genuinely on the host), a GGUF's unsplit total, and the process's RSS growth. Only the first is reported as an offload, and only the first puts a size on the Tasks page -- which had been labelling fully-GPU-resident GGUFs "CPU offload 7.71 GB". The arithmetic is untouched in both paths. The total is what sizing and eviction consume, it was right, and under-reporting a footprint is how eviction stops freeing room and the forward OOMs. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Five faults, found by one client asking where the OCR boxes had gone. **The boxes.** `_page_from_pod` rebuilt a pod's page key by key -- index, text, width, height, and a named list of diagnostics -- and never copied `lines`, `regions` or `tables`. Every page read on a pod therefore arrived with no geometry at all, for every engine, and no request parameter could change it: the pod puts all of it on the wire (it runs the same `OcrPage.to_dict`) and this boundary dropped it. `confidence` survived only by being one of the names. From outside it looked exactly like a missing engine feature, which is the expensive part: the client saw `lines=0` with a populated `conf`, concluded the engines were not emitting boxes, and went hunting for the parameter that turns line detection on. There is none -- `detect` is stamp/signature detection (off|layout|detector|both) and has nothing to do with text lines. The reassembly now lives once, in `codai.ocr.base.page_from_dict`, shared with the worker-subprocess boundary that had the same job and the better implementation of it. Diagnostics ride by exclusion rather than by name, which is what that boundary already learned after losing `accents_repaired` and then `model_attempts` the same way. A test asserted that "confidence" and "confidence_measured" appeared in the SOURCE of the pod reassembly. They did -- in the key-by-key list that was discarding the geometry -- so it passed throughout. It now runs the boundary. **geoclip answered 500 for text.** Reproduced: "una frase" loses its space, and "unafrase" is eight characters of the base64 alphabet, so it decodes to six bytes and only PIL can tell it is not a picture. PIL raises UnidentifiedImageError, which is NOT a ValueError, so it escaped the caller's `except ValueError -> 400` and became "the server broke" for bad input. Fixed at both ends: `_decode_image` raises ValueError quoting the input, and geoclip's IMAGE half joins vpr/salad in the pre-load refusal, because its contract puts the image in `input` -- so a text `input` is answerable from the name, before 1.9 GB is loaded. Its LOCATION half is deliberately excluded: it takes coordinates there and already answers a clean 400 naming the format. **The 5xx logging I added this morning caught nothing.** Starlette lifts a handler registered for status `500` out of the status table and gives it to ServerErrorMiddleware, which fires only for an UNHANDLED exception -- so `raise HTTPException(status_code=500, ...)`, which is how nearly every 500 here is raised, went straight past it. Confirmed live: two geoclip 500s, no `[http]` line, and the cause had to be found by replaying the request. Now registered on the exception class, logging at >= 500, with the status-keyed 400 handler still winning for its own richer line. Exercised with a TestClient rather than asserted from the source, because the first version passed every source-level check and caught nothing. **A model whose package is absent is no longer advertised as fine.** vosk and geoclip were both listed for as long as they existed while answering every request with an import error, because nothing asked. `codai/models/deps.py` answers it with a memoised `find_spec` -- never an import, never a model load -- and is used in two places: the listing annotates `unavailable_reason` (the model stays listed; hiding one that works is the worse mistake) and the embeddings path refuses with a 503 naming the package and the two files it has to be in. Only MAIN-venv dependencies are declared, because a package that lives in an isolated worker venv is absent from this one by design. **A fully-GPU-resident GGUF reported itself as offloaded.** "21.47 GB total ... of which 19.52 GB was offloaded to host RAM; n_gpu_layers=46", for a model with 46 of 60 layers on the card and ~15 GB resident there. That number is the process's RSS growth, and llama.cpp mmaps the whole file, so a GGUF grows RSS by about its file size whatever the split. The arithmetic is untouched -- the TOTAL is load-bearing for sizing, it was right, and under-reporting a footprint is how eviction stops freeing room and the forward OOMs -- but the line now says which of the two numbers it is holding, and the Tasks page no longer prints a delta as an offload size. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The engines run with LD_LIBRARY_PATH=/opt/coderai/local-libs first, and that beats the wheel's DT_RUNPATH. So llama-cpp-python dlopens its own libllama.so by absolute path, and that library then resolves libggml-base.so.0 against local-libs -- which still held the PREVIOUS build, because replacing the wheel never touched that directory. A 0.3.36 wheel (libggml 0.25.3, libllama 0.5.0) against local-libs' 0.9.7/0.0.8351 produced, in all five engines at once: Failed to load shared library '.../llama_cpp/lib/libllama.so': undefined symbol: ggml_rope_set_offset That is no GGUF serving at all. It hid because `docker restart` keeps the container's original image, so the image built this morning was never executed until the container was recreated this evening -- the rebuild was reported as deployed and the running process was still 0.3.30. The libs this build produces are now copied over the stale ones, so both search paths resolve to the same version in whatever order they are found. Orphaned older sonames are left alone: harmless once the soname every binary asks for points at the new file, and removing them would break anything else in the bundle still linked against them. The check that passed on the broken image is replaced. `ldd` answers "can every needed library be found" and is blind to a library that IS found and is the wrong version; `ldd -r` relocates too and names the symbol exactly as the runtime does. Two details decide whether it proves anything, and the first version of it got both wrong: - it must run with LD_LIBRARY_PATH set the way the engines set it, because the search ORDER is the entire bug; - it must check the WHEEL's libllama.so, the file actually dlopened, not local-libs' own copy. Old libllama against old libggml is a self-consistent pair that reports nothing: checked that way, this image passed with zero findings. Checked correctly it reported ten. Verified in a throwaway container against the broken image: 10 undefined symbols before, including ggml_rope_set_offset, and 0 after. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
One request, five minutes, fourteen cores, the GPU at 6%, on a card with room for the whole model. Eviction was never the problem -- it freed 14.2 GB and handed the loader 22.9 GB. Two defects then threw that away. The journal key was `<path>@<n_ctx>`, so a 7.5 GB KV measurement taken before swa_full=False stayed reachable after it. The real cache was 1.0 GB. That inflated the requirement from ~16.2 GB to 22.7 GB. The key now carries a hash of the settings that decide the cache size (swa_full, cache_type_k/v, kv_offload), so a measurement taken under one configuration is a miss under another; an unreadable config still yields a signature, because orphaning old entries is the safe direction and over-invalidating only costs one measurement. The sizing then never asked for more room. It sampled free VRAM once, found itself 0.2 GB inside a 95% trigger, and gave layers away while another engine still held VRAM it would have released on request. It now asks the manager to evict what a full-GPU placement needs, re-samples, and skips the spill entirely if it then fits; a failed eviction is not fatal and leaves the old decision. This is only safe because _evict_models_for_vram consults _is_key_busy and waits for idle -- and that check became trustworthy for embedding models when external_busy was wired into the embeddings path earlier today. Separately, geoclip never worked. `geoclip` and `geoclip-location` were advertised by /v1/models and both answered 500 in half a second, because they share _load_geoclip and the package was absent from the image: declared in requirements.txt, which the incremental overlay never installs from. Exactly the fault vosk had, in the same file, for the same reason, and fixed the same way -- including `--target`, since the venv is not writable by the container user and a plain install lands in ~/.local where the engine will not find it. Found by listing the built image's packages; it is invisible from the source tree. The install was verified in a throwaway container: all three imports resolve. And the reason that took so long to find: there is a 400 handler that logs why a request was rejected, and no 500 handler. The detail existed ("No module named 'geoclip'") and went only to the client, which in a capability sweep is a row in somebody else's report; the engine log held the thermal check, the VRAM estimate, then `Response status: 500`. A 400 is a statement about the request and a 500 is a statement about us, and this side logged the first and not the second. Behaviour unchanged; the class is now diagnosable. Four KV tests asserted exact argument lists and broke on the signature change. They now match the call through collapsed whitespace, via one shared helper, so the next reformat does not break them again. Co-Authored-By:
-
