- 28 Oct, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authored`precision` was read by the loader and listed in main.py's _LOAD_AFFECTING, but it was in neither the admin save allowlist nor the model page, so the one setting that decides how much VRAM a card actually needs could not be configured anywhere. The dtype then came from a default derived from GPU detection -- and a card's precision is a property of the card, not of the hardware it happens to land on. The select sits with the quantization boxes because that is where it is read wrongly: 4-bit and 8-bit quantize the WEIGHTS, while the embeddings, lm_head, the norms, the activations and the KV cache all stay in this dtype, so a 4-bit model left in fp32 still pays fp32 for every one of them. The hint says so, and says that fp32 also rules out Flash Attention. "" is stored as null, so "not configured" stays distinguishable from a value and the loader's own default still applies. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZnLhydX9QqUcfBbqLu
-
Stefy Lanza (nextime / spora ) authored
NvidiaBackend.__init__ leaves self.device None. The loader picked the precision default as `'fp16' if self.device == 'cuda' else 'fp32'` at line 897 and only detected the device at line 1029, so the FIRST load in a fresh engine process defaulted to fp32 on a CUDA card -- while a second load in the same process inherited 'cuda' from the first and quietly got fp16. The same model loaded at two precisions depending on load order. fp32 costs twice the VRAM of the dtype asked for, which on a 24 GB card is the whole headroom argument for quantizing in the first place: Qwen3.5-9B is carded load_in_4bit with an estimate of 7.795 GB and measured 14.403 GB, a 1.85x gap that is the fp32 tell. FlashAttention also refuses fp32 outright, so transformers warned twice at load, cast the inputs back to fp32 at inference, and the model answered every request with "FlashAttention only support fp16 and bf16 data type" -- and before that error was surfaced at all, with an empty completion. It had been unusable since 2026-09-30. Also resolve flash_attn by PRESENCE rather than truthiness: `config.get(k) or global` cannot express "off", so a card saying `flash_attention: false` fell through to the global default. Not the cause here (the global is false too), but the same class of bug one layer up. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The video path reserves a model's VRAM estimate for its whole load window so a concurrent load sees the card as spoken-for. The image path registered only a Tasks-page entry, so for as long as a load took, the card looked free. Measured 2026-10-28: Krea-2-RealDream held 18.8 GiB of a 23.56 GiB card for the 194 s of its load, ensure_vram_for granted a CrisperWhisper reservation against VRAM that was already gone, and every captioned scene of a concurrent compose answered "CUDA out of memory. Tried to allocate 86.00 MiB". Both ends had been fixed in isolation -- the STT path reserves, the worker releases -- and neither helped, because the model they were negotiating over never declared itself. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
place_at padded a mix up to the scene length with apad and let atrim end the stream. That handoff is a race. A bare apad loses it by generating silence for ever: ffmpeg never reaches EOF and spins at 100% CPU with the output half-written -- 4 hangs in 10 runs, and two orphans left on a dev host by test_compose_dialogue had burned 14h and 10h of CPU unnoticed. apad=whole_dur stops the hang and loses the same race the other way, emitting a track one clip short (3s for 4s) in 3 of 14 runs with no error anywhere. Neither -threads 1 nor -filter_threads 1 helps. A silent base input of exactly the target length, mixed under the placements with amix=duration=first, has no handoff to lose: 0 bad in 20 runs, one output size. The other apad sites feed no mix, but take whole_dur too since the length is known there as well. Also pass config/subfolder to the single-file component loader, the way the GGUF branch already does and for the same reason: left to itself from_single_file infers a base repo from the checkpoint's own metadata, so Krea-2-RealDream reached for krea/Krea-2-Raw and died on a repo nobody named. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The lip-sync engines were outside the ledger entirely. A render is gigabytes on the card for minutes, in a process the manager did not start, so the next model to load read the card as emptier than it is, took what it found, and one of the two died. They are models.json rows now -- video_models, backend "lipsync", capabilities lip_sync -- configurable on the models page like anything else, and a run goes through acquire_stt_backend, the same door LongCat and the STT workers use. The footprint is sampled during the run (these are third-party scripts that report nothing about themselves) and kept as a high-water mark, so the next reservation is measured. The registration is dropped when the process exits: one-shot work that stayed in the ledger would claim VRAM nobody holds. The bigger half is a defect that was never about lip sync. acquire_stt_backend POPS the instance pool, and _is_key_busy looks only at that pool -- so every model registered through it reads as permanently idle. LongCat mid-generation, a CrisperWhisper transcription, a lip-sync render: eviction was not choosing to take the card from busy work, it could not see the work was there. LongCat's own source notes this and works around it inside its releaser; nothing else did. So multi_model_manager grows external_busy(key), and _is_key_busy consults it. The existing eviction logic then does the right thing with no further change: the first pass skips a busy model, the second waits for the active one. An in-flight render is something to queue behind, not a candidate -- cleanup() waits for the process and never terminates it, because a half-finished render is worth nothing and the request that triggered the eviction is not more important than the one already running. Also: the Civitai redirect fix moved the seam two tests patched, so they stopped intercepting and started reaching the real site -- which is why they failed on HTTP 404 rather than on the 401/403 pages they assert about. Every request now leaves through one function, which is the thing to patch. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two things that were only reachable from a python -c. The lip-sync engines had no management surface at all: no status, no install, no removal. They are not models.json entries on purpose -- each runs in its own interpreter as a subprocess, so a row in a model section would put them in the VRAM ledger as something the manager can load, which it cannot -- but that is a reason to give them their own surface, not none. So: a tab on the models page, three admin endpoints behind require_admin like every other model action, and an install that runs in the background while the page polls. The log is kept because when an install fails, WHERE it failed is the whole answer: a wheel that will not build reads nothing like a 401. "Rebuild venv" keeps the weights, since a dependency change does not alter six gigabytes. `component_models` could only name a repo. A community fine-tune is very often ONE file -- a retrained transformer with no repo around it, the base supplying the VAE, the text encoder and the scheduler -- and from_pretrained cannot open one, so such a component could only be named if somebody had published a converted directory tree for it. It now accepts a single-file or civitai: reference. And the Civitai resolver picked the LARGEST file of a version. Real Dream's Krea 2 V3 publishes the same weights three times: fp8 13.46 GB, int8 13.18 GB, nvfp4 8.12 GB. Largest-wins lands on fp8 by a 0.28 GB margin nobody chose, and could as easily have landed on nvfp4, which needs Blackwell and cannot load on an Ampere card at all. A reference may now name the file, and without one the choice is made among the builds this GPU can run, by precision rather than size. Two smaller ones found by using it. urllib replays headers across a redirect, so Civitai's signed R2 URL received a bearer token as well and answered 400 -- which looks exactly like a bad token, and meant adding a token BROKE downloads that had worked without one. curl strips it; now so does this. And the cache has its own root (CODERAI_CIVITAI_CACHE) instead of living under the HF cache, which is how a second GGUF directory grew inside one. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Four pins and one probe, each found by building it rather than reading about it. mmpose's RTMO head imports mmdet unconditionally although mmpose lists it as optional, so `import mmpose.apis` dies without it; mmdet then asserts mmcv < 2.2.0 at import; openmmlab publishes a cp311 mmcv 2.1.0 wheel on exactly one index, cu121/torch2.1. So the torch pin is 2.1.2 and it is dictated by mmcv, not by MuseTalk. mmengine imports pkg_resources to find mmpose's config directory, which setuptools dropped in 81, making setuptools<81 a runtime pin. chumpy, which mmpose depends on, has a setup.py that does `import pip` and cannot build in an isolated environment. Named per package rather than turning isolation off wholesale, which would let everything else build against a half-installed venv. The venv probe was wrong, not the install: it ran a bare interpreter, so face_detection -- vendored under musetalk/utils, not a package -- came back missing and a complete install was reported broken. It runs against the install root with the engine's own PYTHONPATH now, as a real run does, and asks for mmpose.apis rather than mmpose, since the package imports fine and the submodule is what fails. Also: uv's HTTP timeout is too short for hundred-megabyte wheels off third-party indexes, and the abort reads as a corrupt wheel rather than a timeout. And a per-run directory is swept after a day -- a failed run keeps its task file and frames, which is what you read to find out why, but not for ever. Verified live: MuseTalk v15 lip-synced a 2s 480x480 clip in 92s, 48 frames with the audio muxed, and the successful run removed its own directory. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
_reject_offloaded_quantized already catches a 4-bit model whose weights landed on the CPU -- that is what stopped Qwen3.5-9B answering every request with "". But it runs at the BOTTOM of the device_map ladder. So a full-GPU attempt that OOMs still walks every rung: 427 weight shards across GPU and host RAM, many minutes of it, before raising an error that was knowable when the ladder started. The line printed immediately above the fallback says as much -- "quantized kernels cannot read offloaded weights" -- and then falls through to the ladder anyway. The other half is that the attempt which failed was never retried honestly. It ran against whatever happened to be free on a shared card, which is a lottery; another model's residency is a reason to evict, not a reason to mis-place this one. Reproduced on the live server today: 7.2 GB free because an embedding model and a whisper-server held the rest, and nothing asked them to move. So a quantized model that fails full-GPU now evicts and tries once more, and refuses immediately if that still does not fit, naming the three things that help. Unquantized models keep the ladder: offload is legitimate for dense weights, only slower. This makes a failure faster and louder rather than slower and quieter, which is a behaviour change on a live path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
LatentSync and MuseTalk have been selectable by name for a while, and `engine: "auto"` has ranked both ahead of wav2lip whenever there was a clip to drive -- wav2lip works at 96x96 and upscales, which is the soft mouth on a 1080p reel. Nothing ever installed them, so every selection ended at "neither on PATH nor installed", and the call that would have been made could not have worked either: LatentSync requires its checkpoint and UNet config as well as the three paths, and MuseTalk takes a YAML task list and writes into a result directory rather than taking paths at all. Coderai orchestrates what it serves, so it installs these. The inference source is vendored under third_party/ and pinned to a commit, for the reason LongCat's is: an image someone else pulls has to run without network access to a third party. The weights come from the Hub on first use, never during a render. Each engine gets its own venv on its own python, because both pin stacks the server cannot host. Upstream MuseTalk takes its BiSeNet face-parsing checkpoint off a Google Drive id and a pytorch.org URL; both come from a Hub mirror here, so the whole set has one provenance. _managed_engine_ready used to answer "an entrypoint and any .pt anywhere", which reads an interrupted download as a working install. It now asks the module that knows the file list, and the old heuristic keeps covering an engine an operator installed by hand. /v1/video/presenter/engines reports what an uninstalled engine is short of, so a client can tell "not yet" from "never here". Verified live: LatentSync installed from nothing and lip-synced a 2s 480x480 clip to real narration in 136s. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three CrisperWhisper models were configured and they stacked: each is a resident Whisper-large in its own process, so the card held two of them plus the server's text and embedding models until one OOMed inside the encoder's softmax. Nothing here was the model's fault. The reservation taken before a load was a hard-coded 3 GB -- roughly the fp16 weights, and about a third of what a verbatim word-timestamp pass peaks at, since that holds the encoder's attention for a whole 30s window. The worker now reports its own peak and the figure is recorded as a high-water mark, so the next reservation is measured rather than guessed. The eviction ladder had no way to reach a worker's VRAM: it lives in another process, and acquire_stt_backend pops the model pool, so the manager's busy signal cannot see a transcription either. There is an external releaser now -- idle workers go at once, a busy one is waited out to the drain budget and only then taken -- and asking for one model stops its idle siblings. A cold start was invisible while it lasted. Free VRAM comes from the driver, so a worker downloading its shards looks like room; live traffic loaded into it and the transcription OOMed on a card nobody over-committed on purpose. The load window holds a reservation now. whisper-server already evicts and knows when it is idle, but it sized itself at file size x 1.1, which counts the weights and not whisper.cpp's compute buffers. Co-Authored-By:
-
- 27 Oct, 2026 28 commits
-
-
Stefy Lanza (nextime / spora ) authored
The map takes 4bit / 8bit / 2bit or a .gguf path. Anything else was skipped without a word -- and one unusable entry empties the mapping, which takes the global load_in_4bit with it, so the component loads from the base repo at full precision while the operator believes it comes from the file they named. LTX-2.5 named a bf16 Gemma-4 12B for its text_encoder. The quantization config came back None, the 12B loaded unquantized, and every generation ended with the card full: 23.53 GiB of 23.56 GiB in use, unable to allocate 30 MiB, on a model whose own measured footprint is 14.45 GB. Four strategies and about an hour of 20-minute attempts, all of them really the same missing line. Now it says so, and names component_models -- the key that does what such a value was meant to do. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
video.py already knew the hazard and wrote it down: accelerate's device_map only dispatches components it LOADS, so one handed to from_pretrained keeps whatever device it is on with no offload hook, and the forward pass hits "index on cuda:0, weights on cpu". It acted on that only for pipeline-cache injections, while a GGUF component arrives exactly the same way. LTX-2.5 therefore failed under every strategy it tried -- auto, balanced, model and pure-disk, three restarts and half an hour each -- always with Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu and video_resident_experts, the flag that exists for precisely this, was never consulted because the gate in front of it was false. An explicit offload_strategy is still respected; only the strategies that let accelerate place components itself are overridden. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The LTX-2.5 key reconciliation shipped, the retry failed identically, and the repair had never run. It needs the component's config.json to know which names the class expects, and _component_dir looked only in the local snapshot -- which, for a component whose weights come from a GGUF, is precisely the subfolder nobody downloads. 20 GB of weights the file replaces are not worth fetching; a few hundred bytes of config are. Cache first, then one hf_hub_download of that component's config.json. Same lesson as a cache-only existence check on a rented pod: an answer of "not here" from a cache is not an answer about the repo. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The repair that fixed LTX-2.3's prompt-adaln names could only strip a suffix. LTX-2.5's GGUF needs a whole segment renamed: it calls the attention norms `q_norm` / `k_norm` where LTX2VideoTransformer3DModel expects `norm_q` / `norm_k`. 724 parameters, one to one, the same tensors -- and unreconciled they stay on the meta device, so a 37-minute load ends with Cannot copy out of meta tensor; no data! about weights the file was carrying all along. Measured by diffing the GGUF's 4349 tensors against the class's 4091 parameters: nothing was absent, 724 were simply spelled differently. Both directions, because nothing says which spelling the next exporter picks, and both repairs now run in one pass -- a checkpoint can need each for a different family of keys. The existing guard is unchanged: a key moves only when the class does not want its current name, does want the renamed one, and nothing already holds the target. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
diffusers reports a missing single-file checkpoint as Invalid `pretrained_model_name_or_path` provided. Please set it to a valid URL. which sends you to check the repo id. The actual cause, measured on LTX-2.5, was an entry pointing into ~/.cache/huggingface on the HOST -- a path the container does not mount -- so the file existed everywhere except in the process trying to open it. Every other part of that model was ready. The path is probed before the load and named in the failure, with the reason it is usually absent. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
_apply_sampler turns a cross-family sampler into a 400 that explains itself: flow-matching and DDPM schedulers parameterise the trajectory differently, so one cannot stand in for the other. The acceleration config applied its `scheduler` with no such check, and a card is exactly where that mistake gets made and never noticed. Krea-2-Turbo's entry asked for EulerDiscreteScheduler on a flow-matching model. The swap succeeded silently and the generation then died with EulerDiscreteScheduler.set_timesteps() got an unexpected keyword argument 'mu' which names neither the card nor the model, arrives from inside diffusers, and reads like a library bug. Now it is refused and logged, naming both schedulers, and the model keeps its own -- acceleration is an optimisation and has always been allowed to decline. One set of family names, imported by both paths, so they cannot drift. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Krea 2 returns (prompt_embeds, prompt_embeds_mask), and the generic path read any 2-tuple as SD 1.x's (embeds, negative_embeds) -- so it handed the mask over as a negative prompt and the pipeline answered If `prompt_embeds` are provided, `prompt_embeds_mask` also have to be passed. which reads as a missing argument rather than a mislabelled one, and arrives wrapped in "Failed to load image model" two and a half minutes into the load. Told apart by the pipeline's own __call__ signature rather than a list of family names, and the answer for those families is to pre-encode nothing: weighted 75-token windows are a CLIP-pair feature that means nothing to them, which is why Z-Image already takes that path. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The OCI image is a capture of a virtualenv: whatever has fallen out of the venv is gone from every install that pulls the image, and nothing in the build log says so. `gguf` left this venv at some point, and the rebuild duly shipped without it -- which breaks every GGUF component load with Unable to load weights from checkpoint file for '<file>.gguf' because diffusers takes the .gguf branch, fails to `import gguf`, and that ImportError is swallowed into a generic OSError about the file. Nothing in the message mentions a package. The baseline is the PREVIOUS IMAGE, not requirements.txt. That file lists optional extras this venv has never had -- vosk, timm, edge-tts and ten more -- so checking against it reports thirteen false alarms and teaches everyone to pass --no-req-check. What the last image shipped is what this install has today, and losing any of it is a regression by definition. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
With PYTHONHOME stripped, SadTalker's 3.10 got as far as importing torch and died there: ImportError: /opt/coderai/local-libs/libtorch_python.so: undefined symbol: PyObject_GetOptionalAttrString a CPython 3.13 API. The container launcher exports LD_LIBRARY_PATH as /opt/coderai/python/lib:/opt/coderai/local-libs:<the server venv's nvidia wheels>, so a child on another python loads THIS one's extension modules -- the same bug as PYTHONHOME, one layer down, and it hides the same way: by hand in the same container the import works, because an interactive shell has none of it. Filtered rather than dropped. An isolated venv ships its own torch and CUDA wheels, but a system CUDA or anything the operator put on that path is theirs to keep -- and an empty LD_LIBRARY_PATH is not an unset one, it means the current directory. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Krea-2-Turbo and LTX-2.5 have sat disabled with the same note: the classes they name -- Krea2Pipeline, LTX2DurationHead, LTX2VideoDiffusionDecoderModel -- did not exist in diffusers 0.38. They are in 0.42.1, a release rather than git main, so the floor moves. The pin that matters is the one next to it. diffusers 0.42 wants huggingface_hub >=1.32, transformers 5.x refuses >=2.0, and pip resolves 2.3.0 given the chance -- which leaves transformers unimportable and every model with it. The usable window is 1.33 and it is written down here rather than rediscovered. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Measured on the live server, on the first presenter request that ever ran: sadtalker failed (rc=1): init_fs_encoding: failed to get the Python codec of the filesystem encoding / No module named 'encodings' The shims on PATH run the bundled python 3.10 venv, and the process that spawns them exports PYTHONHOME=/opt/coderai/python -- the standalone 3.13. The child dies before executing a byte, the job falls back to the scene's visuals, and the warning blames the engine. Same root cause as the ACE-Step worker's "requirements missing", so it uses the same helper; the deliberate PYTHONPATH these repos need is set after cleaning, pointing at the checkout rather than inheriting the server's. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
presenter.remove_background was accepted and then warned about: the one field of the compose contract with nothing behind it. A pip bubble that is a circle of someone's kitchen reads as a sticker; the same bubble with the person cut out reads as a presenter. The mask is the whole feature -- it replaces the bubble's shape mask, so composite_pip needed one line and composite_split needed the B-roll behind the person's half. Two engines: a TorchScript matting model (an RVM export is the obvious one, and being recurrent it is steadier over time) when CODERAI_MATTING_MODEL names one, else torchvision's DeepLabV3, which is already in the stack and needs no configuration. It borrows the GPU under the same lease lip-sync takes and is dropped as soon as the scene's mask is written -- a 100 MB segmenter is not something the manager should have to evict. Nothing about it is fatal: a scene that cannot be cut out still gets its presenter, filmed room and all, and says so in the warnings. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The ACE-Step variant was only reachable by running the whole build again with --with-acestep: ~20 GB of bundle recopied to add 6.4 GB that has nothing to do with it, and two builds that can disagree. The overlay makes the relationship literal -- the baked image IS the slim one plus this venv, one layer on top, 461s instead of a second 15-minute build. The build-time check runs the venv's python with PYTHONHOME unset. The image sets it to the standalone 3.13, and a 3.12 inheriting it dies before executing a byte, so without `env -u` the check would fail on the trap rather than on the venv it is meant to prove. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Live on zeiss, /v1/audio/generate answered in 0.25s with "ACE-Step requirements missing", which was not the problem: the venv was there and imported fine by hand in the same container. The container's per-engine launchers export PYTHONHOME=/opt/coderai/python (the standalone 3.13); PID 1 and an interactive docker exec do not. A 3.12 interpreter inheriting it aborts before running a byte -- "Current thread ... <no Python frame>" -- so the worker's import probe failed instantly, and the caller went down the rebuild path to report a missing file instead. Reproduced both ways before changing anything. longcat_worker had already solved this for its 3.10 venv, behind a docstring saying coderai does not set those variables. It does. The helper is shared now (codai/api/venv_env), longcat delegates to it, and every ACE-Step spawn -- probe, both venv-creation routes, the install, the service -- goes through it. The probe also prints what the child said rather than returning a bare False. Then response_format: declared as url | b64_wav | b64_mp3, and ignored. The response was chosen by whether a file directory was configured, so a client asking for base64 silently got a URL -- the same bug /v1/images had with b64_json. Honoured now, and b64_mp3 is really encoded, because base64 of a wav under an mp3 key is a decode failure at the client. Also corrected in passing: versions.env pins UBUNTU_VERSION=24.04, whose system python IS 3.12, so the baked venv needs no bundled interpreter. The Dockerfile rewrote pyvenv.cfg at /opt/coderai/py312 unconditionally, which would have broken a working venv when nothing was bundled there, and the build refused outright without a relocatable 3.12 it did not need. Verified live after the fix: b64_wav returns 5,742,296 bytes (~15s of 48 kHz stereo), eviction by a 21 GB model stopped the worker and took the GPU from 7958 MiB to 274 MiB in under four seconds, and ACE-Step then reloaded and generated again. Suite at its 15-failure baseline, 5299 passed. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Plan only -- nothing built. Three things it had to establish rather than assume: * the pod cores are python:3.12-slim, which is exactly what ACE-Step needs. The bundled standalone interpreter the local image requires (Ubuntu 22.04's system python is 3.10) is simply not a problem on a pod. * ACE-Step cannot go in the `audio` profile: that one has no `.light` marker, so it builds on the FULL core, and 7 GB of unused CUDA plus a 6.4 GB venv is far over the measured ceiling (10.1 GB boots, 14.5 GB is refused). * the `gguf` package is in requirements.txt but in NEITHER core.txt nor video.txt, and pod images install core.txt -- so no pod image can load an LTX-2.3 single file today. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
ACE-Step generates a whole track -- style prompt, sung lyrics, a length in seconds -- and it ships a package pinning transformers 4.50 and spacy 3.8.4. spacy 3.8.4 publishes no cp313 wheel, so this is not a stack the server would rather not hold: it is one that cannot be installed on the server's interpreter. It gets a python 3.12 venv of its own and an HTTP service in it, driven the way CrisperWhisper, Canary and pyannote are. Built on the first request to such a model, so a model that is configured and never called costs nothing; `uv venv` covers the hosts that have python3.12 without ensurepip. Three upstream details that fail silently rather than loudly: * the pipeline resolves its own dtype as `bfloat16 if dtype == "bfloat16" else float32`, so asking it for float16 quietly doubles the VRAM. ACE_PIPELINE_DTYPE is read after that switch, so precision goes through there instead. * constructing the pipeline loads NOTHING -- it reads the checkpoints on the first call. coderai measures a model's footprint as the free-VRAM delta across the load, and that window closes when the worker reports healthy, so a lazy load would measure ~0 GB and eviction would treat a resident 3.5B model as free. The service loads before answering /health. * `quantized=True` does not quantize: it loads a separate int4wo export from the same directory layout. Missing, that is a FileNotFoundError minutes into a load, so the worker names the file it wants first. The handle cached in the manager exposes cleanup(), which is what eviction calls, and here it stops the worker -- the only thing that frees its VRAM. `duration` and `cfg_coef` are shared with MusicGen and default to 10s/3.0, which would make every ACE-Step call a 10-second clip at a guidance it is not tuned for. The branch reads which fields the client actually SENT, so an omitted one gets ACE-Step's default and an explicit `duration: 10` still means 10. `lyrics` and `steps` become declared fields: both branches already read `steps` off the request, where only extra="allow" was letting it through. The venv is ~6.4 GB, so the default image does not carry it and --with-acestep bakes it in -- copying the dev venv when there is one, building it when there is not. Ubuntu 22.04's system python is 3.10, so a baked 3.12 venv also needs a relocatable interpreter bundled beside it, like the 3.10 venvs. Verified on a 3090: healthy after ~3.5 min, 8.5 GB VRAM, a real 48 kHz stereo WAV from a 10s/12-step request. Suite at its 15-failure baseline, 5292 passed. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The silent one. self.models is keyed by model_key, and model_key was the weights path -- so two entries naming the same path shared ONE resident pipeline. LTX-2.3-PinkCherry requested while LTX-2.3-10Eros was resident never loaded its own transformer: no load, no eviction, straight to generation, and HTTP 200 with 10Eros weights. Every other instance of this family failed loudly; this one answered successfully with the wrong model. It also meant both wrote measured_vram_gb to the same key, so a Q4_K_M and a Q5_K_M of different sizes were overwriting each other sizing. A key is split only when the caller named something, two or more DISTINCT aliases point at that same path, and the named one is among them. A lone aliased model, or a request that names the path, keeps the key it has always had. That narrowness is the point, not caution for its own sake: collapsing every name form onto one key is deliberate a few lines above -- without it two concurrent first-requests under different spellings each load an instance and exceed max_model_instances=1. This is not a licence to key by whatever the caller typed. Verified live: the estimator now reports key=video:LTX-2.3-10Eros where both models previously shared key=video:diffusers/LTX-2.3-Diffusers. Suite: 15 failed, 5268 passed -- the same 15 as the baseline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
LTX-2 masks its text conditioning. The model loaded cleanly and died at the first denoising step: ValueError: `attn_mask` is not supported for sage attention so `auto` choosing sage there is not a slower path, it is a broken one. The first version of this guard moved it to flash instead, and it failed again with the same sentence and a different name: ValueError: `attn_mask` is not supported for flash-attn 2. NEITHER fast backend accepts a mask, so falling back from one to the other is not a fix. For these families the only working backend is the default SDPA, and the guard now keeps it. An operator who names sage or flash explicitly is still obeyed -- the override exists so they can force a backend, and silently ignoring it would be a different kind of wrong from silently choosing a broken one. They get the error, which is theirs to get. Wan and the LTX 0.9 family are unaffected and still take the fast backends; tests pin both directions. Suite: 15 failed, 5258 passed -- the same 15 as the baseline. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A request for LTX-2.3-10Eros loaded LTX-2.3-PinkCherry: a different 15.9 GB transformer, silently, and from an entry that was DISABLED. self.config is keyed <kind>:<model_name> where model_name is the weights path, so entries sharing a path clobber one another and only the last survives. That is not a corner case -- it is how one base checkpoint is served under several configurations: lisa / lisa-32k differ only in context length, 10Eros / PinkCherry only in which GGUF is their transformer. PinkCherry comes second in models.json, so it won, parked or not. The VRAM estimator already disambiguated this way, by the id the caller asked for, after lisa-32k was once sized at lisa 51.97 GB. The config used to LOAD the model did not. Now both do: * every type setter stores a config under its ALIAS as well as its path, so a sibling is reachable by the only name that distinguishes it. The path key keeps its old meaning (last writer wins) for callers that know only a path. * _config_for_model_key takes the requested id and prefers the entry whose alias matches. Applied to all eight setters rather than video alone -- the collision is the same for audio, image and embedding, and fixing one type is how this survived being found twice already. Also: build_gguf_pipeline_components passed `model_name` as the diffusers repo supplying the component configs. Once siblings became reachable by alias that can BE the alias -- "LTX-2.3-10Eros", not a repo -- and from_single_file rejects it. The repo is whatever the entry names in `path`, every time. Verified live: LTX-2.3-10Eros now loads its own transformer and the pipeline assembles, 8/8 components. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The LTX-2.3 GGUFs failed with "Cannot copy out of meta tensor; no data!", and the obvious reading -- parameters the checkpoint lacks -- was wrong. diffusers 0.38 converts an LTX-2 checkpoint prompt-adaln weights to prompt_adaln_single.* and audio_prompt_adaln_single.*, while LTX2VideoTransformer3DModel expects prompt_adaln.* and audio_prompt_adaln.*. Twelve names, one-to-one, the same tensors. They were present all along, under a name nobody reconciled, so they never materialised and stayed on meta. Measured identically on LTX-2.3-10Eros v1.5 and PinkCherry v1.8: 4444 GGUF tensors, 4186 expected parameters, and after the rename 0 missing and 0 unused on both. The tempting fix was to initialise those twelve. That would have invented weights to paper over a rename -- precisely what the surrounding refusal exists to stop -- so it is NOT what this does. A key is moved only when the class does not want its current name AND does want the renamed one AND nothing already occupies the target, which makes it a rename of a tensor the checkpoint has and never the creation of one it lacks. A test pins that: a genuinely absent parameter must stay absent and the load must still fail. The converter is a third-party global, so it is wrapped for the duration of one from_single_file call and restored in a finally. Version-robust: 9 pass against the image diffusers 0.38, including the real checkpoint; against 0.37 the class has no prompt_adaln modules at all, so the repair is correctly a no-op and that test skips rather than asserting a fact about someone else is version. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
"Cannot copy out of meta tensor; no data!" means some parameters were never materialised, and it names none of them -- which sends the reader hunting for a corrupt file. Measured on LTX-2.3-10Eros and -PinkCherry: diffusers already ships convert_ltx2_transformer_to_diffusers and it works almost completely, mapping their 4444 GGUF tensors onto exactly the 4186 the class expects. Precisely 12 remain unprovided, all in one conditioning branch, identically in both fine-tunes. So the checkpoints are fine and the converter is fine; those fine-tunes were exported without that branch, and the 12 stay on the meta device. The refusal added earlier now carries that analysis: how many tensors mapped, how many remain, and which submodules they belong to. Best effort throughout -- any failure to analyse returns nothing and the original error stands. It deliberately does NOT offer to fill them in. Supplying parameters a checkpoint does not contain is a substitution, and the surrounding refusal exists precisely because a silent substitution (here: the 38 GB bf16 transformer, on a 24 GB card) is worse than a failure. A test pins that: the hint must not grow a flag that papers over it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/models re-reads models.json on every call, but the loader's alias map is built ONCE at startup. So an entry added while the server runs is advertised immediately and cannot be loaded at all: the alias resolves to nothing, the raw alias reaches diffusers as a repo id, and the caller gets Repository Not Found for url https://huggingface.co/api/models/<alias> -- a 401 against HuggingFace for a model sitting on the local disk. Measured today registering Wan2.2-TI2V-5B: weights cached, `path` correct, and the request failed that way; the SAME model requested by its path loaded immediately, which is what pinned the cause to the alias map rather than the entry or the weights. The pre-existing Z-Image alias, updated rather than added, worked throughout -- that contrast is the whole diagnosis. This is the fifth instance of one pattern in this codebase: a name in models.json not matching the name the runtime registered. v0.5.4 read a backend from the wrong map, v0.5.5 looked for a binary where it does not live, v0.5.7 published an id that resolved to nothing, the LTX pair shared one path between two entries, and now the listing advertises what the loader cannot resolve. A cross-cutting identity-resolution audit is still the right fix; this is the narrow one. An alias the map has not seen is now looked up in the config itself and cached. A parked entry is deliberately NOT resolved -- parking is how an entry waits on a blocker, and resolving it would hand clients a model that answers 500. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Found by generating a real image against the live server: a request with "response_format": "b64_json" came back {url, path} and no bytes. The request model documents OpenAI's values -- url and b64_json -- and every OpenAI SDK sends the latter. save_image_response compared against this codebase's own older internal spelling, "base64", so with --file-path configured the documented behaviour ("if the client explicitly requested base64, include it") could never fire for a standards-compliant client. It got a URL, no bytes, and nothing in the response to say why. Nine call sites were affected: generations, edits, inpaint, upscale, depth, segment and the rest. Both spellings are accepted now. The url path is unchanged -- the point of --file-path is not to ship megabytes inline. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
LatentSync (Apache-2.0) and MuseTalk (MIT) are now first-class presenter engines and come before wav2lip for a clip, so the compose documentation says which engine engine:"auto" will pick and why: wav2lip works at 96x96 and upscales, which is where the soft mouth on a 1080p reel comes from, while these two run at 256-512 px natively. sadtalker still leads for a still portrait because it is the only one that invents head motion rather than replacing a mouth. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Found by running the endpoint against the live server. A request naming `face_match.character_profiles` came back 400: face_match was requested with no reference images which is the opposite of the truth -- the caller HAD supplied references. The cause was a bare `except Exception: pass` around the per-profile lookup, so a misspelt name, a profile with no saved images, and a characters directory the server resolves elsewhere all collapsed into that one message with nothing to distinguish them. Establishing which it was took a long sequence of probes against a running server: the field is declared (a bad type 422s with loc body.face_match.character_profiles), the profile exists under both candidate paths in the container, and resolve_character_profiles returns its four images when called in-process -- so the swallowed exception was the only thing standing between the symptom and the cause, and it is the defect. Now each profile is resolved separately and its failure is named. When none resolve the 400 lists them; when some do, the request proceeds (there ARE references to match against) and the unusable ones are logged, because a performer the caller believes is enrolled and is not would make a genuine face read as "matches no consenting performer" for the wrong reason. Verified live on the same server, all four checks against a real face: nsfw 0.0003, one face at det_score 0.879, age 38.0 with the interval 30.0-46.0 and minor_probability 0.0, face_match similarity 1.0 matched against itself, unmatched_faces 0, flagged false. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
9ae240b0 committed its tests and its documentation and none of its implementation: four files instead of twelve. The `git add` that should have staged the rest named a pathspec for requirements-acestep.txt, which did not exist, so git refused the whole invocation and I did not check its output before committing. The message described work that was not in the tree, and the suite at that commit could not have passed. This is that work: * video.py — LTX-2.x gets its own pipeline classes. This diffusers has LTX2Pipeline, LTX2ImageToVideoPipeline, a video AND an audio VAE, but only the LTX 0.9 family was mapped, so every LTX-2 entry in models.json sat parked: the name matched and the class did not. Split on the version in the name, since both families are just called "LTX" and a fine-tune keeps its base version (10Eros is LTX2.3, PinkCherry is LTX23). * hf_loading.py — `civitai:<modelVersionId>` references, and a GGUF component that fails to load now fails the load instead of being skipped. A skipped component is simply absent from the dict, so the base pipeline loaded its own full-precision copy: the operator asked for a 14 GB quantized transformer and got a 38 GB bf16 one, while the VRAM estimate still said 14 GB and the planner therefore freed about a third of what the load needed. * presenter.py — LatentSync and MuseTalk as first-class engines, ahead of wav2lip for a clip (wav2lip works at 96x96 and upscales, which is the soft mouth on a 1080p reel). Both are video-driven, so sadtalker still leads for a still portrait. Recognised from a shim on PATH or a managed install whose weights are already present — never "could be downloaded", because a compose job is the wrong place to fetch 5 GB. * cache/__init__.py + app.py — the loop stall. scan_cache_dir is not a listing: it stats every blob of every revision of every repo, measured at 27.3 SECONDS on this machine's 900 GB cache, and it was called from a per-model VRAM estimate, a GGUF path lookup and a POLLED endpoint, all on the event loop. 6631 stalls over 0.5s, worst 13.0s, serving nothing. Both halves were needed: memoised for 30s (short enough that a model downloaded while the server runs still becomes visible) and the poll moved to asyncio.to_thread, since memoising alone still puts the first 27-second call on the loop. * manager.py — /v1/models reads `modes` from a video entry's long-standing `video_subtypes` as well as the newer spellings. * requirements-acestep.txt — the file whose absence caused this. It is marked NOT YET INSTALLED, because it is: the install was refused as external code, nothing in it has been run, and the audio_gen loader refuses an ACE-Step model explicitly rather than mis-loading it as a MusicGen. Suite: 15 failed, 5194 passed — the same 15 as the baseline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
serve the LTX-2 family, fetch a Civitai checkpoint, and stop walking 900 GB of cache on the event loop Model installation work from the client's C list, plus one latency bug found on the way. LTX-2 is wired. This diffusers has LTX2Pipeline, LTX2ImageToVideoPipeline, a video AND an audio VAE — but video.py only mapped the LTX 0.9 family, so every LTX-2 entry in models.json sat parked: the name matched and the class did not. Split on the version in the name, since both families are just called "LTX" and a fine-tune keeps its base version (10Eros is LTX2.3, PinkCherry is LTX23). Stable Audio had its own entry in the type map pointing at AudioLDM2Pipeline — a different architecture with a different projection model and sample rate, so a configured stable-audio-open entry could only ever fail at load. It has its own loader now, keeps the stereo field (folding it to mono throws away the point of the model) and clips before the int16 cast, because a float above 1.0 wraps to a loud negative sample and is heard as a click. ACE-Step gets requirements-acestep.txt and an explicit refusal naming what it needs, rather than being loaded as a MusicGen and failing with a shape error three minutes later. It ships its own package pinning a stack this venv cannot hold, so it needs an isolated-venv worker like CrisperWhisper and Canary; that worker does not exist yet and the refusal says so. A GGUF component that fails to load now fails the load. It was skipped with a log line — and a skipped component is simply absent from the dict, so the base pipeline loaded its own full-precision copy: the operator asked for a 14 GB quantized transformer and got a 38 GB bf16 one, while the VRAM estimate still said 14 GB and the planner therefore freed about a third of what the load needed. A silent downgrade to something that does not fit is worse than a refusal. `civitai:<modelVersionId>` references, because a large part of the community's fine-tunes are published only there, as one ComfyUI-style checkpoint — exactly the shape build_single_file_pipeline already handles. The only thing missing was a way to fetch one. The VERSION id, not the model id: a Civitai "model" is a family, and an entry naming the family would silently change underneath the operator on a new release. Content type and size are both checked, because an unauthenticated download answers with a web page at HTTP 200 and a naive fetch saves it under a .safetensors name — which surfaces hours later as an unreadable tensor header. LatentSync and MuseTalk are first-class presenter engines, ahead of wav2lip for a clip: wav2lip works at 96x96 and upscales, which is where the soft mouth on a 1080p reel comes from. Both are video-driven, so sadtalker still leads for a still portrait — it is the one that invents head motion. Recognised from a shim on PATH or a managed install whose weights are already present, never "could be downloaded": a compose job is the wrong place to fetch 5 GB. And the loop stall. huggingface_hub.scan_cache_dir is not a listing: it stats every blob of every revision of every repo, measured at 27.3 SECONDS on this machine's 899 GB cache. It was called from a per-model VRAM estimate, a GGUF path lookup, and app.internal_engine_state — which is POLLED — all synchronously on the event loop. The watchdog's verdict was 6631 stalls over 0.5s with a worst of 13.0s, during every one of which no request of any kind was served. Fixed with both halves, because either alone is insufficient: the scan is memoised for 30s (short, so a model downloaded while the server runs still becomes visible) and the engine-state poll runs it through asyncio.to_thread, since memoising alone still puts the first 27-second call on the loop. Also: /v1/models reads `modes` from a video entry's long-standing `video_subtypes` as well as the newer spellings. Suite: 15 failed, 5194 passed — the same 15 as the baseline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
normalise a language before transcribing, stop inflating weights that are already quantized, and never answer 200 with nothing (v0.5.9) Five bugs and five features from a client report. The bugs share a shape: the server held a belief about a model that nothing checked against the model. A1. A TTS voice is chosen for an accent, so the language a compose render carries is "en-us" / "pt-br". Passed to transformers' Whisper -- which accepts only the English NAME -- it raised "Unsupported language: en-us" and 500'd every captioned scene of every render. Reproduced verbatim against the running server before the fix. New codai/api/lang_codes.py normalises once, per backend: the NAME for the transformers pipeline, the ISO code for whisper.cpp, Canary and faster-whisper, and None (auto-detect) for a tag that maps to nothing -- refusing to transcribe because the caller named a dialect is never the better outcome. Language VALIDATION now compares by meaning, so a model declaring "english" no longer refuses "en-gb", and compose's model picker stops disqualifying the only model that can time words. A3. FLUX.2 (dev/klein/pro) is a unified generate-and-edit model whose name carries neither "edit" nor "kontext", so it fell through to the generation branch and advertised plain img2img -- telling clients to show a `strength` slider for a pipeline that has no such argument. Confirmed live: ['image_generation','image_to_image','inpainting']. diffusers 0.37 agrees it is an instruction edit (Flux2Pipeline takes `image`, not `strength`), which is what the runtime already decided on; only the advertised list disagreed. A4. The estimator normalises a disk size to the load precision. A GGUF was special-cased to 1.0 because its bytes ARE its footprint; an HF checkpoint published pre-quantized is the same case by another route and was not. Z-Image-Turbo-unsloth-bnb-4bit: 16.3 GB of 4-bit weights, 1.70x factor, estimated 30.03 GB against a 24 GB card. Nothing can free 30 of 24, so every still ran partly on the CPU -- ~72 s for one image from a few-step turbo model. A5. Qwen3.5-9B answered content "" with HTTP 200 for four weeks. All three rungs of the generation ladder raised ("Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)") and the last one assigned generated_text = "" and returned normally. Now raises. The Triton error is a quantized model whose weights were CPU-offloaded: such a load succeeds and then fails every request, so it is refused at load time and a quantized model tries full-GPU whatever the estimate says. That also breaks a loop -- measured_vram_gb = GPU + offloaded + reserve means one offloaded load writes a number bigger than the card, and the next load reads it and offloads again. B1. A scene can carry its clip's own sound: visual.keep_audio, scene.audio_mix (voice|clip|both) and clip_volume. "clip" makes the scene as long as the clip and skips the TTS call; "both" ducks the clip under the voice. Folded into the narration track, so the music bed ducks under it like a voice. B2. Two characters talking: scene.speakers, each line in its own voice, each character lip-synced in their own column (or the frame cutting to whoever speaks). One sync run per CHARACTER, not per line -- each gets a track holding only their own lines, so the mouth moves on their words and stays closed while the other talks. Captions never mix two speakers and carry the attribution into the .srt/.vtt. B3. characters: ["alice","bob"] on images and video. The gap was never the field: IP-Adapter conditions on the MEAN of its references, so a second character's pictures produced one person who looked like neither. Reference sets are kept apart and handed to a model that takes several (FLUX.2, Qwen-Image-Edit); a model that cannot do it says so instead of blending. B4. POST /v1/moderation -- nsfw, faces, age, face_match over images or sampled video frames. It fails CLOSED: a missing model is a 503 naming it, not a result with the check quietly absent, because a caller cannot tell a missing check from a passed one. minor_detected is nullable (explicit content with no face cannot be cleared on age) and keys on the LOW end of an age interval, not a point estimate from a model with a multi-year error. C5. Per-model defaults (steps, guidance, sampler) that fill in only what the caller omitted, published in /v1/models alongside modes and licence -- so a client stops keeping a table keyed on the model NAME and an 8-step model stops being run for 30. A sampler switcher that refuses to cross the flow-matching boundary (a FLUX-family pipeline handed a DDIM produces noise, not a worse picture). Stable Audio gets its own pipeline: it was mapped to AudioLDM2, so a configured entry could only fail at load. Two of my own regressions, caught by the suite's guard tests and fixed here: /v1/moderation was unmapped in the remote gateway, and the moderation availability probe LOADED each model -- which turned the /v1/models listing into a 350 MB download on the event loop. Suite: 15 failed, 5146 passed -- the same 15 as the baseline. Co-Authored-By:
-
- 26 Oct, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
v0.5.6 registered operator fonts by dropping a snippet in $XDG_CONFIG_HOME/fontconfig/conf.d. On a host install that is right. In the container it is not: the launcher points XDG_CONFIG_HOME at the config VOLUME ROOT, which an unprivileged process cannot create a directory in, so the first restart after shipping it said [fonts] could not register ['/home/ubuntu/.coderai/fonts']: [Errno 13] Permission denied: '/config/fontconfig' and captions kept using the substitute. Two faults, not one: that path is not writable, and it is also inside the container's own filesystem, which a --rm container discards at every restart -- so even where it worked it had to be rewritten on each boot and could never be relied upon. The snippet is still tried first, because fontconfig reads it unprompted and so it survives into a grandchild. When it cannot be written, coderai now writes its own fonts.conf into the config dir one level down -- writable, and bind-mounted, so it persists -- and exports FONTCONFIG_FILE, which every child this process starts inherits. That file <include>s /etc/fonts/fonts.conf, so replacing the main configuration costs nothing: the system families still resolve. Verified in the container under the real environment, with the check that actually matters -- fc-match in a CHILD process, which is the lookup ffmpeg/libass performs: child fc-match 'Montserrat:style=Bold' -> Montserrat fc-match 'DejaVu Sans' -> DejaVu Sans (44 faces) The bug got through because it was verified with `docker exec`, which inherits a different environment than the supervised server process. A font this process can find is not necessarily a font the renderer can use, and an exec is not the deployment. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A whisper-server model could not be used by the id /v1/models advertises for it. Asking for it reported the weights as a missing HuggingFace repository: 500 Transcription error: whisper-large-v3-turbo-q8_0.bin is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' which reads like a broken download and is nothing of the kind. One whisper-server is described by TWO models.json rows: a RUNNER, carrying model_path + server_path + port, registered under its id (whisper0..whisper3 here); and a MODEL row naming the weights, which registration skips on purpose as "the model config, not a runner". But /v1/models takes its id from the model row -- so the published name was the one name that matched nothing: not a runner key, not an alias. The lookup missed, and the request fell through to the generic audio path, which can only interpret a string it does not recognise as a hub repo id. Now every name that identifies the same runner resolves to it: its own id, an alias (round-robin as before), and the weights by full path, basename, or basename without extension. Where several runners share one file, naming the file prefers an idle one, so it balances the way an alias does instead of always landing on whichever registered first. This is the third defect in one week with the same shape -- a name in models.json not matching the name the runtime registered. v0.5.4 read a backend from the runtime map instead of the file that declares it; v0.5.5 looked for a binary in a directory that does not exist; this one publishes an id it cannot resolve. The pattern is worth an audit of its own. Also pins a rule on the VRAM estimator that it already followed in most places and now cannot lose: an estimate of zero means NOTHING WAS MEASURED, never "measured as needing nothing". A stored zero was already read as absent; v0.5.6 stopped a configured zero from claiming authority over a real measurement; the two tests added here also cover a zero arriving with force_vram_update, which is the one flag that makes a measurement beat the operator's own number. Co-Authored-By:
-