- 27 Oct, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authoredA request for LTX-2.3-10Eros loaded LTX-2.3-PinkCherry: a different 15.9 GB transformer, silently, and from an entry that was DISABLED. self.config is keyed <kind>:<model_name> where model_name is the weights path, so entries sharing a path clobber one another and only the last survives. That is not a corner case -- it is how one base checkpoint is served under several configurations: lisa / lisa-32k differ only in context length, 10Eros / PinkCherry only in which GGUF is their transformer. PinkCherry comes second in models.json, so it won, parked or not. The VRAM estimator already disambiguated this way, by the id the caller asked for, after lisa-32k was once sized at lisa 51.97 GB. The config used to LOAD the model did not. Now both do: * every type setter stores a config under its ALIAS as well as its path, so a sibling is reachable by the only name that distinguishes it. The path key keeps its old meaning (last writer wins) for callers that know only a path. * _config_for_model_key takes the requested id and prefers the entry whose alias matches. Applied to all eight setters rather than video alone -- the collision is the same for audio, image and embedding, and fixing one type is how this survived being found twice already. Also: build_gguf_pipeline_components passed `model_name` as the diffusers repo supplying the component configs. Once siblings became reachable by alias that can BE the alias -- "LTX-2.3-10Eros", not a repo -- and from_single_file rejects it. The repo is whatever the entry names in `path`, every time. Verified live: LTX-2.3-10Eros now loads its own transformer and the pipeline assembles, 8/8 components. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZnmWyKjWZerwsb27jZ
-
Stefy Lanza (nextime / spora ) authored
The LTX-2.3 GGUFs failed with "Cannot copy out of meta tensor; no data!", and the obvious reading -- parameters the checkpoint lacks -- was wrong. diffusers 0.38 converts an LTX-2 checkpoint prompt-adaln weights to prompt_adaln_single.* and audio_prompt_adaln_single.*, while LTX2VideoTransformer3DModel expects prompt_adaln.* and audio_prompt_adaln.*. Twelve names, one-to-one, the same tensors. They were present all along, under a name nobody reconciled, so they never materialised and stayed on meta. Measured identically on LTX-2.3-10Eros v1.5 and PinkCherry v1.8: 4444 GGUF tensors, 4186 expected parameters, and after the rename 0 missing and 0 unused on both. The tempting fix was to initialise those twelve. That would have invented weights to paper over a rename -- precisely what the surrounding refusal exists to stop -- so it is NOT what this does. A key is moved only when the class does not want its current name AND does want the renamed one AND nothing already occupies the target, which makes it a rename of a tensor the checkpoint has and never the creation of one it lacks. A test pins that: a genuinely absent parameter must stay absent and the load must still fail. The converter is a third-party global, so it is wrapped for the duration of one from_single_file call and restored in a finally. Version-robust: 9 pass against the image diffusers 0.38, including the real checkpoint; against 0.37 the class has no prompt_adaln modules at all, so the repair is correctly a no-op and that test skips rather than asserting a fact about someone else is version. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
"Cannot copy out of meta tensor; no data!" means some parameters were never materialised, and it names none of them -- which sends the reader hunting for a corrupt file. Measured on LTX-2.3-10Eros and -PinkCherry: diffusers already ships convert_ltx2_transformer_to_diffusers and it works almost completely, mapping their 4444 GGUF tensors onto exactly the 4186 the class expects. Precisely 12 remain unprovided, all in one conditioning branch, identically in both fine-tunes. So the checkpoints are fine and the converter is fine; those fine-tunes were exported without that branch, and the 12 stay on the meta device. The refusal added earlier now carries that analysis: how many tensors mapped, how many remain, and which submodules they belong to. Best effort throughout -- any failure to analyse returns nothing and the original error stands. It deliberately does NOT offer to fill them in. Supplying parameters a checkpoint does not contain is a substitution, and the surrounding refusal exists precisely because a silent substitution (here: the 38 GB bf16 transformer, on a 24 GB card) is worse than a failure. A test pins that: the hint must not grow a flag that papers over it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/models re-reads models.json on every call, but the loader's alias map is built ONCE at startup. So an entry added while the server runs is advertised immediately and cannot be loaded at all: the alias resolves to nothing, the raw alias reaches diffusers as a repo id, and the caller gets Repository Not Found for url https://huggingface.co/api/models/<alias> -- a 401 against HuggingFace for a model sitting on the local disk. Measured today registering Wan2.2-TI2V-5B: weights cached, `path` correct, and the request failed that way; the SAME model requested by its path loaded immediately, which is what pinned the cause to the alias map rather than the entry or the weights. The pre-existing Z-Image alias, updated rather than added, worked throughout -- that contrast is the whole diagnosis. This is the fifth instance of one pattern in this codebase: a name in models.json not matching the name the runtime registered. v0.5.4 read a backend from the wrong map, v0.5.5 looked for a binary where it does not live, v0.5.7 published an id that resolved to nothing, the LTX pair shared one path between two entries, and now the listing advertises what the loader cannot resolve. A cross-cutting identity-resolution audit is still the right fix; this is the narrow one. An alias the map has not seen is now looked up in the config itself and cached. A parked entry is deliberately NOT resolved -- parking is how an entry waits on a blocker, and resolving it would hand clients a model that answers 500. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Found by generating a real image against the live server: a request with "response_format": "b64_json" came back {url, path} and no bytes. The request model documents OpenAI's values -- url and b64_json -- and every OpenAI SDK sends the latter. save_image_response compared against this codebase's own older internal spelling, "base64", so with --file-path configured the documented behaviour ("if the client explicitly requested base64, include it") could never fire for a standards-compliant client. It got a URL, no bytes, and nothing in the response to say why. Nine call sites were affected: generations, edits, inpaint, upscale, depth, segment and the rest. Both spellings are accepted now. The url path is unchanged -- the point of --file-path is not to ship megabytes inline. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
LatentSync (Apache-2.0) and MuseTalk (MIT) are now first-class presenter engines and come before wav2lip for a clip, so the compose documentation says which engine engine:"auto" will pick and why: wav2lip works at 96x96 and upscales, which is where the soft mouth on a 1080p reel comes from, while these two run at 256-512 px natively. sadtalker still leads for a still portrait because it is the only one that invents head motion rather than replacing a mouth. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Found by running the endpoint against the live server. A request naming `face_match.character_profiles` came back 400: face_match was requested with no reference images which is the opposite of the truth -- the caller HAD supplied references. The cause was a bare `except Exception: pass` around the per-profile lookup, so a misspelt name, a profile with no saved images, and a characters directory the server resolves elsewhere all collapsed into that one message with nothing to distinguish them. Establishing which it was took a long sequence of probes against a running server: the field is declared (a bad type 422s with loc body.face_match.character_profiles), the profile exists under both candidate paths in the container, and resolve_character_profiles returns its four images when called in-process -- so the swallowed exception was the only thing standing between the symptom and the cause, and it is the defect. Now each profile is resolved separately and its failure is named. When none resolve the 400 lists them; when some do, the request proceeds (there ARE references to match against) and the unusable ones are logged, because a performer the caller believes is enrolled and is not would make a genuine face read as "matches no consenting performer" for the wrong reason. Verified live on the same server, all four checks against a real face: nsfw 0.0003, one face at det_score 0.879, age 38.0 with the interval 30.0-46.0 and minor_probability 0.0, face_match similarity 1.0 matched against itself, unmatched_faces 0, flagged false. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
9ae240b0 committed its tests and its documentation and none of its implementation: four files instead of twelve. The `git add` that should have staged the rest named a pathspec for requirements-acestep.txt, which did not exist, so git refused the whole invocation and I did not check its output before committing. The message described work that was not in the tree, and the suite at that commit could not have passed. This is that work: * video.py — LTX-2.x gets its own pipeline classes. This diffusers has LTX2Pipeline, LTX2ImageToVideoPipeline, a video AND an audio VAE, but only the LTX 0.9 family was mapped, so every LTX-2 entry in models.json sat parked: the name matched and the class did not. Split on the version in the name, since both families are just called "LTX" and a fine-tune keeps its base version (10Eros is LTX2.3, PinkCherry is LTX23). * hf_loading.py — `civitai:<modelVersionId>` references, and a GGUF component that fails to load now fails the load instead of being skipped. A skipped component is simply absent from the dict, so the base pipeline loaded its own full-precision copy: the operator asked for a 14 GB quantized transformer and got a 38 GB bf16 one, while the VRAM estimate still said 14 GB and the planner therefore freed about a third of what the load needed. * presenter.py — LatentSync and MuseTalk as first-class engines, ahead of wav2lip for a clip (wav2lip works at 96x96 and upscales, which is the soft mouth on a 1080p reel). Both are video-driven, so sadtalker still leads for a still portrait. Recognised from a shim on PATH or a managed install whose weights are already present — never "could be downloaded", because a compose job is the wrong place to fetch 5 GB. * cache/__init__.py + app.py — the loop stall. scan_cache_dir is not a listing: it stats every blob of every revision of every repo, measured at 27.3 SECONDS on this machine's 900 GB cache, and it was called from a per-model VRAM estimate, a GGUF path lookup and a POLLED endpoint, all on the event loop. 6631 stalls over 0.5s, worst 13.0s, serving nothing. Both halves were needed: memoised for 30s (short enough that a model downloaded while the server runs still becomes visible) and the poll moved to asyncio.to_thread, since memoising alone still puts the first 27-second call on the loop. * manager.py — /v1/models reads `modes` from a video entry's long-standing `video_subtypes` as well as the newer spellings. * requirements-acestep.txt — the file whose absence caused this. It is marked NOT YET INSTALLED, because it is: the install was refused as external code, nothing in it has been run, and the audio_gen loader refuses an ACE-Step model explicitly rather than mis-loading it as a MusicGen. Suite: 15 failed, 5194 passed — the same 15 as the baseline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
serve the LTX-2 family, fetch a Civitai checkpoint, and stop walking 900 GB of cache on the event loop Model installation work from the client's C list, plus one latency bug found on the way. LTX-2 is wired. This diffusers has LTX2Pipeline, LTX2ImageToVideoPipeline, a video AND an audio VAE — but video.py only mapped the LTX 0.9 family, so every LTX-2 entry in models.json sat parked: the name matched and the class did not. Split on the version in the name, since both families are just called "LTX" and a fine-tune keeps its base version (10Eros is LTX2.3, PinkCherry is LTX23). Stable Audio had its own entry in the type map pointing at AudioLDM2Pipeline — a different architecture with a different projection model and sample rate, so a configured stable-audio-open entry could only ever fail at load. It has its own loader now, keeps the stereo field (folding it to mono throws away the point of the model) and clips before the int16 cast, because a float above 1.0 wraps to a loud negative sample and is heard as a click. ACE-Step gets requirements-acestep.txt and an explicit refusal naming what it needs, rather than being loaded as a MusicGen and failing with a shape error three minutes later. It ships its own package pinning a stack this venv cannot hold, so it needs an isolated-venv worker like CrisperWhisper and Canary; that worker does not exist yet and the refusal says so. A GGUF component that fails to load now fails the load. It was skipped with a log line — and a skipped component is simply absent from the dict, so the base pipeline loaded its own full-precision copy: the operator asked for a 14 GB quantized transformer and got a 38 GB bf16 one, while the VRAM estimate still said 14 GB and the planner therefore freed about a third of what the load needed. A silent downgrade to something that does not fit is worse than a refusal. `civitai:<modelVersionId>` references, because a large part of the community's fine-tunes are published only there, as one ComfyUI-style checkpoint — exactly the shape build_single_file_pipeline already handles. The only thing missing was a way to fetch one. The VERSION id, not the model id: a Civitai "model" is a family, and an entry naming the family would silently change underneath the operator on a new release. Content type and size are both checked, because an unauthenticated download answers with a web page at HTTP 200 and a naive fetch saves it under a .safetensors name — which surfaces hours later as an unreadable tensor header. LatentSync and MuseTalk are first-class presenter engines, ahead of wav2lip for a clip: wav2lip works at 96x96 and upscales, which is where the soft mouth on a 1080p reel comes from. Both are video-driven, so sadtalker still leads for a still portrait — it is the one that invents head motion. Recognised from a shim on PATH or a managed install whose weights are already present, never "could be downloaded": a compose job is the wrong place to fetch 5 GB. And the loop stall. huggingface_hub.scan_cache_dir is not a listing: it stats every blob of every revision of every repo, measured at 27.3 SECONDS on this machine's 899 GB cache. It was called from a per-model VRAM estimate, a GGUF path lookup, and app.internal_engine_state — which is POLLED — all synchronously on the event loop. The watchdog's verdict was 6631 stalls over 0.5s with a worst of 13.0s, during every one of which no request of any kind was served. Fixed with both halves, because either alone is insufficient: the scan is memoised for 30s (short, so a model downloaded while the server runs still becomes visible) and the engine-state poll runs it through asyncio.to_thread, since memoising alone still puts the first 27-second call on the loop. Also: /v1/models reads `modes` from a video entry's long-standing `video_subtypes` as well as the newer spellings. Suite: 15 failed, 5194 passed — the same 15 as the baseline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
normalise a language before transcribing, stop inflating weights that are already quantized, and never answer 200 with nothing (v0.5.9) Five bugs and five features from a client report. The bugs share a shape: the server held a belief about a model that nothing checked against the model. A1. A TTS voice is chosen for an accent, so the language a compose render carries is "en-us" / "pt-br". Passed to transformers' Whisper -- which accepts only the English NAME -- it raised "Unsupported language: en-us" and 500'd every captioned scene of every render. Reproduced verbatim against the running server before the fix. New codai/api/lang_codes.py normalises once, per backend: the NAME for the transformers pipeline, the ISO code for whisper.cpp, Canary and faster-whisper, and None (auto-detect) for a tag that maps to nothing -- refusing to transcribe because the caller named a dialect is never the better outcome. Language VALIDATION now compares by meaning, so a model declaring "english" no longer refuses "en-gb", and compose's model picker stops disqualifying the only model that can time words. A3. FLUX.2 (dev/klein/pro) is a unified generate-and-edit model whose name carries neither "edit" nor "kontext", so it fell through to the generation branch and advertised plain img2img -- telling clients to show a `strength` slider for a pipeline that has no such argument. Confirmed live: ['image_generation','image_to_image','inpainting']. diffusers 0.37 agrees it is an instruction edit (Flux2Pipeline takes `image`, not `strength`), which is what the runtime already decided on; only the advertised list disagreed. A4. The estimator normalises a disk size to the load precision. A GGUF was special-cased to 1.0 because its bytes ARE its footprint; an HF checkpoint published pre-quantized is the same case by another route and was not. Z-Image-Turbo-unsloth-bnb-4bit: 16.3 GB of 4-bit weights, 1.70x factor, estimated 30.03 GB against a 24 GB card. Nothing can free 30 of 24, so every still ran partly on the CPU -- ~72 s for one image from a few-step turbo model. A5. Qwen3.5-9B answered content "" with HTTP 200 for four weeks. All three rungs of the generation ladder raised ("Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)") and the last one assigned generated_text = "" and returned normally. Now raises. The Triton error is a quantized model whose weights were CPU-offloaded: such a load succeeds and then fails every request, so it is refused at load time and a quantized model tries full-GPU whatever the estimate says. That also breaks a loop -- measured_vram_gb = GPU + offloaded + reserve means one offloaded load writes a number bigger than the card, and the next load reads it and offloads again. B1. A scene can carry its clip's own sound: visual.keep_audio, scene.audio_mix (voice|clip|both) and clip_volume. "clip" makes the scene as long as the clip and skips the TTS call; "both" ducks the clip under the voice. Folded into the narration track, so the music bed ducks under it like a voice. B2. Two characters talking: scene.speakers, each line in its own voice, each character lip-synced in their own column (or the frame cutting to whoever speaks). One sync run per CHARACTER, not per line -- each gets a track holding only their own lines, so the mouth moves on their words and stays closed while the other talks. Captions never mix two speakers and carry the attribution into the .srt/.vtt. B3. characters: ["alice","bob"] on images and video. The gap was never the field: IP-Adapter conditions on the MEAN of its references, so a second character's pictures produced one person who looked like neither. Reference sets are kept apart and handed to a model that takes several (FLUX.2, Qwen-Image-Edit); a model that cannot do it says so instead of blending. B4. POST /v1/moderation -- nsfw, faces, age, face_match over images or sampled video frames. It fails CLOSED: a missing model is a 503 naming it, not a result with the check quietly absent, because a caller cannot tell a missing check from a passed one. minor_detected is nullable (explicit content with no face cannot be cleared on age) and keys on the LOW end of an age interval, not a point estimate from a model with a multi-year error. C5. Per-model defaults (steps, guidance, sampler) that fill in only what the caller omitted, published in /v1/models alongside modes and licence -- so a client stops keeping a table keyed on the model NAME and an 8-step model stops being run for 30. A sampler switcher that refuses to cross the flow-matching boundary (a FLUX-family pipeline handed a DDIM produces noise, not a worse picture). Stable Audio gets its own pipeline: it was mapped to AudioLDM2, so a configured entry could only fail at load. Two of my own regressions, caught by the suite's guard tests and fixed here: /v1/moderation was unmapped in the remote gateway, and the moderation availability probe LOADED each model -- which turned the /v1/models listing into a 350 MB download on the event loop. Suite: 15 failed, 5146 passed -- the same 15 as the baseline. Co-Authored-By:
-
- 26 Oct, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored
v0.5.6 registered operator fonts by dropping a snippet in $XDG_CONFIG_HOME/fontconfig/conf.d. On a host install that is right. In the container it is not: the launcher points XDG_CONFIG_HOME at the config VOLUME ROOT, which an unprivileged process cannot create a directory in, so the first restart after shipping it said [fonts] could not register ['/home/ubuntu/.coderai/fonts']: [Errno 13] Permission denied: '/config/fontconfig' and captions kept using the substitute. Two faults, not one: that path is not writable, and it is also inside the container's own filesystem, which a --rm container discards at every restart -- so even where it worked it had to be rewritten on each boot and could never be relied upon. The snippet is still tried first, because fontconfig reads it unprompted and so it survives into a grandchild. When it cannot be written, coderai now writes its own fonts.conf into the config dir one level down -- writable, and bind-mounted, so it persists -- and exports FONTCONFIG_FILE, which every child this process starts inherits. That file <include>s /etc/fonts/fonts.conf, so replacing the main configuration costs nothing: the system families still resolve. Verified in the container under the real environment, with the check that actually matters -- fc-match in a CHILD process, which is the lookup ffmpeg/libass performs: child fc-match 'Montserrat:style=Bold' -> Montserrat fc-match 'DejaVu Sans' -> DejaVu Sans (44 faces) The bug got through because it was verified with `docker exec`, which inherits a different environment than the supervised server process. A font this process can find is not necessarily a font the renderer can use, and an exec is not the deployment. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A whisper-server model could not be used by the id /v1/models advertises for it. Asking for it reported the weights as a missing HuggingFace repository: 500 Transcription error: whisper-large-v3-turbo-q8_0.bin is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' which reads like a broken download and is nothing of the kind. One whisper-server is described by TWO models.json rows: a RUNNER, carrying model_path + server_path + port, registered under its id (whisper0..whisper3 here); and a MODEL row naming the weights, which registration skips on purpose as "the model config, not a runner". But /v1/models takes its id from the model row -- so the published name was the one name that matched nothing: not a runner key, not an alias. The lookup missed, and the request fell through to the generic audio path, which can only interpret a string it does not recognise as a hub repo id. Now every name that identifies the same runner resolves to it: its own id, an alias (round-robin as before), and the weights by full path, basename, or basename without extension. Where several runners share one file, naming the file prefers an idle one, so it balances the way an alias does instead of always landing on whichever registered first. This is the third defect in one week with the same shape -- a name in models.json not matching the name the runtime registered. v0.5.4 read a backend from the runtime map instead of the file that declares it; v0.5.5 looked for a binary in a directory that does not exist; this one publishes an id it cannot resolve. The pattern is worth an audit of its own. Also pins a rule on the VRAM estimator that it already followed in most places and now cannot lose: an estimate of zero means NOTHING WAS MEASURED, never "measured as needing nothing". A stored zero was already read as absent; v0.5.6 stopped a configured zero from claiming authority over a real measurement; the two tests added here also cover a zero arriving with force_vram_update, which is the one flag that makes a measurement beat the operator's own number. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
fit the caption to the frame, install the font that draws it, and stop reading an unset field as a free GPU (v0.5.6) Four defects behind one report: vertical renders came back with caption lines running off both edges, in the wrong typeface, and speech-to-text answered 500/503 while a text model held the card. THE FONT WAS NEVER IN THE IMAGE. Dockerfile.oci and Dockerfile.update install fonts-montserrat; Dockerfile.oci-venv -- the one build_oci_image.sh actually builds -- installed no fonts at all, so no image built the normal way has ever had the family the compose API documents and defaults to. libass does not fail on a missing family, it substitutes: DejaVu is considerably wider, which is how a missing package became a layout bug with nothing but a log line to say so. All three Dockerfiles now install the fonts AND fontconfig (Dockerfile.oci had the fonts but not fc-match, so it could not even tell whether a family existed). And because an image rebuild is a poor way to add a brand font, fonts can now be supplied per installation: files dropped in <config>/fonts -- or wherever CODERAI_FONT_DIR points -- are registered at startup by writing a fontconfig snippet into the user config. The snippet matters more than it looks: captions are burned by libass inside an ffmpeg CHILD process, which does its own fontconfig lookup, so a font this process can find is not necessarily a font the renderer can use. The config volume outlives both a restart and a new image, which is the point. CAPTION SIZE CAME FROM THE FRAME HEIGHT. On a vertical video the height is the LONG side: at 1080x1920 the font was sized 0.052 * 1920 ~ 100px to be drawn across a 1080px frame. It is now sized from the short side, which leaves 16:9 byte-identical (there the short side IS the height) and gives 9:16 the same text-to-width ratio as 16:9. Line breaking upstream works on a word budget and cannot know how wide the words are, and WrapStyle is 2 -- no automatic wrapping -- so a line that still overflows was simply drawn past the edge. Each event's widest line is now measured against the real font file, and one that does not fit gets its own smaller size through an inline \fs override: the long line shrinks, not the whole video. Measured with the actual face rather than estimated per character, because the families involved differ in width by exactly the margin that decides the question. Clients scaling font_scale down by hand for 9:16 and 4:5 can stop. AN UNSET FIELD READ AS A FREE GPU. used_vram_gb=0 was honoured as authoritative -- yet zero is this estimator's own "could not determine" sentinel, and it is what the admin UI stores for a field the operator never filled in. A 27B Q4_K_P with used_vram_gb=0 and measured_vram_gb=18.0 therefore estimated 0.00 GB against 8852 MiB of free VRAM. Nothing was evicted, every layer was aimed at the GPU, the load failed and retried down a ladder -- all, 52, 39, 26 layers, re-reading the weights at each rung -- for 11m50s, and settled with 39 of 66 layers on the CPU. The request then generated for another 19 minutes, 31m04s in all, holding GPU ownership the whole time: every other model queued behind it, which is what made speech-to-text answer 500/503. The swap gate was working correctly; it swapped the moment the owner finished. A real measurement is better information than an unfilled field, so it now wins. An operator who means "no weights on the GPU" has n_gpu_layers=0, which is handled separately and still reserves nothing. Worth recording for the next time this looks like a quantization problem: it was not. Those weights are 4-bit on disk and the KV cache was already q4_0/q4_0 with flash attention -- about 2.2 GB for n_ctx=32000. Both were at their most aggressive setting while the model sat on the CPU. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The DINOv2 GGUF embedder answered 500 for about twelve hours -- 6,176 image embedding requests -- for two independent reasons, and the capability can be dropped again by an ordinary rebuild, so both the lookup and the packaging are fixed here. The resolver looked only under /opt/coderai/bin, a directory that does not exist in the image, and reported its absence as a generic load failure. It now tries DINOV2_EMBED_BIN, then PATH, then the places the binary is actually installed, and when it finds nothing it names every one of them plus the build script -- because there is no fallback: llama.cpp cannot load a ViT-only arch, so this binary is the only runtime this model has. The binary was also never listed in the image build, having been compiled by hand once; the next rebuild silently shipped without it. It is listed now, along with the private shared libraries it cannot start without. Those libraries are the larger half of the problem. The image has no system OpenCV -- only the venv's cv2 wheel, a different version with mangled sonames -- so the closure travels beside the binary in dinov2-embed-libs, reached through its own RPATH rather than LD_LIBRARY_PATH: OpenCV placed in local-libs would be offered to every process in the image, cv2 included, while a private directory on one binary's RPATH cannot shadow anything. Four things make that work, each learned by watching it fail: - the whole dependency tail, not just libopencv_*. Debian's imgcodecs links GDAL, GDCM and OpenEXR, which a JPEG decoder never calls and the loader demands anyway. Copying three libraries left six still "not found". - cp -L. Most of these paths are symlinks, and copying the link gives a 20-byte file -- which also made a first size estimate off by 7x. - RPATH on every library, not only on the binary. patchelf writes DT_RUNPATH, which unlike DT_RPATH is not inherited by a dependency's dependencies, so the binary found imgcodecs and imgcodecs found nothing. - everything above the base ABI, not only what the image lacks. libtiff.so.6 exists there, so it was skipped, and the newer GDAL then wanted LIBTIFF_4.6.1 from it. Mixing distributions works only if the whole tier comes from one of them. What stays behind is what any Linux has: libc, libm, libstdc++, libgcc, libgomp, pthread, dl, rt, the loader, and libvulkan, which must match the GPU driver. The tarball build had the same two gaps as the image build and is fixed with it: it copied a binary without its -libs sibling, shipping a program that cannot start, and a --local-binary-dir sweep would have flattened those .so files into bin/ as programs, away from the only place the RPATH looks. Verified against the HF vision backend on the same images: cosine 0.990-0.995 per image, with image-to-image geometry agreeing to 0.002-0.017 -- the quantized binary and the reference implementation describe the same space, so an index built by one stays valid under the other. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Both of these were found by running the previous commit rather than trusting it, which is the only reason they are not still in. The audio backend was read from the manager's per-model config map. That map holds RUNTIME kwargs, and a whisper-server model has no row in it at all -- it lives in `whisper_servers` -- so the lookup answered "no backend" for exactly the models whose backend decides the question, and the pick that was supposed to avoid them chose `whisper2`. The entry is now read from models.json, where `backend` is actually written, matched on id, alias, path or basename because an audio model is referred to by any of them and the same alias appears twice in a real catalogue. Verified against this install's sixteen audio models: every one resolves, and the pick is now the model measured to return fourteen word timings rather than the one measured to return none. The new warning is what caught it: the render reported "'whisper2' returned no word timings" instead of quietly estimating. That is the whole point of it. And `enabled: false` was hidden behind `all_engines`, which is the wrong seam. The node-wide list the system worker serves -- the one the front hands clients -- MUST pass all_engines=True, because that worker has no engine of its own and would otherwise list nothing. So the filter was off on the only path that mattered, and /v1/models offered LTX-2.5, Krea-2-Turbo and Krea-2-Edit: three models configured ahead of a diffusers version that does not exist yet, each of which answers 500. `include_disabled` is now a separate question, asked only by the admin config view, because that is where a parked model gets re-enabled. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Follow-up to v0.5.2, from measuring the fix instead of trusting it. Asking the running install for word timings on 4.3 seconds of Italian it had just synthesised showed two more faults in the same chain, both of which answer 200 with the timings missing. A CTC pipeline cannot take `generate_kwargs=None`: it iterates it and raises `TypeError: 'NoneType' object is not iterable`. The backend passed exactly that whenever a model had no generate kwargs, which is every wav2vec2/MMS checkpoint -- and the retry below it then turned the failure into a text-only success, so the only trace was one line in the log. `wav2vec2-it` returned 0 words before this and the same 14 the pipeline always produced after. The kwarg is now omitted when empty rather than passed as None. And the pick has to consider the language, not just the backend: a model that declares `languages` REFUSES the rest with a 400, so an install with `wav2vec2-it` and `wav2vec2-en` has two models that each report word timings and each reject the other's language. Picking one for the wrong language would have replaced one silent fallback with another. Measured, on sixteen configured audio models: ten cannot time words at all. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two faults seen on a long render, plus the image-side work from the same reports. Caption timings never worked. The render asks for word timestamps on the narration clip it just wrote, so it passes a NAME where the HTTP endpoint passes an upload -- and the transcription core read `.filename` off it unconditionally, raising `'str' object has no attribute 'filename'` on every scene of every render. The caller caught it and fell back to estimating from character counts, so captions drifted against the voice and nothing ever failed. `_upload_name` now takes a path, an upload or nothing; only the extension is read from it, so a name was always enough. Fixing that alone would have changed nothing visible. The model the renderer picked by default was chosen by NAME -- "whisper" first, then canary -- and both are the two backends that cannot time words: the whisper.cpp runner returns text with no word array, and the NeMo worker ignores the flag. The pick now asks which backends report them (the HF pipelines, vosk, faster-whisper), and a model that answers with none of them is named in the job's warnings instead of silently downgrading the captions. Polling a render cost a render. `GET /v1/video/compose/{id}` sits under the `/v1/video/` prefix, which allows 10 requests a minute per IP, so a client polling a four-minute render from the address it submits from spent the whole budget on the polls and got 429 on the work. Read-only job status -- compose and talking-head -- is now exempt from the limit, by prefix and for GET only, so the sibling POST .../cancel stays limited. It is NOT exempt from authentication: the document names the job's output, unlike the progress endpoints it sits beside. Also here, from the image-edit reports: a uniformly black or white result is refused with the dtype named rather than answered 200, `--no-ram` no longer coerces a configured bf16 to fp16 (same two bytes, a quarter of the range, NaN latents, black picture), single-file and GGUF checkpoints load through the configured loader instead of falling back to a different one, per-model LoRAs are an identity rather than a hint, and a repository may run its own pipeline code only from an allow-list that starts empty. And: the measurements stay, the customer names go -- shipped images are public. Co-Authored-By:
-
- 25 Oct, 2026 14 commits
-
-
Stefy Lanza (nextime / spora ) authored
Both found by reading the output of a real page through a real pod, which is the only place either was visible. WHICH CONFIGURATION. A pod answers with the engine it ran, because the configuration ids in ocr.engine_configs are an orchestrator-side naming layer it has never heard of. So `engine=surya-layout` came back `engine: surya`, same as plain surya -- and telling two configurations of one engine apart in the answer is most of the point of having two ids. The local path already reported the id; now both do, and a plain engine name still reports itself. A CONFIDENCE NOBODY MEASURED. surya's table recogniser does not score cells in the mode we call it in, so every cell carried `conf: 0.0` -- a placeholder wearing a measurement's clothes, which is the confusion `confidence_measured` already exists to prevent for pages. The key is now present only when surya reported a number, and the table says `confidence_measured` so a caller gating on cell confidence can tell whether there is anything to gate on. Presence decides it rather than truthiness, because a real 0.0 IS a measurement. Table markup likewise: `html` is absent rather than empty, so "this engine returns no HTML" (surya, which returns cells) is distinguishable from "this table's HTML is empty". Two of the tests written for this asserted against a fixed byte window after a `def`, and failed on the correct edit that grew the function -- the same brittleness fixed in test_image_edit_uses_an_image_pipeline an hour earlier. They read the function by AST now. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The layout pass read its per-page option with `_opt("layout", True)`. The accessor is `_page_override`. A pod answered 500 {"detail": "surya ocr failed: name '_opt' is not defined"} after the image was built, pushed, signed and booted, because nothing executes these modules until a page reaches a pod: `codai/ocr/workers/*` run OUTSIDE the package -- the surya worker in its own venv, forbidden from importing codai -- so no test had ever run a line of them. The test I wrote for the layout pass asserted the SOURCE STRING of that call and proved only that the text was there. So the fix comes with a guard that would have caught it: every `_`-prefixed name a worker module loads must be bound somewhere in that module. Those are the module's own helpers, so an unbound one is a typo and not a global from elsewhere. Pure ast + builtins -- no linter, nothing added to requirements -- and it is checked against the bug itself, because a checker that passes everything is worse than none. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`pod_ocr_engine` rewrote EVERY whole-request OCR forward to docTR. The premise was that a pod image cannot carry a VLM's model server, which was true of `coderai-vllm` and never true of `coderai-vllm-ocr`: that image carries surya, the rasteriser and its own vLLM -- it IS the pod-side pipeline -- and it carries no docTR at all, which is why its pods launch with CODERAI_OCR_FALLBACK_ENGINES=off. So the rewrite asked a pipeline pod for the one engine it cannot possibly have, and the pod answered "OCR engine 'doctr' is not enabled". The question cannot be asked without the image, and the gateway was not passing one. Found by sending a page through a pod, not by reading the code: the id resolved, the pool provisioned, the pod booted and went ready, and the request still came back 400 naming an engine nobody had asked for. A named configuration (ocr.engine_configs) also resolves to its base engine here. The ids are an orchestrator-side naming layer and a pod is a pre-configured instance of one -- its env already carries the overrides -- so telling it the id names an engine it has never heard of. Omitting the image keeps the old answer, so callers that cannot know it are unchanged. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A model id is a configuration, not a set of weights. Image models have worked that way for a long time -- one checkpoint, several published ids, one copy in VRAM -- and five things that should have followed from it did not. PER-MODEL LoRA STACKS. A diffusion pipeline could carry an acceleration LoRA and per-request LoRAs, with nothing in between: no way to say "this id is that base plus this style LoRA at 0.8". `loras` on a model entry says it, in the shape text models already use. `lora_path` is honoured too -- it had been in the config and the admin page for years while nothing on the diffusers image path read it. And `set_adapters` replaces the ACTIVE SET, so the image path's request-LoRA call was switching the pipeline's own adapters off: a 4-step distilled preset running un-distilled, and an id published as a fine-tune serving its base checkpoint. 200 OK, a plausible picture, nothing in the logs. The video path had always re-included them. A per-model LoRA that will not load now FAILS THE LOAD. The acceleration path logs and continues because the cost there is speed; here it is identity. SINGLE-FILE CHECKPOINTS. `from_single_file` was wired up only for GGUF components, so the one .safetensors Civitai ships for every community fine-tune could not be opened at all. A model path may now name a file, or `repo:file.safetensors` -- community repos hold every version side by side, so the repo id alone does not say which to serve. A SAMPLER IS A CLASS PLUS ITS OPTIONS. "DPM++ 2M SDE Karras" is DPMSolverMultistepScheduler and two settings; naming the class alone gave plain DPM++ 2M, a different sampler, silently. `scheduler_config` carries them. RASTERISATION, PER REQUEST. A scan is normally handed to the model as the scanner wrote it -- 0.1 ms and 147 KB against a render plus ~1.8 MB of PNG -- but only between the target DPI and `page_image_max_dpi_factor` times it, and only install-wide. `native_dpi` removes the ceiling for one request; the floor stays, because a page passed through BELOW the target is a coarser page, which is what cost surya-ocr-2 its Italian accents once already. An uploaded image now carries its own bytes too: it was decoded and re-encoded to arrive at the picture the caller already had. SEVERAL IDS FOR ONE OCR ENGINE. `ocr.engine_configs` gives OCR what image models had: named configurations of one engine, each its own id, its own catalogue row, its own pool. An override naming a setting that does not exist is refused rather than quietly serving the base settings under another name. Which forced the last one into the open. `/v1/ocr/engines` advertised `surya: {regions: true, tables: true}` while the surya worker built every page with `"regions": [], "tables": []` hardcoded. surya ships a layout predictor and a table recogniser; they were never run. A client that chose surya BECAUSE the catalogue said it returned regions got empty arrays and no error. Both halves are now true by construction: the row is read from `surya_layout` / `surya_table_rec`, and those load the predictors. Regions come back with surya's own reading order (LayoutBox.position, explicit as `order`, -1 where an engine states none) and tables as cells with row/column ids. Loading is configured; spending is per request. `layout=true` chooses whether to pay for the pass on an engine that HOLDS the predictor -- it cannot make one acquire a model mid-page, which is how a footprint grows past what the VRAM accounting reserved. Asked for where none is loaded, it is refused with the configuration to use instead; answering 200 with empty regions would be the same lie wearing a request field. Pods differ from their siblings too: a pool whose entry names an `ocr_engine_config` boots pods with that configuration's settings, and a named config overriding `<engine>_model_id` reaches its own pods rather than sharing its sibling's. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
set_image_model keys config by path: self.config[f"image:{model_name}"] = config or {} so a second image_models entry for the same checkpoint overwrote the first and said nothing. Two configs of one checkpoint is not an odd thing to want -- the same base served with a different transformer, quantization or step count -- and text models have supported it for a long time, addressing the second under its alias and skipping an unaliased duplicate with a message. Image models had none of that. The symptom is worse than losing an entry, because nothing looks lost: both entries still appear in /v1/models and both register their aliases, so a request for EITHER is answered with whichever config was registered last. Found while testing two configs of Qwen-Image-Edit-2511, one with a GGUF transformer and one with a fine-tuned transformer quantized at load: a request for the GGUF config was served by the 4-bit one, and the only clue was "Registered image model: Qwen/Qwen-Image-Edit-2511" appearing twice in the startup log. So the image path now does what the text path does: a sibling sharing a checkpoint is registered under its alias, which gives it a config key of its own, and is NOT aliased back to the shared path -- that mapping is what collapses the two again. An unaliased duplicate is skipped with the same explanation text models give. `path` now travels in every image config, because a sibling is addressed by an alias and an alias is not something from_pretrained can open: the config has to be able to say which checkpoint it is a config OF. _load_diffusers_pipeline prefers it when it differs from the name it was called with, and for a primary entry the two are the same string. Needs a restart to take effect, registration being a startup act: a collision that has already happened cannot be undone in the running process's config dict. Full suite 4558 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.4.96 added gguf>=0.10.0 to requirements.txt, which was half right. Two things about how this image is built that the change missed: * requirements-oci.txt has carried `gguf>=0.9.0` since June, so the dependency was not undeclared after all -- only unsatisfiable. diffusers' GGUFQuantizationConfig refuses anything below 0.10.0 ("requires both PyTorch and GGUF>=0.10.0"), so a 0.9.x install satisfies the line and still cannot load a GGUF pipeline component. Pinned to 0.10.0. * Dockerfile.oci COPIES requirements.txt, requirements-nvidia.txt and requirements-vulkan.txt into the build and then deletes them unused: the RUN installs `-r /tmp/requirements-oci.txt` and nothing else. So for the container image, requirements-oci.txt is the only list that counts. requirements.txt still matters for a host build.sh install, and the in-image upgrade reads BOTH, which is why the entry there is not wasted. No rebuild needed to ship it: coderai-upgrade's delta install takes requirements.txt plus requirements-oci.txt, installs what is new, and the autoupgrade commits the result onto the image tag. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Requested for a client that edits photos by instruction -- "make the dress red", "add a necklace" -- which plain img2img cannot do at any strength, because `strength` is the fraction of the source it destroys before re-imagining it. Serving Qwen-Image-Edit / FLUX.1-Kontext class models turned up two latent bugs on the way. The instruction path. /v1/images/edits now shapes its call to what the pipeline accepts, from the signature, rather than branching on the model: QwenImageEditPlus / QwenImageEdit / FluxKontext image ✓ strength ✗ SDXL-Img2Img / QwenImageImg2Img / FluxImg2Img image ✓ strength ✓ Taking an `image` with no `strength` IS the distinction, and it generalises, so no list of class names has to be maintained. `strength` is dropped where it does not exist -- a caller cannot know which kind of model it is addressing, and these pipelines carry no **kwargs to swallow it, so passing it through would be a hard TypeError. `guidance_scale` becomes `true_cfg_scale`, the knob they actually read. Steps default to 4 at cfg 1.0 for them instead of inheriting img2img's 30 at 7.5, which would waste a minute an image and overcook the result. Up to three images may be given (`image` plus `images`) for a composed edit, conditioned at one shape; more is a 400. `instruction_edit` is reported in /v1/models so a client can hide the strength slider and treat the prompt as an instruction. Verified to classify the four Qwen/Kontext/FireRed ids and to leave Pony, Z-Image and Illustrious as img2img. component_models is new: a map from a component to ANOTHER repo, loaded and handed to the pipeline as a prebuilt object. Several Qwen-Image-Edit derivatives ship a transformer and nothing else -- the base supplies the VAE, the text encoder and the scheduler -- and there was previously no way to express that, so they could not be served at all. Its entry in component_quantization still applies, so a transformer too large for the card is quantized as it loads. Two latent bugs, both of which anyone using GGUF image components would have hit: * `gguf` was never a declared dependency. build_gguf_pipeline_components has existed through several releases and could not have worked: diffusers needs gguf>=0.10.0 for GGUFQuantizationConfig and only says so mid-load, as "Unable to load weights from checkpoint file". Now in requirements.txt, which build.sh installs. * from_single_file was called with no `config`, so it INFERRED the architecture -- and its fallback is Stable Diffusion 1.5. A Qwen-Image, Flux or Wan transformer therefore failed with "stable-diffusion-v1-5 does not appear to have a file named config.json". It now passes the base pipeline's own component subfolder, which was already at hand. Measured on an RTX 3090 after the fixes: the Q4_K_M transformer binds as QwenImageTransformer2DModel (13.23B params, 85s) and the assembled QwenImageEditPlusPipeline is 22.68 GiB resident with a 4-bit text encoder -- which fits 24 GB but leaves 1.32 GiB and OOMs on 390 MiB of activations. That is a deployment setting, not a code one, and it is why the figure is measured here rather than asserted. Full suite 4558 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Fixes a NameError shipped in 0.4.94: `name 'model_key' is not defined` on every /v1/images/edits request. Replacing that endpoint's load block with the shared _pipeline_for_task call removed the line defining model_key, while a line further down still used it to look up the model's config. The inpaint endpoint got its replacement definition; this one did not. Editing was broken from the moment it was deployed, and a client reported it from production. 187 tests covered this code and none of them caught it, because every one read the source as text -- is this guard present, does this call precede that one. A structural test cannot see an undefined name on the first line of the request path. That is a flaw in how the work was tested, not bad luck. So the endpoints are now executed. tests/test_image_edit_endpoints_actually_run.py calls create_image_edit and create_image_inpaint with their collaborators stubbed -- the manager, image decode, the response writer, the pipeline -- so the whole body between the request arriving and the pipeline being called actually runs. Checked the way a regression test has to be: with the bug reintroduced it fails with the production message verbatim, and passes with the fix. One that passed either way would have been worth nothing. The tests assert what the body is for, not just that it returns: that the SOURCE IMAGE arrives at the pipeline at the size requested, that an omitted size keeps the photo's aspect instead of squaring it, that the mask matches the image's shape, that a malformed size is a 400 from the endpoint rather than a 500 from inside it, and that a model's prompt_prefix survives every layer in between. Also noted for whoever reaches for a linter next: a stdlib undefined-name scan was tried first and would NOT have caught this one, since it cannot tell a function-local assignment from a module-level name. Real scope analysis means pyflakes, which is not a dependency here. Full suite 4558 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A checkpoint is a UNet, a VAE and its text encoders; which task it performs is a property of the pipeline class wrapped around them, not of the weights. Both tasks already worked after 0.4.93, but each endpoint loaded its OWN pipeline under its own key -- so a client that creates a character with a model and then edits it with the same one held an 18 GB SDXL fine-tune twice, and the second load evicted the first. Alternating between the two thrashes the card continuously. Reading the two loaders side by side turned up the other half of it: _load_img2img_pipeline ignored the model's configuration altogether -- dtype hardcoded from global args, no offload, no quantization, no VAE override. Only _load_diffusers_pipeline honours models.json. A 4-bit model was therefore served at 4 bits for generation and bf16 for editing, out of two copies. So the model is loaded once, through the configured loader, under the manager's own key -- its VRAM accounting and eviction now see one model -- and each task gets a view built with AutoPipelineFor*.from_pipe around the components already resident. /v1/images/generations gets the same treatment in reverse, so a model an edit loaded can create a character without reloading. The views are cached on the base pipeline, not as extra models-dict entries. Deliberately: they share the base's tensors and so must die exactly when it does, and a second models-dict key per task is what let a cache key reach from_pretrained in 0.4.93. The img2img:/inpaint: keys are gone from the module entirely. Task detection asks diffusers' own tables rather than a signature, because a ControlNet text-to-image pipeline requires an `image` -- the control hint, not a source -- and a signature check would file it as img2img. Tested. Two behaviours worth knowing: the views share the UNet, so a LoRA or IP-Adapter applied through one task is visible to the others (same weights, by design); and SD, SDXL, Flux and Z-Image all have entries for all three tasks, so this is not an SD-only arrangement. I had assumed Z-Image was unmapped and wrote a test asserting it; the test caught that and both it and a code comment are corrected. 28 tests here, 187 across the image files; full suite 4549 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Reported from a client against nesaorg/PonyRealism-Uncensored as three symptoms -- "the edit result is garbage", "size is ignored", and a 500. Two defects, neither of which announced itself. The edit was never an edit. _load_img2img_pipeline picked its class by looking for 'xl', 'sdxl', 'stable-diffusion' or 'sd' in the MODEL NAME, and "nesaorg/ponyrealism-uncensored" contains none of them, so it fell through to the generic DiffusionPipeline -- which faithfully honours the checkpoint's model_index.json, and that one declares StableDiffusionXLPipeline: the text-to-image class. Every diffusers __call__ ends in **kwargs (it carries the deprecated callback pair), so `image` and `strength` were accepted, dropped, and the pipeline rendered a fresh picture from the prompt alone at its default 1024x1024. One defect, all three symptoms: garbage (a prompt saying "add X, keep everything else" rendered from scratch), size ignored (1024x1024 is the text-to-image default, not the source's shape), and 200 OK throughout. stable-diffusion-xl-base-1.0 worked only because its NAME contains "xl". Both loaders now ask AutoPipelineForImage2Image / AutoPipelineForInpainting, which read the declared class and substitute the one for this task. A guard backs it up, on cached pipelines as well as freshly loaded ones -- an install already holding a text-to-image pipeline would otherwise keep using it until it restarted. A pipeline with no `image` input is converted with from_pipe (the same components, no second copy in VRAM) or refused with 501. A 200 carrying a picture of something else is the one outcome worth ruling out. The 500 was a cache key used as a repo id. These endpoints keep task-specific pipelines in MultiModelManager.models under "img2img:<repo>", and the manager's fuzzy already-loaded pass matched one by its last path segment, then assigned the whole key to resolved_name -- so from_pretrained was asked for a repo called "img2img:nesaorg/PonyRealism-Uncensored" and answered "Repo id must use alphanumeric chars" about a model that loads perfectly well. That pass now skips task namespaces, which the endpoints look up by exact key anyway. Writing the test for that found two more namespaces with the identical latent bug -- depth: and segment: -- so _TASK_NAMESPACES covers four, and an invariant test asserts every task prefix the endpoints invent is declared. A future endpoint cannot reintroduce this quietly. Not fixed, because it is not a bug: /v1/images/edits is plain img2img, so "add X and keep everything else exactly the same" cannot work however correct the pipeline is. That needs a mask through /v1/images/inpaint, or an instruction-edit model exposed with a capability clients can prefer. 16 tests; full suite 4537 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
clip_skip was mapped onto diffusers' argument of the same name, and that argument counts something else. Its SDXL path reads hidden_states[-2] # clip_skip is None hidden_states[-(clip_skip + 2)] # otherwise so diffusers' clip_skip is the number of layers skipped BEYOND the penultimate one SDXL already uses. Every model card, A1111, ComfyUI and InvokeAI mean something simpler by the name: 1 is the final layer, 2 the penultimate, n the n-th from the end. So a request asking for 2 -- which is what the booru-tag SDXL families are trained at, and what their model cards all say -- read hidden_states[-4] instead: two layers too deep, the equivalent of clip-skip 4, which on a Pony-family model is the difference between a photograph and mush. It was inconsistent as well as wrong: the same 2 meant [-4] on SDXL and [-3] on SD 1.x, two depths from one number. The mapping now happens here, once, in the sense users mean, and clip_skip 2 reaches the same layer whichever family the model belongs to. "Mirrors diffusers exactly" was the mistake in 0.4.89: mirroring is the wrong goal for a field a caller fills in. SDXL's pooled conditioning vector is also handled properly now. The UNet cannot run without it and diffusers refuses prompt_embeds that arrive without one, so handing it half the set is a ValueError raised in check_inputs -- an instant 500, before any denoising. Where a pipeline does not offer the vector, it is rebuilt the way CLIP produces it (pooled at the end-of-text token, then projected through text_projection) rather than giving up the weighting and the window count this module exists for; only when even that is impossible does the encoder stand down and let the pipeline encode the prompt itself. The invalid combination is no longer constructible: either all four keys or None. A 500 from /v1/images/edits, /inpaint or /outfit now logs its traceback. The cause used to reach the caller alone, so the log showed the request and then the status, and a failure there could not be diagnosed afterwards at all. Test note: test_a_400_is_not_relabelled_as_a_server_fault scanned a fixed 250-character window for the HTTPException pass-through above each catch-all; the new log line pushed that to 316-382 and broke a passing check on correct code. It now asserts structure instead of proximity -- the pass-through must be the nearest handler inside the same function, with the logging between it and the raise -- which also catches it drifting into a different function. 58 tests in the prompt file; full suite 4521 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two asks that turned out to be about the same thing: a request taken as meant. A model fine-tuned on a quality ladder needs that ladder on every request, and a client speaking the plain OpenAI image API sends a sentence with nowhere to put it. CoderAI adds no tags of its own and still does not: prompt_prefix and negative_prompt_prefix on the MODEL's config entry are comma-joined in front of whatever the caller sent, and both default to empty, so an install that configures nothing sends exactly what was written. A request overrides either, and an empty string switches one off for that call -- what a caller writing its own complete prompt needs. Applied at most once: a prompt that already opens with the ladder is left alone rather than given it twice, which would only dilute the rest. The prefix resolves once, before the encode, and the same text feeds the embedding cache key. Keyed on the prompt as SENT, two models with different prefixes would share one entry and the second would be served the first's text. The archive keeps the prompt that ran plus the one requested when they differ. On OCR: there is no per-page byte limit, and there never was. A page over a megabyte is ordinary -- an A4 scan at 300 DPI is several -- and both a 3 MB and a 50 MB body upload completely and are rejected only at authentication, with the bundled nginx at 4096m and the generated /v1/ocr routing not lowering it. What did exist was a hardcoded 100 MB ceiling on one document, which a 300 DPI colour filing passes without being unusual and which nothing could raise. It is ocr.max_upload_mb now (0 = no ceiling here), in Settings, and its 413 names the setting and says the document was NOT read -- the loud refusal ocr.max_pages already makes, instead of "file too large (max 100 MB)". The 1 MB figure that makes a large page fail in the field is nginx's own client_max_body_size default, in a reverse proxy added in front. That is exactly the shape of "anything over a megabyte fails", so it is recorded in the code, the settings hint and docs/ocr.md rather than left to be rediscovered. Also documented there: what IS capped is pixels, not bytes -- Surya-2 and Chandra top out near 6.29 MP, an A4 page at about 255 DPI, and above that the page is silently downscaled to fit. 37 tests, including a real 1.25 MB single-page PDF built from noise and read back through pdfium as one page. Full suite 4512 passed with the same 15 pre-existing failures as 0.4.90 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three things the image endpoints accepted and then quietly did something else with. size. A size that did not parse fell through to the default, so "1024" or "1k x 1k" generated a square nobody asked for and the response looked like a success. A side that was not a multiple of 8 went to the pipeline as-is, where the VAE cannot represent it. Nothing capped it either, so 4096x4096 was accepted and came back as a CUDA OOM -- which reads like a busy GPU rather than a size the caller chose. Now: a 400 quoting what was sent, a 400 for a zero dimension (snapping it up would answer an uninitialised client variable with an 8x8 image), snapping to the grid with a log line, and a 400 naming image.max_side / image.max_pixels when it is over the ceiling. "auto" is accepted as in the OpenAI API, and the dead `width != width` NaN guard on a value that was already an int is gone. The archive records the size actually generated alongside the one requested. /v1/images/inpaint no longer defaults to 1024x1024, which squashed every non-square photo and pulled the mask out of alignment with it -- worse than the same default on /v1/images/edits, where it was fixed in 0.4.86. Omitted now keeps the source's shape. Textual inversions. easynegative, bad-hands-5 and the rest are vectors living in CLIP's embedding table under a word of their own, and until the file is loaded that word is an ordinary unknown token -- so a large part of the ecosystem's negative prompting asked for nothing at all. They load from the request (embeddings: [...]), from the model's config (textual_inversion), or by filename from image.embeddings_dir, which is how A1111 works and what a prompt copied off a model card assumes. Two details decide whether this works at all: an SDXL-format embedding holds a clip_l and a clip_g tensor that must go into the matching tokenizer/encoder pair, and a multi-vector embedding occupies several tokens (tok, tok_1, tok_2...) that the prompt has to be expanded into -- without it an eight-vector embedding contributes one vector and appears inert. A name the caller gave that will not load is a 400; a bad file in the directory is a line in the log, because one file should not break every request. /v1/images/outfit goes through the same encoder now, so the weight grammar and clip_skip reach it too. Its per-frame video path encodes the prompt once for the clip instead of once per frame. And a 400 is no longer reported as a failed backend. On the first request for a model that is not loaded yet, the generate call sits inside the try that falls back from diffusers to sd.cpp, so a rejected request was caught as a load failure and answered with a report about which backends failed. The three edit endpoints did the same through their catch-alls. All four re-raise HTTPException untouched. Found by the tests while writing them: the embeddings directory was searched by the renamed trigger word rather than the filename, so "easynegative:myneg" never resolved, and "768X512" did not parse. 64 new tests; full suite 4475 passed with the same 15 pre-existing failures as 0.4.89 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A CLIP text encoder reads 77 tokens, two of them markers. Handing a longer prompt to a diffusers pipeline does not fail: the tokenizer truncates it and the model renders what fits. The booru-tag fine-tune families (Pony, Illustrious, NoobAI) open with a run of quality tags, so the part that got dropped was the subject -- and the request answered 200 with an image, which is why it was easy to miss. Image prompts are now encoded here rather than by the pipeline, which is what makes the rest possible: * as many 75-token windows as the prompt needs, concatenated -- no limit * the A1111 weight grammar: (tag:1.3), [tag], bare (tag) = 1.1, nesting multiplies, \( escapes. Per-token scaling with the window mean restored, so emphasis does not drag the brightness along with it * BREAK, which closes the current window instead of blending tags meant for different subjects * clip_skip, mapped to diffusers' own convention -- SDXL counts from the penultimate layer, SD 1.x from the final output and needs the layer norm applied by hand. Getting that backwards shifts every image Three places were dropping the negative prompt outright. The sd.cpp path hardcoded negative_prompt='', so for every GGUF image model whatever the caller asked to avoid stopped at the API -- half the idiom, since the quality tags have a negative half (score_4, score_5, score_6). /v1/images/edits and /v1/images/inpaint had no field for one at all, so an edit could only ever be steered positively. All three now pass it, and take clip_skip and prompt_weighting with it. clip_skip and prompt_weighting are also per-model and server-wide settings, because a client speaking the plain OpenAI image API has no field to ask for clip_skip and a model trained at 2 needs it on every request. Request beats model beats global. Weighting defaults on, so a literal '(' in prose now weighs 1.1 -- the ecosystem's convention, with \( and prompt_weighting:false as the way out. What that cannot be allowed to change is prompts that already worked: against a real CLIPTextModel a short unweighted prompt encodes bit-identically to what diffusers produced, because CLIP pads with its own end-of-text token, so [bos]+toks+[eos]+pad... and [bos]+toks+pad...+[eos] are the same ids. The test records that reasoning, so a tokenizer where it stops holding is caught. Flux, SD3 and Z-Image carry a T5 with its own limit and conventions and keep their pipeline's native text path. 49 tests; full suite 4410 passed with the same 15 pre-existing failures as 0.4.88 (diff empty). Co-Authored-By:
-
- 24 Oct, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored
A deployment has to point HF_HOME somewhere before any config is read, so a container bakes it into the image. models.hf_cache_dir was then applied in exactly one place -- inside the server's main(), by mutating os.environ, which reaches no other process. So the front used the configured cache and everything else used the image's: supervisord's other children, and the isolated-venv workers, which were handed HF_HUB_CACHE and HUGGINGFACE_HUB_CACHE but not HF_HOME or TRANSFORMERS_CACHE, enough for a worker to re-fetch its model into the directory the image named. Measured on one install: 22 GB in the image's directory, 13 GB of it a byte-for-byte duplicate of the configured cache, three copies of one model, and the whole of it on the smaller of the two disks -- which was 99% full. The configured value is now applied from codai/__init__.py, before anything can import huggingface_hub, transformers or torch and read these variables once at import time. Every coderai process imports the package, so every one of them agrees without the launcher being involved. All four variables, since leaving any behind splits the cache again. main() keeps an authoritative pass against the config it actually loaded (--config may name another directory), the venv workers get the same four through child_env(), and the OCR fleet supervisor -- stdlib-only on purpose -- reads the same key from the config.json it already opens, instead of building the path from CODERAI_CACHE_DIR. Nothing is invented: with no hf_cache_dir configured the environment is left exactly as found, and CODERAI_NO_HF_CACHE_ENV opts a process out. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
POST /v1/files/upload exists so a client sends a file once and then names it: inlining a 40 MB clip in every request is the waste it was built to remove. But it stored its content-addressed files in the output directory and answered 501 when there was none configured -- so on such an install the only way to send media was the inlining the endpoint replaces, and a client that had already switched to uploading saw its photo edits fail outright. An install always has a cache directory, so uploads fall back to it (media.uploads_dir()). With an output directory nothing moves: an upload still lands there beside the generated files. Both stores stay in the lookup path, so an install that gains --file-path later does not lose track of what it already holds, and /v1/files/<name> serves from the cache store only for up- names -- the cache directory does not quietly become a second public output directory. 501 now means there is nowhere at all to write; a non-writable store answers 507 rather than a 500 out of makedirs. Composing a new video still needs an output directory, since there has to be somewhere to write the result. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Four ways an edit could answer 200 while quietly doing the wrong thing. The enhancement paths decode a clip to frames and encode a new video, so the audio was left behind: a clip that arrived with speech came back silent. Only the ffmpeg minterpolate fallback -- the path nobody takes by default -- ever kept it. _mux_original_audio() puts the track back on all three in-process paths, stream-copied where MP4 accepts the codec and re-encoded to AAC where it does not. Both paths already preserve the source duration, so nothing is stretched. A silent source, or a remux PyAV cannot do, returns the frames unchanged rather than losing a finished GPU job. The image and video editing endpoints took base64 (and, for video, an http URL), so a caller that had already uploaded a file had to re-send it on every edit -- megabytes a call for a file on the same disk. They now resolve through the compose resolver, so a /v1/files path or a sha256: upload id works everywhere, and our own files are read off disk instead of fetched back over a loopback that would need a token. Responses carry the relative path beside the url, as compose does, so a caller need not depend on the public hostname. Routing through that resolver would have tightened what callers may send: it validates base64 strictly, which rejects the 76-column wrapping that MIME, the base64 tool and encodebytes all produce. Whitespace carries no meaning in base64, so it is now stripped rather than refused. /v1/images/edits defaulted size to the string "1024x1024", so omitting it squashed every photo that was not square. Omitted now keeps the source's shape at the ~1 MP a diffusion model wants, on the 64-grid it needs. /v1/images/upscale fell through to PIL LANCZOS for an unrecognised model and still answered 200, charging for an AI upscale that never ran and returning something merely soft. That is a 400 naming the models that would work. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The line was printed on the way INTO a re-read, so an install with `retry_degraded` off -- or on "auto" behind `fallback_engines: off`, which is the same thing -- produced no record anywhere of a degraded page. The engine answers 200, nothing raises, and the only other copy of the finding is in the reply, which belongs to the client: a batch could come out almost entirely truncated and leave the server log looking like a clean run. That is how ~97% of one run went unnoticed. Finding a ruined page is now separate from being able to fix it, and when the chain is empty the log says so and names the setting that emptied it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
With `ocr.workers` set, the workers dispatch every page and never rent -- the controller owns the fleet -- but the controller only ever decided to grow inside its own `acquire()`, which it never calls while the workers serve the pages. The one signal the workers did send was keyed by POD, which is exactly what a cold fleet has none of: a worker holding pages with zero pods reported `{}`, and the controller read an idle install. So a cold OCR fleet never booted for any load, and a warm one never grew past `min_pods` -- "scale to 128 pods" was unreachable whenever the workers were on. Observed live: `workers: 2`, `min_pods: 0`, `pod_pipeline: require`, a page waiting 5.8 minutes for a pod that nothing would rent; the same request rented one within 20 seconds with `workers: 0`. Demand now travels on its own, keyed by pool rather than by pod, and the scaler pass acts on it -- single-threaded with the shedding it must not race, and still through `_grow_reason`, so max_pods and the parallel-boot caps are unchanged. Second, independent fix in the same area: under `ocr.pod_pipeline: require` a fallback chain now keeps only engines that can run on a pod. `auto` orders the chain fewest-ways-to-fail first, which put paddle and docTR -- neither of which has a pod-side pipeline -- at the front of it on an install whose whole rule is that this box never reads. The documented way out was `fallback_engines: off`, and that ALSO switched off the degraded-page recovery, which reads `auto` as "the fallback order": an install protecting its orchestrator thereby lost the mechanism written for the run that came out ~97% truncated with no error anywhere. Found live, in exactly that state. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
vLLM enables async scheduling -- CPU scheduling overlapped with GPU execution -- by default, and lists plain "ngram" among the methods it cannot run with, so asking for ngram silently turns it off: the draft is paid for with a throughput loss somewhere else, and the log says so in a warning nobody reads twice. "ngram_gpu" drafts by the same prompt-lookup, needs no model and no VRAM either, and keeps async scheduling. Both spellings work; the field now suggests the one that does not take something away. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Decode on one GPU is bound by how fast the weights can be read: measured at 30 tok/s against a ~28-30 tok/s bandwidth ceiling, it is already at the wall and no setting moves it. Prefill is not -- it is compute-bound, and on a long prompt it is nearly all of the request. So the levers that matter are the ones that avoid work rather than hurry it, and each of them now has a field of its own. Prefill batch (--max-num-batched-tokens). vLLM's default is 2048, so a 60k-token prompt is prefilled in ~30 sequential passes of small GEMMs that leave the tensor cores idle. The cost of raising it is a higher activation peak, taken out of the KV cache, so the field says to check the boot report afterwards rather than implying it is free. Prefix caching (--enable-prefix-caching), as a tri-state. vLLM's own default is per model -- on for an ordinary decoder, off for a hybrid-attention one, where it is supported but still opt-in upstream -- so "leave it alone" has to be expressible and has to be the default. For those models this switch is the only way to get it, and where the same document is asked about repeatedly it turns a whole prefill into almost nothing. Speculative decoding (--speculative-config). "ngram" is spelled out as a word because it needs no draft model, no second checkpoint and no VRAM: it drafts by finding the current text earlier in the prompt, which pays off precisely on answers that quote their source. Anything else is passed through as the JSON vLLM documents, so a real draft model needs no code change. Quantization and dtype, which were accepted by the save route but had no field anywhere. The hint says to leave quantization blank: it DECLARES how the weights on disk are packed and converts nothing, vLLM reads that from the checkpoint and picks its fastest kernel, and a repo whose name says AWQ can still be compressed-tensors -- so naming a format can only lose speed or stop the engine booting. Thinking, per model. A thinking model generates its thought first and charges it against max_tokens, so on extraction work it is most of the latency and none of the answer; on a tight budget it spends the whole allowance before the answer starts and returns an empty one. Until now only a request could turn it off, which an install does not control -- its clients do. Three states, since unset must never be read as "do not think" for a model whose answers depend on it, and an explicit request value still wins either way. The vLLM card is also shown for a model served by a vLLM POD, which is where these settings matter most, and that fixes a silent data loss: the card was hidden for `backend: runpod`, so the collector sent `vllm: {}` and the save route read an empty block as "this model has no vLLM settings" and deleted the stored one. Merely opening such a model and saving it dropped its block -- kv-cache dtype and the cache floors with it -- with no error, and nothing changed until the next pod booted in a different precision. An engine block is now sent only when its card was on screen to be edited; otherwise the key is left out, which the route ignores. The regression test fails on the previous code. Co-Authored-By:
-
- 23 Oct, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
Measured against the production pod: a reasoning model asked for 64 tokens spent all 64 inside its thought, returned NO content, and the response said finish_reason "stop". max_tokens=64 finish=stop content=0 chars '' reasoning=262 max_tokens=1500 finish=stop content=13 chars 'XMU-5635-FB' reasoning=448 Two layers were each sure the other knew better. The proxy hardcoded finish_reason = "stop" and never read choices[0].finish_reason from the remote; OpenAIFormatter then hardcoded it again, so even a caller that worked the reason out had it overwritten on the way to the client. vLLM had been saying "length" all along. It matters because "length" is the one reason a client can act on. An answer cut off at the limit is half a sentence, and told it stopped normally the client has nothing to retry on and no sign anything is missing. On a reasoning model the thought is generated FIRST and spends the same budget, so a tight limit returns an EMPTY answer that claims to have finished of its own accord -- which is how this surfaced, as a retrieval test scoring zero for reasons that had nothing to do with what it was measuring. The remote's word is now carried through the per-call sink beside reasoning, tool_calls and logprobs (get_last_finish_reason, buffered and streamed), the formatter accepts the caller's reason and keeps "stop" only as the default, and a tool call still wins -- that describes the answer's shape, and a truncation after the call was formed must not relabel it. Fixed at all five response exits, not just the one it was found at: streamed chat (both terminal-chunk paths), buffered chat, and streamed and buffered /v1/completions. A local backend is never told a reason, so the budget decides: generation that used every token it was allowed was cut off, not finished. Drawn only when a limit was actually set and reached -- never from a token count alone, or every long completion would be mislabelled. Tests verified against the reverted files: assert None == 'length' for the backends, TypeError: unexpected keyword 'finish_reason' for the formatter. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The pods page drew one bar -- host RAM -- and printed VRAM underneath it as a line of small text. On a page whose job is "what is this rented A40 doing", the GPU was the one reading an operator had to take differently from everything else on the row. Both now get a bar, each labelled, each scaled to its own total. The colours differ on purpose: host RAM keeps the green/amber/red bands, because that is the ceiling that has actually killed pods (an OCR pod was OOM-killed at 128 in-flight with the GPU never the constraint). VRAM gets a neutral bar and says "reserved", because vLLM claims its gpu_memory_utilization share at boot and so reads ~90% from the first second on every healthy pod -- run that through the load bands and every pod is permanently red, an alarm that is always on, which an operator learns to ignore and then ignores the real ones with it. Either bar is omitted when its telemetry is missing: a booting or unreachable pod has none, and an empty bar reads as "idle" when the truth is "unknown". Tested by running the page's own _memCell() in node over real payload shapes. Co-Authored-By:
-