- Oct 27, 2026
-
-
Stefy Lanza (nextime / spora ) authored
The LTX-2.5 key reconciliation shipped, the retry failed identically, and the repair had never run. It needs the component's config.json to know which names the class expects, and _component_dir looked only in the local snapshot -- which, for a component whose weights come from a GGUF, is precisely the subfolder nobody downloads. 20 GB of weights the file replaces are not worth fetching; a few hundred bytes of config are. Cache first, then one hf_hub_download of that component's config.json. Same lesson as a cache-only existence check on a rented pod: an answer of "not here" from a cache is not an answer about the repo. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZnLhydX9QqUcfBbqLu
-
Stefy Lanza (nextime / spora ) authored
The repair that fixed LTX-2.3's prompt-adaln names could only strip a suffix. LTX-2.5's GGUF needs a whole segment renamed: it calls the attention norms `q_norm` / `k_norm` where LTX2VideoTransformer3DModel expects `norm_q` / `norm_k`. 724 parameters, one to one, the same tensors -- and unreconciled they stay on the meta device, so a 37-minute load ends with Cannot copy out of meta tensor; no data! about weights the file was carrying all along. Measured by diffing the GGUF's 4349 tensors against the class's 4091 parameters: nothing was absent, 724 were simply spelled differently. Both directions, because nothing says which spelling the next exporter picks, and both repairs now run in one pass -- a checkpoint can need each for a different family of keys. The existing guard is unchanged: a key moves only when the class does not want its current name, does want the renamed one, and nothing already holds the target. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
diffusers reports a missing single-file checkpoint as Invalid `pretrained_model_name_or_path` provided. Please set it to a valid URL. which sends you to check the repo id. The actual cause, measured on LTX-2.5, was an entry pointing into ~/.cache/huggingface on the HOST -- a path the container does not mount -- so the file existed everywhere except in the process trying to open it. Every other part of that model was ready. The path is probed before the load and named in the failure, with the reason it is usually absent. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
_apply_sampler turns a cross-family sampler into a 400 that explains itself: flow-matching and DDPM schedulers parameterise the trajectory differently, so one cannot stand in for the other. The acceleration config applied its `scheduler` with no such check, and a card is exactly where that mistake gets made and never noticed. Krea-2-Turbo's entry asked for EulerDiscreteScheduler on a flow-matching model. The swap succeeded silently and the generation then died with EulerDiscreteScheduler.set_timesteps() got an unexpected keyword argument 'mu' which names neither the card nor the model, arrives from inside diffusers, and reads like a library bug. Now it is refused and logged, naming both schedulers, and the model keeps its own -- acceleration is an optimisation and has always been allowed to decline. One set of family names, imported by both paths, so they cannot drift. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The silent one. self.models is keyed by model_key, and model_key was the weights path -- so two entries naming the same path shared ONE resident pipeline. LTX-2.3-PinkCherry requested while LTX-2.3-10Eros was resident never loaded its own transformer: no load, no eviction, straight to generation, and HTTP 200 with 10Eros weights. Every other instance of this family failed loudly; this one answered successfully with the wrong model. It also meant both wrote measured_vram_gb to the same key, so a Q4_K_M and a Q5_K_M of different sizes were overwriting each other sizing. A key is split only when the caller named something, two or more DISTINCT aliases point at that same path, and the named one is among them. A lone aliased model, or a request that names the path, keeps the key it has always had. That narrowness is the point, not caution for its own sake: collapsing every name form onto one key is deliberate a few lines above -- without it two concurrent first-requests under different spellings each load an instance and exceed max_model_instances=1. This is not a licence to key by whatever the caller typed. Verified live: the estimator now reports key=video:LTX-2.3-10Eros where both models previously shared key=video:diffusers/LTX-2.3-Diffusers. Suite: 15 failed, 5268 passed -- the same 15 as the baseline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A request for LTX-2.3-10Eros loaded LTX-2.3-PinkCherry: a different 15.9 GB transformer, silently, and from an entry that was DISABLED. self.config is keyed <kind>:<model_name> where model_name is the weights path, so entries sharing a path clobber one another and only the last survives. That is not a corner case -- it is how one base checkpoint is served under several configurations: lisa / lisa-32k differ only in context length, 10Eros / PinkCherry only in which GGUF is their transformer. PinkCherry comes second in models.json, so it won, parked or not. The VRAM estimator already disambiguated this way, by the id the caller asked for, after lisa-32k was once sized at lisa 51.97 GB. The config used to LOAD the model did not. Now both do: * every type setter stores a config under its ALIAS as well as its path, so a sibling is reachable by the only name that distinguishes it. The path key keeps its old meaning (last writer wins) for callers that know only a path. * _config_for_model_key takes the requested id and prefers the entry whose alias matches. Applied to all eight setters rather than video alone -- the collision is the same for audio, image and embedding, and fixing one type is how this survived being found twice already. Also: build_gguf_pipeline_components passed `model_name` as the diffusers repo supplying the component configs. Once siblings became reachable by alias that can BE the alias -- "LTX-2.3-10Eros", not a repo -- and from_single_file rejects it. The repo is whatever the entry names in `path`, every time. Verified live: LTX-2.3-10Eros now loads its own transformer and the pipeline assembles, 8/8 components. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The LTX-2.3 GGUFs failed with "Cannot copy out of meta tensor; no data!", and the obvious reading -- parameters the checkpoint lacks -- was wrong. diffusers 0.38 converts an LTX-2 checkpoint prompt-adaln weights to prompt_adaln_single.* and audio_prompt_adaln_single.*, while LTX2VideoTransformer3DModel expects prompt_adaln.* and audio_prompt_adaln.*. Twelve names, one-to-one, the same tensors. They were present all along, under a name nobody reconciled, so they never materialised and stayed on meta. Measured identically on LTX-2.3-10Eros v1.5 and PinkCherry v1.8: 4444 GGUF tensors, 4186 expected parameters, and after the rename 0 missing and 0 unused on both. The tempting fix was to initialise those twelve. That would have invented weights to paper over a rename -- precisely what the surrounding refusal exists to stop -- so it is NOT what this does. A key is moved only when the class does not want its current name AND does want the renamed one AND nothing already occupies the target, which makes it a rename of a tensor the checkpoint has and never the creation of one it lacks. A test pins that: a genuinely absent parameter must stay absent and the load must still fail. The converter is a third-party global, so it is wrapped for the duration of one from_single_file call and restored in a finally. Version-robust: 9 pass against the image diffusers 0.38, including the real checkpoint; against 0.37 the class has no prompt_adaln modules at all, so the repair is correctly a no-op and that test skips rather than asserting a fact about someone else is version. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
"Cannot copy out of meta tensor; no data!" means some parameters were never materialised, and it names none of them -- which sends the reader hunting for a corrupt file. Measured on LTX-2.3-10Eros and -PinkCherry: diffusers already ships convert_ltx2_transformer_to_diffusers and it works almost completely, mapping their 4444 GGUF tensors onto exactly the 4186 the class expects. Precisely 12 remain unprovided, all in one conditioning branch, identically in both fine-tunes. So the checkpoints are fine and the converter is fine; those fine-tunes were exported without that branch, and the 12 stay on the meta device. The refusal added earlier now carries that analysis: how many tensors mapped, how many remain, and which submodules they belong to. Best effort throughout -- any failure to analyse returns nothing and the original error stands. It deliberately does NOT offer to fill them in. Supplying parameters a checkpoint does not contain is a substitution, and the surrounding refusal exists precisely because a silent substitution (here: the 38 GB bf16 transformer, on a 24 GB card) is worse than a failure. A test pins that: the hint must not grow a flag that papers over it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/models re-reads models.json on every call, but the loader's alias map is built ONCE at startup. So an entry added while the server runs is advertised immediately and cannot be loaded at all: the alias resolves to nothing, the raw alias reaches diffusers as a repo id, and the caller gets Repository Not Found for url https://huggingface.co/api/models/<alias > -- a 401 against HuggingFace for a model sitting on the local disk. Measured today registering Wan2.2-TI2V-5B: weights cached, `path` correct, and the request failed that way; the SAME model requested by its path loaded immediately, which is what pinned the cause to the alias map rather than the entry or the weights. The pre-existing Z-Image alias, updated rather than added, worked throughout -- that contrast is the whole diagnosis. This is the fifth instance of one pattern in this codebase: a name in models.json not matching the name the runtime registered. v0.5.4 read a backend from the wrong map, v0.5.5 looked for a binary where it does not live, v0.5.7 published an id that resolved to nothing, the LTX pair shared one path between two entries, and now the listing advertises what the loader cannot resolve. A cross-cutting identity-resolution audit is still the right fix; this is the narrow one. An alias the map has not seen is now looked up in the config itself and cached. A parked entry is deliberately NOT resolved -- parking is how an entry waits on a blocker, and resolving it would hand clients a model that answers 500. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
9ae240b0 committed its tests and its documentation and none of its implementation: four files instead of twelve. The `git add` that should have staged the rest named a pathspec for requirements-acestep.txt, which did not exist, so git refused the whole invocation and I did not check its output before committing. The message described work that was not in the tree, and the suite at that commit could not have passed. This is that work: * video.py — LTX-2.x gets its own pipeline classes. This diffusers has LTX2Pipeline, LTX2ImageToVideoPipeline, a video AND an audio VAE, but only the LTX 0.9 family was mapped, so every LTX-2 entry in models.json sat parked: the name matched and the class did not. Split on the version in the name, since both families are just called "LTX" and a fine-tune keeps its base version (10Eros is LTX2.3, PinkCherry is LTX23). * hf_loading.py — `civitai:<modelVersionId>` references, and a GGUF component that fails to load now fails the load instead of being skipped. A skipped component is simply absent from the dict, so the base pipeline loaded its own full-precision copy: the operator asked for a 14 GB quantized transformer and got a 38 GB bf16 one, while the VRAM estimate still said 14 GB and the planner therefore freed about a third of what the load needed. * presenter.py — LatentSync and MuseTalk as first-class engines, ahead of wav2lip for a clip (wav2lip works at 96x96 and upscales, which is the soft mouth on a 1080p reel). Both are video-driven, so sadtalker still leads for a still portrait. Recognised from a shim on PATH or a managed install whose weights are already present — never "could be downloaded", because a compose job is the wrong place to fetch 5 GB. * cache/__init__.py + app.py — the loop stall. scan_cache_dir is not a listing: it stats every blob of every revision of every repo, measured at 27.3 SECONDS on this machine's 900 GB cache, and it was called from a per-model VRAM estimate, a GGUF path lookup and a POLLED endpoint, all on the event loop. 6631 stalls over 0.5s, worst 13.0s, serving nothing. Both halves were needed: memoised for 30s (short enough that a model downloaded while the server runs still becomes visible) and the poll moved to asyncio.to_thread, since memoising alone still puts the first 27-second call on the loop. * manager.py — /v1/models reads `modes` from a video entry's long-standing `video_subtypes` as well as the newer spellings. * requirements-acestep.txt — the file whose absence caused this. It is marked NOT YET INSTALLED, because it is: the install was refused as external code, nothing in it has been run, and the audio_gen loader refuses an ACE-Step model explicitly rather than mis-loading it as a MusicGen. Suite: 15 failed, 5194 passed — the same 15 as the baseline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
normalise a language before transcribing, stop inflating weights that are already quantized, and never answer 200 with nothing (v0.5.9) Five bugs and five features from a client report. The bugs share a shape: the server held a belief about a model that nothing checked against the model. A1. A TTS voice is chosen for an accent, so the language a compose render carries is "en-us" / "pt-br". Passed to transformers' Whisper -- which accepts only the English NAME -- it raised "Unsupported language: en-us" and 500'd every captioned scene of every render. Reproduced verbatim against the running server before the fix. New codai/api/lang_codes.py normalises once, per backend: the NAME for the transformers pipeline, the ISO code for whisper.cpp, Canary and faster-whisper, and None (auto-detect) for a tag that maps to nothing -- refusing to transcribe because the caller named a dialect is never the better outcome. Language VALIDATION now compares by meaning, so a model declaring "english" no longer refuses "en-gb", and compose's model picker stops disqualifying the only model that can time words. A3. FLUX.2 (dev/klein/pro) is a unified generate-and-edit model whose name carries neither "edit" nor "kontext", so it fell through to the generation branch and advertised plain img2img -- telling clients to show a `strength` slider for a pipeline that has no such argument. Confirmed live: ['image_generation','image_to_image','inpainting']. diffusers 0.37 agrees it is an instruction edit (Flux2Pipeline takes `image`, not `strength`), which is what the runtime already decided on; only the advertised list disagreed. A4. The estimator normalises a disk size to the load precision. A GGUF was special-cased to 1.0 because its bytes ARE its footprint; an HF checkpoint published pre-quantized is the same case by another route and was not. Z-Image-Turbo-unsloth-bnb-4bit: 16.3 GB of 4-bit weights, 1.70x factor, estimated 30.03 GB against a 24 GB card. Nothing can free 30 of 24, so every still ran partly on the CPU -- ~72 s for one image from a few-step turbo model. A5. Qwen3.5-9B answered content "" with HTTP 200 for four weeks. All three rungs of the generation ladder raised ("Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)") and the last one assigned generated_text = "" and returned normally. Now raises. The Triton error is a quantized model whose weights were CPU-offloaded: such a load succeeds and then fails every request, so it is refused at load time and a quantized model tries full-GPU whatever the estimate says. That also breaks a loop -- measured_vram_gb = GPU + offloaded + reserve means one offloaded load writes a number bigger than the card, and the next load reads it and offloads again. B1. A scene can carry its clip's own sound: visual.keep_audio, scene.audio_mix (voice|clip|both) and clip_volume. "clip" makes the scene as long as the clip and skips the TTS call; "both" ducks the clip under the voice. Folded into the narration track, so the music bed ducks under it like a voice. B2. Two characters talking: scene.speakers, each line in its own voice, each character lip-synced in their own column (or the frame cutting to whoever speaks). One sync run per CHARACTER, not per line -- each gets a track holding only their own lines, so the mouth moves on their words and stays closed while the other talks. Captions never mix two speakers and carry the attribution into the .srt/.vtt. B3. characters: ["alice","bob"] on images and video. The gap was never the field: IP-Adapter conditions on the MEAN of its references, so a second character's pictures produced one person who looked like neither. Reference sets are kept apart and handed to a model that takes several (FLUX.2, Qwen-Image-Edit); a model that cannot do it says so instead of blending. B4. POST /v1/moderation -- nsfw, faces, age, face_match over images or sampled video frames. It fails CLOSED: a missing model is a 503 naming it, not a result with the check quietly absent, because a caller cannot tell a missing check from a passed one. minor_detected is nullable (explicit content with no face cannot be cleared on age) and keys on the LOW end of an age interval, not a point estimate from a model with a multi-year error. C5. Per-model defaults (steps, guidance, sampler) that fill in only what the caller omitted, published in /v1/models alongside modes and licence -- so a client stops keeping a table keyed on the model NAME and an 8-step model stops being run for 30. A sampler switcher that refuses to cross the flow-matching boundary (a FLUX-family pipeline handed a DDIM produces noise, not a worse picture). Stable Audio gets its own pipeline: it was mapped to AudioLDM2, so a configured entry could only fail at load. Two of my own regressions, caught by the suite's guard tests and fixed here: /v1/moderation was unmapped in the remote gateway, and the moderation availability probe LOADED each model -- which turned the /v1/models listing into a 350 MB download on the event loop. Suite: 15 failed, 5146 passed -- the same 15 as the baseline. Co-Authored-By:
-
- Oct 26, 2026
-
-
Stefy Lanza (nextime / spora ) authored
A whisper-server model could not be used by the id /v1/models advertises for it. Asking for it reported the weights as a missing HuggingFace repository: 500 Transcription error: whisper-large-v3-turbo-q8_0.bin is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models ' which reads like a broken download and is nothing of the kind. One whisper-server is described by TWO models.json rows: a RUNNER, carrying model_path + server_path + port, registered under its id (whisper0..whisper3 here); and a MODEL row naming the weights, which registration skips on purpose as "the model config, not a runner". But /v1/models takes its id from the model row -- so the published name was the one name that matched nothing: not a runner key, not an alias. The lookup missed, and the request fell through to the generic audio path, which can only interpret a string it does not recognise as a hub repo id. Now every name that identifies the same runner resolves to it: its own id, an alias (round-robin as before), and the weights by full path, basename, or basename without extension. Where several runners share one file, naming the file prefers an idle one, so it balances the way an alias does instead of always landing on whichever registered first. This is the third defect in one week with the same shape -- a name in models.json not matching the name the runtime registered. v0.5.4 read a backend from the runtime map instead of the file that declares it; v0.5.5 looked for a binary in a directory that does not exist; this one publishes an id it cannot resolve. The pattern is worth an audit of its own. Also pins a rule on the VRAM estimator that it already followed in most places and now cannot lose: an estimate of zero means NOTHING WAS MEASURED, never "measured as needing nothing". A stored zero was already read as absent; v0.5.6 stopped a configured zero from claiming authority over a real measurement; the two tests added here also cover a zero arriving with force_vram_update, which is the one flag that makes a measurement beat the operator's own number. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
fit the caption to the frame, install the font that draws it, and stop reading an unset field as a free GPU (v0.5.6) Four defects behind one report: vertical renders came back with caption lines running off both edges, in the wrong typeface, and speech-to-text answered 500/503 while a text model held the card. THE FONT WAS NEVER IN THE IMAGE. Dockerfile.oci and Dockerfile.update install fonts-montserrat; Dockerfile.oci-venv -- the one build_oci_image.sh actually builds -- installed no fonts at all, so no image built the normal way has ever had the family the compose API documents and defaults to. libass does not fail on a missing family, it substitutes: DejaVu is considerably wider, which is how a missing package became a layout bug with nothing but a log line to say so. All three Dockerfiles now install the fonts AND fontconfig (Dockerfile.oci had the fonts but not fc-match, so it could not even tell whether a family existed). And because an image rebuild is a poor way to add a brand font, fonts can now be supplied per installation: files dropped in <config>/fonts -- or wherever CODERAI_FONT_DIR points -- are registered at startup by writing a fontconfig snippet into the user config. The snippet matters more than it looks: captions are burned by libass inside an ffmpeg CHILD process, which does its own fontconfig lookup, so a font this process can find is not necessarily a font the renderer can use. The config volume outlives both a restart and a new image, which is the point. CAPTION SIZE CAME FROM THE FRAME HEIGHT. On a vertical video the height is the LONG side: at 1080x1920 the font was sized 0.052 * 1920 ~ 100px to be drawn across a 1080px frame. It is now sized from the short side, which leaves 16:9 byte-identical (there the short side IS the height) and gives 9:16 the same text-to-width ratio as 16:9. Line breaking upstream works on a word budget and cannot know how wide the words are, and WrapStyle is 2 -- no automatic wrapping -- so a line that still overflows was simply drawn past the edge. Each event's widest line is now measured against the real font file, and one that does not fit gets its own smaller size through an inline \fs override: the long line shrinks, not the whole video. Measured with the actual face rather than estimated per character, because the families involved differ in width by exactly the margin that decides the question. Clients scaling font_scale down by hand for 9:16 and 4:5 can stop. AN UNSET FIELD READ AS A FREE GPU. used_vram_gb=0 was honoured as authoritative -- yet zero is this estimator's own "could not determine" sentinel, and it is what the admin UI stores for a field the operator never filled in. A 27B Q4_K_P with used_vram_gb=0 and measured_vram_gb=18.0 therefore estimated 0.00 GB against 8852 MiB of free VRAM. Nothing was evicted, every layer was aimed at the GPU, the load failed and retried down a ladder -- all, 52, 39, 26 layers, re-reading the weights at each rung -- for 11m50s, and settled with 39 of 66 layers on the CPU. The request then generated for another 19 minutes, 31m04s in all, holding GPU ownership the whole time: every other model queued behind it, which is what made speech-to-text answer 500/503. The swap gate was working correctly; it swapped the moment the owner finished. A real measurement is better information than an unfilled field, so it now wins. An operator who means "no weights on the GPU" has n_gpu_layers=0, which is handled separately and still reserves nothing. Worth recording for the next time this looks like a quantization problem: it was not. Those weights are 4-bit on disk and the KV cache was already q4_0/q4_0 with flash attention -- about 2.2 GB for n_ctx=32000. Both were at their most aggressive setting while the model sat on the CPU. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Both of these were found by running the previous commit rather than trusting it, which is the only reason they are not still in. The audio backend was read from the manager's per-model config map. That map holds RUNTIME kwargs, and a whisper-server model has no row in it at all -- it lives in `whisper_servers` -- so the lookup answered "no backend" for exactly the models whose backend decides the question, and the pick that was supposed to avoid them chose `whisper2`. The entry is now read from models.json, where `backend` is actually written, matched on id, alias, path or basename because an audio model is referred to by any of them and the same alias appears twice in a real catalogue. Verified against this install's sixteen audio models: every one resolves, and the pick is now the model measured to return fourteen word timings rather than the one measured to return none. The new warning is what caught it: the render reported "'whisper2' returned no word timings" instead of quietly estimating. That is the whole point of it. And `enabled: false` was hidden behind `all_engines`, which is the wrong seam. The node-wide list the system worker serves -- the one the front hands clients -- MUST pass all_engines=True, because that worker has no engine of its own and would otherwise list nothing. So the filter was off on the only path that mattered, and /v1/models offered LTX-2.5, Krea-2-Turbo and Krea-2-Edit: three models configured ahead of a diffusers version that does not exist yet, each of which answers 500. `include_disabled` is now a separate question, asked only by the admin config view, because that is where a parked model gets re-enabled. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two faults seen on a long render, plus the image-side work from the same reports. Caption timings never worked. The render asks for word timestamps on the narration clip it just wrote, so it passes a NAME where the HTTP endpoint passes an upload -- and the transcription core read `.filename` off it unconditionally, raising `'str' object has no attribute 'filename'` on every scene of every render. The caller caught it and fell back to estimating from character counts, so captions drifted against the voice and nothing ever failed. `_upload_name` now takes a path, an upload or nothing; only the extension is read from it, so a name was always enough. Fixing that alone would have changed nothing visible. The model the renderer picked by default was chosen by NAME -- "whisper" first, then canary -- and both are the two backends that cannot time words: the whisper.cpp runner returns text with no word array, and the NeMo worker ignores the flag. The pick now asks which backends report them (the HF pipelines, vosk, faster-whisper), and a model that answers with none of them is named in the job's warnings instead of silently downgrading the captions. Polling a render cost a render. `GET /v1/video/compose/{id}` sits under the `/v1/video/` prefix, which allows 10 requests a minute per IP, so a client polling a four-minute render from the address it submits from spent the whole budget on the polls and got 429 on the work. Read-only job status -- compose and talking-head -- is now exempt from the limit, by prefix and for GET only, so the sibling POST .../cancel stays limited. It is NOT exempt from authentication: the document names the job's output, unlike the progress endpoints it sits beside. Also here, from the image-edit reports: a uniformly black or white result is refused with the dtype named rather than answered 200, `--no-ram` no longer coerces a configured bf16 to fp16 (same two bytes, a quarter of the range, NaN latents, black picture), single-file and GGUF checkpoints load through the configured loader instead of falling back to a different one, per-model LoRAs are an identity rather than a hint, and a repository may run its own pipeline code only from an allow-list that starts empty. And: the measurements stay, the customer names go -- shipped images are public. Co-Authored-By:
-
- Oct 25, 2026
-
-
Stefy Lanza (nextime / spora ) authored
A model id is a configuration, not a set of weights. Image models have worked that way for a long time -- one checkpoint, several published ids, one copy in VRAM -- and five things that should have followed from it did not. PER-MODEL LoRA STACKS. A diffusion pipeline could carry an acceleration LoRA and per-request LoRAs, with nothing in between: no way to say "this id is that base plus this style LoRA at 0.8". `loras` on a model entry says it, in the shape text models already use. `lora_path` is honoured too -- it had been in the config and the admin page for years while nothing on the diffusers image path read it. And `set_adapters` replaces the ACTIVE SET, so the image path's request-LoRA call was switching the pipeline's own adapters off: a 4-step distilled preset running un-distilled, and an id published as a fine-tune serving its base checkpoint. 200 OK, a plausible picture, nothing in the logs. The video path had always re-included them. A per-model LoRA that will not load now FAILS THE LOAD. The acceleration path logs and continues because the cost there is speed; here it is identity. SINGLE-FILE CHECKPOINTS. `from_single_file` was wired up only for GGUF components, so the one .safetensors Civitai ships for every community fine-tune could not be opened at all. A model path may now name a file, or `repo:file.safetensors` -- community repos hold every version side by side, so the repo id alone does not say which to serve. A SAMPLER IS A CLASS PLUS ITS OPTIONS. "DPM++ 2M SDE Karras" is DPMSolverMultistepScheduler and two settings; naming the class alone gave plain DPM++ 2M, a different sampler, silently. `scheduler_config` carries them. RASTERISATION, PER REQUEST. A scan is normally handed to the model as the scanner wrote it -- 0.1 ms and 147 KB against a render plus ~1.8 MB of PNG -- but only between the target DPI and `page_image_max_dpi_factor` times it, and only install-wide. `native_dpi` removes the ceiling for one request; the floor stays, because a page passed through BELOW the target is a coarser page, which is what cost surya-ocr-2 its Italian accents once already. An uploaded image now carries its own bytes too: it was decoded and re-encoded to arrive at the picture the caller already had. SEVERAL IDS FOR ONE OCR ENGINE. `ocr.engine_configs` gives OCR what image models had: named configurations of one engine, each its own id, its own catalogue row, its own pool. An override naming a setting that does not exist is refused rather than quietly serving the base settings under another name. Which forced the last one into the open. `/v1/ocr/engines` advertised `surya: {regions: true, tables: true}` while the surya worker built every page with `"regions": [], "tables": []` hardcoded. surya ships a layout predictor and a table recogniser; they were never run. A client that chose surya BECAUSE the catalogue said it returned regions got empty arrays and no error. Both halves are now true by construction: the row is read from `surya_layout` / `surya_table_rec`, and those load the predictors. Regions come back with surya's own reading order (LayoutBox.position, explicit as `order`, -1 where an engine states none) and tables as cells with row/column ids. Loading is configured; spending is per request. `layout=true` chooses whether to pay for the pass on an engine that HOLDS the predictor -- it cannot make one acquire a model mid-page, which is how a footprint grows past what the VRAM accounting reserved. Asked for where none is loaded, it is refused with the configuration to use instead; answering 200 with empty regions would be the same lie wearing a request field. Pods differ from their siblings too: a pool whose entry names an `ocr_engine_config` boots pods with that configuration's settings, and a named config overriding `<engine>_model_id` reaches its own pods rather than sharing its sibling's. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Requested for a client that edits photos by instruction -- "make the dress red", "add a necklace" -- which plain img2img cannot do at any strength, because `strength` is the fraction of the source it destroys before re-imagining it. Serving Qwen-Image-Edit / FLUX.1-Kontext class models turned up two latent bugs on the way. The instruction path. /v1/images/edits now shapes its call to what the pipeline accepts, from the signature, rather than branching on the model: QwenImageEditPlus / QwenImageEdit / FluxKontext image ✓ strength ✗ SDXL-Img2Img / QwenImageImg2Img / FluxImg2Img image ✓ strength ✓ Taking an `image` with no `strength` IS the distinction, and it generalises, so no list of class names has to be maintained. `strength` is dropped where it does not exist -- a caller cannot know which kind of model it is addressing, and these pipelines carry no **kwargs to swallow it, so passing it through would be a hard TypeError. `guidance_scale` becomes `true_cfg_scale`, the knob they actually read. Steps default to 4 at cfg 1.0 for them instead of inheriting img2img's 30 at 7.5, which would waste a minute an image and overcook the result. Up to three images may be given (`image` plus `images`) for a composed edit, conditioned at one shape; more is a 400. `instruction_edit` is reported in /v1/models so a client can hide the strength slider and treat the prompt as an instruction. Verified to classify the four Qwen/Kontext/FireRed ids and to leave Pony, Z-Image and Illustrious as img2img. component_models is new: a map from a component to ANOTHER repo, loaded and handed to the pipeline as a prebuilt object. Several Qwen-Image-Edit derivatives ship a transformer and nothing else -- the base supplies the VAE, the text encoder and the scheduler -- and there was previously no way to express that, so they could not be served at all. Its entry in component_quantization still applies, so a transformer too large for the card is quantized as it loads. Two latent bugs, both of which anyone using GGUF image components would have hit: * `gguf` was never a declared dependency. build_gguf_pipeline_components has existed through several releases and could not have worked: diffusers needs gguf>=0.10.0 for GGUFQuantizationConfig and only says so mid-load, as "Unable to load weights from checkpoint file". Now in requirements.txt, which build.sh installs. * from_single_file was called with no `config`, so it INFERRED the architecture -- and its fallback is Stable Diffusion 1.5. A Qwen-Image, Flux or Wan transformer therefore failed with "stable-diffusion-v1-5 does not appear to have a file named config.json". It now passes the base pipeline's own component subfolder, which was already at hand. Measured on an RTX 3090 after the fixes: the Q4_K_M transformer binds as QwenImageTransformer2DModel (13.23B params, 85s) and the assembled QwenImageEditPlusPipeline is 22.68 GiB resident with a 4-bit text encoder -- which fits 24 GB but leaves 1.32 GiB and OOMs on 390 MiB of activations. That is a deployment setting, not a code one, and it is why the figure is measured here rather than asserted. Full suite 4558 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reported from a client against nesaorg/PonyRealism-Uncensored as three symptoms -- "the edit result is garbage", "size is ignored", and a 500. Two defects, neither of which announced itself. The edit was never an edit. _load_img2img_pipeline picked its class by looking for 'xl', 'sdxl', 'stable-diffusion' or 'sd' in the MODEL NAME, and "nesaorg/ponyrealism-uncensored" contains none of them, so it fell through to the generic DiffusionPipeline -- which faithfully honours the checkpoint's model_index.json, and that one declares StableDiffusionXLPipeline: the text-to-image class. Every diffusers __call__ ends in **kwargs (it carries the deprecated callback pair), so `image` and `strength` were accepted, dropped, and the pipeline rendered a fresh picture from the prompt alone at its default 1024x1024. One defect, all three symptoms: garbage (a prompt saying "add X, keep everything else" rendered from scratch), size ignored (1024x1024 is the text-to-image default, not the source's shape), and 200 OK throughout. stable-diffusion-xl-base-1.0 worked only because its NAME contains "xl". Both loaders now ask AutoPipelineForImage2Image / AutoPipelineForInpainting, which read the declared class and substitute the one for this task. A guard backs it up, on cached pipelines as well as freshly loaded ones -- an install already holding a text-to-image pipeline would otherwise keep using it until it restarted. A pipeline with no `image` input is converted with from_pipe (the same components, no second copy in VRAM) or refused with 501. A 200 carrying a picture of something else is the one outcome worth ruling out. The 500 was a cache key used as a repo id. These endpoints keep task-specific pipelines in MultiModelManager.models under "img2img:<repo>", and the manager's fuzzy already-loaded pass matched one by its last path segment, then assigned the whole key to resolved_name -- so from_pretrained was asked for a repo called "img2img:nesaorg/PonyRealism-Uncensored" and answered "Repo id must use alphanumeric chars" about a model that loads perfectly well. That pass now skips task namespaces, which the endpoints look up by exact key anyway. Writing the test for that found two more namespaces with the identical latent bug -- depth: and segment: -- so _TASK_NAMESPACES covers four, and an invariant test asserts every task prefix the endpoints invent is declared. A future endpoint cannot reintroduce this quietly. Not fixed, because it is not a bug: /v1/images/edits is plain img2img, so "add X and keep everything else exactly the same" cannot work however correct the pipeline is. That needs a mask through /v1/images/inpaint, or an instruction-edit model exposed with a capability clients can prefer. 16 tests; full suite 4537 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
clip_skip was mapped onto diffusers' argument of the same name, and that argument counts something else. Its SDXL path reads hidden_states[-2] # clip_skip is None hidden_states[-(clip_skip + 2)] # otherwise so diffusers' clip_skip is the number of layers skipped BEYOND the penultimate one SDXL already uses. Every model card, A1111, ComfyUI and InvokeAI mean something simpler by the name: 1 is the final layer, 2 the penultimate, n the n-th from the end. So a request asking for 2 -- which is what the booru-tag SDXL families are trained at, and what their model cards all say -- read hidden_states[-4] instead: two layers too deep, the equivalent of clip-skip 4, which on a Pony-family model is the difference between a photograph and mush. It was inconsistent as well as wrong: the same 2 meant [-4] on SDXL and [-3] on SD 1.x, two depths from one number. The mapping now happens here, once, in the sense users mean, and clip_skip 2 reaches the same layer whichever family the model belongs to. "Mirrors diffusers exactly" was the mistake in 0.4.89: mirroring is the wrong goal for a field a caller fills in. SDXL's pooled conditioning vector is also handled properly now. The UNet cannot run without it and diffusers refuses prompt_embeds that arrive without one, so handing it half the set is a ValueError raised in check_inputs -- an instant 500, before any denoising. Where a pipeline does not offer the vector, it is rebuilt the way CLIP produces it (pooled at the end-of-text token, then projected through text_projection) rather than giving up the weighting and the window count this module exists for; only when even that is impossible does the encoder stand down and let the pipeline encode the prompt itself. The invalid combination is no longer constructible: either all four keys or None. A 500 from /v1/images/edits, /inpaint or /outfit now logs its traceback. The cause used to reach the caller alone, so the log showed the request and then the status, and a failure there could not be diagnosed afterwards at all. Test note: test_a_400_is_not_relabelled_as_a_server_fault scanned a fixed 250-character window for the HTTPException pass-through above each catch-all; the new log line pushed that to 316-382 and broke a passing check on correct code. It now asserts structure instead of proximity -- the pass-through must be the nearest handler inside the same function, with the logging between it and the raise -- which also catches it drifting into a different function. 58 tests in the prompt file; full suite 4521 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three things the image endpoints accepted and then quietly did something else with. size. A size that did not parse fell through to the default, so "1024" or "1k x 1k" generated a square nobody asked for and the response looked like a success. A side that was not a multiple of 8 went to the pipeline as-is, where the VAE cannot represent it. Nothing capped it either, so 4096x4096 was accepted and came back as a CUDA OOM -- which reads like a busy GPU rather than a size the caller chose. Now: a 400 quoting what was sent, a 400 for a zero dimension (snapping it up would answer an uninitialised client variable with an 8x8 image), snapping to the grid with a log line, and a 400 naming image.max_side / image.max_pixels when it is over the ceiling. "auto" is accepted as in the OpenAI API, and the dead `width != width` NaN guard on a value that was already an int is gone. The archive records the size actually generated alongside the one requested. /v1/images/inpaint no longer defaults to 1024x1024, which squashed every non-square photo and pulled the mask out of alignment with it -- worse than the same default on /v1/images/edits, where it was fixed in 0.4.86. Omitted now keeps the source's shape. Textual inversions. easynegative, bad-hands-5 and the rest are vectors living in CLIP's embedding table under a word of their own, and until the file is loaded that word is an ordinary unknown token -- so a large part of the ecosystem's negative prompting asked for nothing at all. They load from the request (embeddings: [...]), from the model's config (textual_inversion), or by filename from image.embeddings_dir, which is how A1111 works and what a prompt copied off a model card assumes. Two details decide whether this works at all: an SDXL-format embedding holds a clip_l and a clip_g tensor that must go into the matching tokenizer/encoder pair, and a multi-vector embedding occupies several tokens (tok, tok_1, tok_2...) that the prompt has to be expanded into -- without it an eight-vector embedding contributes one vector and appears inert. A name the caller gave that will not load is a 400; a bad file in the directory is a line in the log, because one file should not break every request. /v1/images/outfit goes through the same encoder now, so the weight grammar and clip_skip reach it too. Its per-frame video path encodes the prompt once for the clip instead of once per frame. And a 400 is no longer reported as a failed backend. On the first request for a model that is not loaded yet, the generate call sits inside the try that falls back from diffusers to sd.cpp, so a rejected request was caught as a load failure and answered with a report about which backends failed. The three edit endpoints did the same through their catch-alls. All four re-raise HTTPException untouched. Found by the tests while writing them: the embeddings directory was searched by the renamed trigger word rather than the filename, so "easynegative:myneg" never resolved, and "768X512" did not parse. 64 new tests; full suite 4475 passed with the same 15 pre-existing failures as 0.4.89 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A CLIP text encoder reads 77 tokens, two of them markers. Handing a longer prompt to a diffusers pipeline does not fail: the tokenizer truncates it and the model renders what fits. The booru-tag fine-tune families (Pony, Illustrious, NoobAI) open with a run of quality tags, so the part that got dropped was the subject -- and the request answered 200 with an image, which is why it was easy to miss. Image prompts are now encoded here rather than by the pipeline, which is what makes the rest possible: * as many 75-token windows as the prompt needs, concatenated -- no limit * the A1111 weight grammar: (tag:1.3), [tag], bare (tag) = 1.1, nesting multiplies, \( escapes. Per-token scaling with the window mean restored, so emphasis does not drag the brightness along with it * BREAK, which closes the current window instead of blending tags meant for different subjects * clip_skip, mapped to diffusers' own convention -- SDXL counts from the penultimate layer, SD 1.x from the final output and needs the layer norm applied by hand. Getting that backwards shifts every image Three places were dropping the negative prompt outright. The sd.cpp path hardcoded negative_prompt='', so for every GGUF image model whatever the caller asked to avoid stopped at the API -- half the idiom, since the quality tags have a negative half (score_4, score_5, score_6). /v1/images/edits and /v1/images/inpaint had no field for one at all, so an edit could only ever be steered positively. All three now pass it, and take clip_skip and prompt_weighting with it. clip_skip and prompt_weighting are also per-model and server-wide settings, because a client speaking the plain OpenAI image API has no field to ask for clip_skip and a model trained at 2 needs it on every request. Request beats model beats global. Weighting defaults on, so a literal '(' in prose now weighs 1.1 -- the ecosystem's convention, with \( and prompt_weighting:false as the way out. What that cannot be allowed to change is prompts that already worked: against a real CLIPTextModel a short unweighted prompt encodes bit-identically to what diffusers produced, because CLIP pads with its own end-of-text token, so [bos]+toks+[eos]+pad... and [bos]+toks+pad...+[eos] are the same ids. The test records that reasoning, so a tokenizer where it stops holding is caught. Flux, SD3 and Z-Image carry a T5 with its own limit and conventions and keep their pipeline's native text path. 49 tests; full suite 4410 passed with the same 15 pre-existing failures as 0.4.88 (diff empty). Co-Authored-By:
-
- Oct 23, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Measured against the production pod: a reasoning model asked for 64 tokens spent all 64 inside its thought, returned NO content, and the response said finish_reason "stop". max_tokens=64 finish=stop content=0 chars '' reasoning=262 max_tokens=1500 finish=stop content=13 chars 'XMU-5635-FB' reasoning=448 Two layers were each sure the other knew better. The proxy hardcoded finish_reason = "stop" and never read choices[0].finish_reason from the remote; OpenAIFormatter then hardcoded it again, so even a caller that worked the reason out had it overwritten on the way to the client. vLLM had been saying "length" all along. It matters because "length" is the one reason a client can act on. An answer cut off at the limit is half a sentence, and told it stopped normally the client has nothing to retry on and no sign anything is missing. On a reasoning model the thought is generated FIRST and spends the same budget, so a tight limit returns an EMPTY answer that claims to have finished of its own accord -- which is how this surfaced, as a retrieval test scoring zero for reasons that had nothing to do with what it was measuring. The remote's word is now carried through the per-call sink beside reasoning, tool_calls and logprobs (get_last_finish_reason, buffered and streamed), the formatter accepts the caller's reason and keeps "stop" only as the default, and a tool call still wins -- that describes the answer's shape, and a truncation after the call was formed must not relabel it. Fixed at all five response exits, not just the one it was found at: streamed chat (both terminal-chunk paths), buffered chat, and streamed and buffered /v1/completions. A local backend is never told a reason, so the budget decides: generation that used every token it was allowed was cut off, not finished. Drawn only when a limit was actually set and reached -- never from a token count alone, or every long completion would be mislabelled. Tests verified against the reverted files: assert None == 'length' for the backends, TypeError: unexpected keyword 'finish_reason' for the formatter. Co-Authored-By:
-
- Oct 22, 2026
-
-
Stefy Lanza (nextime / spora ) authored
v0.4.67 asked each pod for /admin/api/gpu-stats. Pods answer 404: a pod runs the SLIM app — /healthz, /v1/models, /internal/engine-state — and no admin API at all. Built on an assumption; the live check caught it. Probe the pod before choosing its endpoint. /internal/engine-state already carries a `vram` block: used, free and total per card with the GPU's name, built context-free and cached 4s there. It is what the front health-polls, so every pod has it and it costs the pod nothing extra. Verified against a live A40: 43.8/48 GB with qwen38 resident. The pod reports FREE, not used — that is what a context-free query returns — so used is the difference. Inverting that would show an idle pod as full. Utilisation and temperature added AT THE SOURCE: nvidia-smi now asks for utilization.gpu and temperature.gpu in the same call as the memory query rather than paying for a second one (a saturated card answers slowly enough that one call matters), with the pynvml equivalent, and both ride into engine-state's vram.devices. Absent or "[N/A]" is left out, never zeroed: a zero draws a bar that reads as "idle" when the truth is "not known". So VRAM works on every pod already running; utilisation needs the next pod image, and until then that column shows a dash — which is the degradation the page was built for. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`enable_thinking: false` came back still thinking, verified on a pod running the v0.4.65 image with --reasoning-parser qwen3 (the other four checks passed: reasoning_content populated, content clean, and streamed thinking starting 0.4s in against the answer at 2.8s). generate_chat negotiates optional kwargs by calling the backend and retrying on TypeError. Every variant that carried `passthrough` ALSO carried `enable_thinking`, which a proxy backend does not accept -- so those variants raised TypeError, and the variant that survived had dropped the passthrough with it. A caller's `chat_template_kwargs` therefore never reached a pod at all. `enable_thinking` was merely the visible case: OCR's `continue_final_message` would have been dropped on a pod the same way. `_lp` and `_pt` are already SIGNATURE-gated -- empty for a backend that cannot take them, so they never raise TypeError. Only enable_thinking and the penalties are negotiated that way, so only those are peeled off now, and the passthrough is dropped last rather than first. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
v0.4.64 aimed one layer too high. Measured against the pod directly: content = 'We need answer in Italian... 17 x 23 = ... **391**.' reasoning_content = '' has </think> = False finish = stop The thought and the answer come back as ONE string with no separator of any kind. </think> is a SPECIAL TOKEN and vLLM decodes with skip_special_tokens, so the boundary is deleted before the text exists. No parsing downstream can split on a boundary that was removed -- the earlier reading, "extraction fails on the bare </think>", was simply wrong. coderai never passed --reasoning-parser to vLLM, for any model, ever. With it vLLM splits the two at the TOKEN level, before the tag is discarded, and reports reasoning_content as its own field; the v0.4.64 side channel and delta re-tagging then carry it to the client. Chosen from the template like the tool parser, and only for a model whose template says it thinks -- a reasoning parser on a model that does not think waits for a tag that is never coming. An explicit vllm.reasoning_parser wins, "off" included, per model or globally. AND A REGRESSION FROM v0.4.64, fixed here: with the gate switched on and no close tag ever arriving, a streamed reply was held back in full as "thinking" and the client was sent an empty message -- worse than the leak it replaced (reasoning=229 chars, content=0, measured on diogene-alpha and qwen38 alike). An unclosed thought now flushes as CONTENT. A thought truly cut off mid sentence then reads as the answer, which is untidy; an empty answer is a lost reply. With the server separating the channel this path stops being reached. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
From ONE pod, ONE set of weights, behind two aliases: qwen38 -> clean answer diogene-alpha -> "The user is asking to list hearings from 2026-10-21 to..." The entire chain of thought went to the client as the answer. Four breaks: 1. THE GATE ASKED THE NAME. _reasoning_active was a regex (qwen3|qwq| deepseek-r1|...) over the request's model name: `qwen3` matches "qwen38" and not "diogene-alpha", so one alias of one model separated its thought and the other did not. It is read from the template now -- whose generation prompt ends with '<think>\n', so the model starts INSIDE the thought and emits a bare </think>, the form that leaks when nothing looks for it. The regex stays as a last resort for a template that cannot be read. 2. THE BUFFERED FORMATTER TOOK NO `reasoning`. Where the thought WAS separated it was then dropped: no leak, but no thinking either. litellm's Message has no such field, so it is attached after the dump rather than lost -- under both spellings, since Qwen's own template reads `reasoning_content`. 3. THE PROXY HOP CARRIED NEITHER FORM. Buffered, the remote sends reasoning_content beside an already-cleaned content, which the string contract cannot hold: get_last_reasoning(), like get_last_tool_calls(). Streamed, it sends reasoning DELTAS, and dropping them meant a client saw nothing at all until the answer began -- so the <think>/</think> tags go back on as they arrive and the caller's existing reasoning gate splits them live. An all-thinking stream leaves the tag open on purpose: the gate flushes an unclosed thought as reasoning, and there is no content for a close tag to precede. 4. THINKING COULD NOT BE TURNED OFF. enable_thinking defaulted to False, so "unset" and "explicitly off" were the same value, and the field sat in _CHAT_CONSUMED_FIELDS while no backend forwarded it. It defaults to None now: unset sends nothing and the model keeps its own default (a thinking model thinks), an explicit value travels as chat_template_kwargs. That needed the POD backend to gain passthrough at all -- the local vLLM backend has forwarded unknown fields for releases, a pod got none of them, so chat_template_kwargs could never reach a pod-served model. ChatMessage silently dropped `reasoning_content`, so a previous turn's thinking never reached the template that re-renders it inside <think>. Accepted and forwarded now. The chat page hardcoded stream:false, so thinking could never arrive live there, and it read only message.content. It has `stream` and `thinking` toggles (both on), renders the thought in a block that fills AS IT ARRIVES and collapses when the answer starts, and keeps it in the history it replays. Streaming is the request's choice, not forced: cleared, it takes the buffered path exactly as before. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The admin model dialog has had a "Chat template parser" dropdown all along (auto/llama/mistral/chatml/phi/gemma/qwen). It is saved on the model entry, and seed_model_env carries `parser` in its keep-list so it travels to rented pods. NOTHING EVER READ IT: choosing a parser for a model changed nothing anywhere, and selection stayed a substring match on the name -- with a second, separate guess in the browser (_autoDetectParser). A tool call is text in the format the model was trained on, and the chat template is where that format is written down, so the style belongs to the model. It now resolves, strongest first: 1. the model's own `parser` setting -- that dropdown now does something; 2. the global default, read the way resolve_gemma_tool_mode reads its own; 3. the model's chat template; 4. the name. The template reading moves out of vllm_worker into codai/models/chat_style.py, with ONE style table mapping each style to both consumers -- vLLM's --tool-call-parser and coderai's own parser class -- so the two choices cannot drift apart. Qwen XML, Hermes JSON, GLM <arg_key>, Mistral [TOOL_CALLS], Llama <|python_tag|> and DeepSeek's call tokens. Cached per model path: a model's template cannot change under us, and this is consulted per request on some paths. The template does NOT override a name that matched a family. A per-family parser carries more than the tool syntax -- gemma's token repair, Qwen's repetition guard -- so one template marker should not displace a family that matched. It decides where the name places the model in no family, which fell to the catch-all before: _family_name only resolves an alias for an ADAPTER, so a plain alias like `lisa` named no family and got the generic parser even though its entry says exactly where the weights are. A value the dispatcher does not know falls back to detection rather than silently disabling tool calls, and an unreadable template changes nothing -- not knowing is survivable, inventing a parser is not. Co-Authored-By:
-
- Oct 21, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Every tool-using request to a pod-served model answered an EMPTY assistant message: content "", finish_reason "stop", no tool_calls. Asked directly the pod answered the call it had generated; through the orchestrator the same request answered nothing. Both the LoRA (diogene-alpha) and the base model were affected, so no adapter was involved. A proxy backend's generate_chat returns a STRING. The remote server owns the chat template, so it is the side that recognises a tool call and reports it structurally -- `{"content": null, "tool_calls": [...]}`. `message.content or ""` dropped that on the floor; the caller then had an empty string, found no call to parse in it, and answered the client with an empty message. The calls now travel beside the text, where usage and logprobs already do: get_last_tool_calls() on the backend, forwarded by the manager, preferred over re-parsing text in both the streamed and buffered chat paths. A backend that generates locally still keeps its call inside the text and answers None, so that path is untouched. Streaming needed the same: a streamed call arrives as deltas with no content at all, so merge_tool_call_deltas folds the fragments -- keyed by the call's index, never by arrival order, since concatenating the arguments of the wrong call yields valid JSON for a call nobody made. Co-Authored-By:
-
- Oct 20, 2026
-
-
Stefy Lanza (nextime / spora ) authored
2026-10-20, qwen38: a restart generated a fresh random pool token (none is configured), adopted a pod still carrying the old one, and rented a second pod with the new one. A pool has one api_key, so whichever token it held answered 401 on the other pod — "Invalid API key" on every request until both pods were destroyed. The adoption guard ("only swap while we hold no pod") could not help: the rent and the adoption happened in the same minute. - PodHandle.api_key: the token THIS pod was launched with, or adopted carrying. pool.pod_key(pod) resolves it, and the pool remembers the tokens of pods it no longer holds so a quarantined one stays reachable. - Adoption keeps the adopted pod's token on that pod instead of replacing the pool's, so pods with different tokens coexist. - _acquire_endpoint returns (base, release, bearer): the key is handed out WITH the address. Reading it afterwards from the pool was wrong twice — the pool's token could change mid-acquire, and _last_pod is one attribute shared by every concurrent request, so it could name another request's pod. - The OCR worker's majority vote over the registry's keys is gone; each pod's key rides on its own handle. - The pool's own token is minted once and persisted beside the pod registry (0600), so a restart no longer orphans every pod it had rented. A configured api_key still wins, and an unwritable directory still serves. Also tried and reverted: treating an orphaned mid-boot pod as "on the way" to stop a restart renting beside it. That reintroduces the twenty-minute freeze of test_runpod_restart_orphans. Renting twice costs cents; a pool that will not rent serves nothing. The reasoning is recorded in sibling_is_provisioning. Co-Authored-By:
-
- Oct 17, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Third run, third failure, and this one was mine. A 5,000-document run dispatched ZERO pages to 50 healthy pods in 41 minutes: not one `POST /v1/ocr` in any pod's log, every GPU at `KV 0.0%`, 811 pages waiting, and the scaler reading that unservable queue as demand and renting to 73 pods at $43/hr. `/healthz` answered in 1.6 ms the whole time, so the loop fixes from 0.4.16/0.4.17 were holding and nothing looked wrong from outside. 0.4.18 bounded a lease attempt at 20 s and ran it on a 256-thread pool. That made the LEASE STAGE the ceiling on everything: 256 threads at one attempt per 20 s passes ~13 pages a second however many pods are up, and a thread that drew the growth branch sat through an entire ~9-minute boot. Against 811 pages each retrying every couple of seconds, nothing got through. So `acquire(grow_inline=False)`: answer now, or start growth in the background and raise `NoPodRightNow`. The caller waits on its event loop, where waiting is free, and `POD_LEASE_ATTEMPT_S` drops 20 s -> 2 s for the pools whose acquire predates the flag. Threaded through `placement.choose(grow_inline=)` by signature inspection, so a candidate that cannot take it keeps working. Demand had to survive that. `_waiting` only ever counted THREADS parked in acquire, and the non-blocking path has none, so the pool would have stopped growing. It now counts DENIALS over a window equal to the caller's retry interval, which makes the count distinct waiting PAGES rather than the attempts they have made between them -- a page retrying every 2 s appears once in any 2-second window. A longer window multiplies demand by the retry rate, which is how the same run read its own retries as new work. Two more things found on the way: * `pod_weight`/`pod_capacity`/`pod_threshold` depend only on a card's NAME, but were recomputed per POD per acquire -- `_pick` once per free pod, `_pods_for_demand` once per serving pod, `_preferred_vram_gb` walking every pod inside each -- all under the pool lock. Memoised per card: measured 286us -> 69us per acquire at 70 pods, 528us -> 102us at 128. The memo self-initialises via getattr, because a pool built without __init__ must not have an optimisation turn into an AttributeError. * `_pick` returned a bare None for four different causes -- no pods, none healthy, all draining, all at their in-flight cap -- which need opposite responses, and the caller could only report the queue it could not drain. It now says which, once per 15 s, with every count in the line. I also mis-read `futex_do_wait` on 324 of 352 threads as lock contention. A condition-variable wait IS a futex wait, so that is what pages legitimately waiting for a pod look like. The pod logs settled it instead: zero POSTs means the dispatch never happened, not that it was slow. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A 5,000-document run rented 127 pods, reached $79/hr and completed zero documents in ten minutes. The front had stopped answering entirely: /v1/runpod/status timed out on every port, which is also what the client had been reporting as thousands of timeouts. The pods were fine. Probed from inside the container, both transports answered /healthz in milliseconds, vLLM was loaded on each one, and the GPUs sat at KV 0.0% with no surya worker line, because no page had ever reached them. `placement.choose` is synchronous and ends in a pool's `acquire()`, which waits on a condition variable for up to its default 1200 s and may rent a pod INLINE -- a candidate sweep over every region, then a boot wait, in whichever thread asked. It was called straight from `_read_on_pods`'s coroutine, so it ran on the event loop and took the whole process with it for as long as one page waited for a pod. It has been there since the pod pipeline was written (0.3.70) and cost nothing while acquire() returned at once: a pool with a free pod answers in microseconds. It turns fatal exactly when the pool is the thing under strain, which is when it matters most. So the lease runs on the pod pool now, and because a pod takes MINUTES to boot, three things follow: * One attempt is bounded (`POD_LEASE_ATTEMPT_S`, 20 s), threaded through `placement.choose(acquire_timeout=)` to `pool.acquire(timeout=)`. An unbounded attempt parks a thread for a whole boot, per page; bounded, the waiting happens in `_lease_a_pod`'s own loop where it is free. * `ocr.pod_pipeline_wait_s` goes 60 s -> 600 s. The 60 s was chosen because a waiting page held a thread, so a long ceiling meant thousands of parked threads. It no longer does, and at 60 s a cold start handed its whole first wave of pages to a GPU-less orchestrator to rasterise -- the exact failure `prefer` exists to prevent. * The per-page PDF split moves to the DEFAULT executor, not the pod pool: pdfium permits one caller at a time, and putting a split behind thousands of leases waiting for a boot is a cold start that cannot produce the next page while it waits for the pods that page is for. The forward itself stays on the event loop, which is the point of 0.4.8 and is now covered by a test that fails if it moves back. All of this is invisible to a test that awaits one page at a time: one page blocking the loop for 0.5 s looks exactly like one page taking 0.5 s. The new tests measure the longest gap between two ticks of a coroutine running alongside, and the thread-seconds held while waiting out a boot. Five of them fail on 0.4.15. Co-Authored-By:
-
- Oct 16, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Three things, and the third is the reason the first two were impossible. **A mixed fleet was distributed as if every pod were the same.** `PodPool._pick` was `min(free, key=inflight)` and growth was `load / scale_up_inflight_per_pod` -- both card-blind. An A100 and an A40 were handed the same number of pages, so the fast card drained its share and idled while the slow one straggled; and a document completes only when its LAST page lands, so the slowest pod set the latency the client saw. The scaler had the matching fault in reverse: a fleet of bigger cards was still sized as if each were the preferred one, which is the same arithmetic that reached 94 pods at 10 % utilisation. Now: weighted least-connections, an in-flight cap scaled per pod, and growth that divides demand by the fleet's real capacity. **`gpu_limits`: per-card numbers, nothing multiplied.** "gpu_limits": { "NVIDIA A100-SXM4-80GB": {"parallel": 186, "queue": 744, "scale_up": 93}, "NVIDIA A40": {"parallel": 64, "queue": 256, "scale_up": 32} } A scaled single number cannot stay right: the model-level figures describe ONE card, and which card that was came from the first entry of `gpu_types` -- so changing the RENTAL preference silently re-scaled the concurrency, and a figure measured on an A40 came to describe an A100 by a factor nobody had measured. A table states what was measured, per card. The share of load is DERIVED from it (two cards with depths 32 and 93 are given work in that ratio) rather than configured twice, with `gpu_weights` and then the VRAM ratio as fallbacks for cards nobody has measured yet. Editable per model in the admin UI; a row with a card name and no numbers is dropped, because "not measured yet" must not be saved as a limit of zero. `parallel`/`queue` are applied AFTER the candidate card is chosen. The launch plan is built before, so its env carried the model-level number whatever card won -- and the gate is env, set once, so a pod rented on a 3x card spent its whole life admitting a 1x card's work. **An adopted pod recorded `gpu="(adopted)"`.** A pool that cannot name a pod's card cannot size it, weight it, or report the fleet: a whole day's status output read `by_gpu: {"(adopted)": 32}`. `list_pods`/`get_pod` now ask RunPod for `gpuCount` and `machine { gpuDisplayName gpuTypeId gpuType { memoryInGb } }` (every field probed against the live schema first), the registry stores the card and its size so adoption needs no API call, and an unreadable card costs a weight of 1.0, never a pod. **And a catalogue entry could crash an engine.** Adding a `vision_models` row for a model served on RunPod quarantined the cpu engine for ninety minutes: its id is not a path, not a URL and not a repo id, so resolution fell through to "assume a repo id", then to "assume a URL", and `requests.get("qwen38-vision")` raised MissingSchema -- five exits in 120 s, QUARANTINED, taking /v1/ocr and every pool with it while `/healthz` still answered `ok: true` because the FRONT was up. Two independent guards: * a model pinned to a backend that serves it ELSEWHERE is never resolved, downloaded or cached here -- which is also what lets one row run on rented pods and load locally on a box with a GPU, the pin being the whole difference and the section not overriding it; * a name that is none of the three resolvable kinds is refused with a sentence naming all three and pointing at `backend`. The pin does NOT make local weights useless: `resolve_model_source` is configurable (`auto | hf | url | upload`) and reads the entry directly, so an id still means "the pod downloads it" and a local path still means "push it". Tested, so a later change cannot couple the two. 43 new tests. 3411 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A rented pod answered, for a model it was serving perfectly well over its own loopback: model 'datalab-to/surya-ocr-2' is not served over HTTP by this install Not rasterisation: a PNG, which skips rasterising entirely, failed the same way in 0.26s. The pipeline on the pod needs a LOCAL placement candidate, where the pipeline on the orchestrator needs a POD one -- so the local path is the one that had never run. Two defects, and the second is why the first took all night. 1. `_model_entry_for` reads the catalogue from one place: the ConfigManager the admin module holds. A pod has no models.json -- its catalogue is seeded from CODERAI_SEED_MODELS into memory -- so any process or import order where that global is not that object sees an EMPTY catalogue. The seeded model then has no backend pin, resolves to no engine, yields no candidate, and the pod disowns a model it is serving. It now falls back to the seed list in the environment, which needs no global and no import order. A real catalogue entry still wins. 2. `_local_candidate` ended in `except Exception: return None`, and `_pod_candidate` too. Every decline now records a reason -- which of the catalogue entry, the backend pin or the enabled flag was missing, or the exception -- says it once, and clears it when the install starts working. And the reason travels IN THE ERROR, not only in a log. The machine that reads a pod's 503 is not the machine that can read its log; a pod's log is not reachable from the orchestrator at all. Blaming configuration for what may be a lookup, in the one sentence that does cross the network, is what made this invisible. Co-Authored-By:
-
- Oct 15, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Five fixes, two of them reported from production in the same hour. EMPTY REPLIES from a chat naming `diogene-alpha`, while `qwen38` — the same weights on the same pod — answered normally. Parser selection reads the model NAME: if 'qwen' in model_lower: -> QwenParser `qwen38-awq` matches; `diogene-alpha` matches nothing and fell to the generic parser, so the model's <think>…</think> block was never separated from the answer. With a modest max_tokens the whole budget goes into reasoning that never closes, and the caller sees an empty reply from a 200 — the hardest kind of wrong to see. A LoRA is named for what it was trained to do; it is the same FAMILY as the weights it adapts, so a name matching nothing now resolves to the base's. Nothing is discarded: the thinking comes back as `reasoning` and `reasoning_content`, as it always did for the base. This also corrects a reading of my own: every temperature-0 comparison in this chain showed the adapter leaking its reasoning while the base answered cleanly, and I took that as evidence the adapter was not loading. It was the parser. /v1/chunk with X-Coderai-Placement: remote answered 502 "nothing configures 'embeddings' to run remotely" while a coderai-embeddings pod — which has the chunker in it — sat there serving bge-m3. A capability-only request names no model, so the per-model placement that moved everything else cannot place it. It now borrows the destination of a model OF THE SAME CAPABILITY that is already remote. Same capability only (an embeddings pod has no OCR deps), and only on an explicit `remote`: nothing should rent a GPU to run a regex split because nobody said otherwise. Registering an adapter is not serving it. vLLM takes its adapters at launch and the code that adds one to a live engine sits on the model-LOAD path, which does not run again for a model this instance already loaded. So a pod could be sent an adapter, register it, report it in /v1/loras and go on answering from the base. Registration now hands it to the running engine, swapping a stale one. A stale content id repairs itself. v0.3.94 fixed what new uploads are stored as; an entry already registered under a non-canonical id stayed broken with no symptom but base-model answers. Seeding and pushing resolve an id against the CONTENT first. And `it-legal` now sets min_chunk_chars: 300 (strict stays 0) — it is the OCR-aware template, so it meets short trailing sections more often, and a 56-character chunk is too small to carry meaning and long enough to match. A runt folds only into a chunk that can CARRY it, so a document of short sections stays several chunks instead of collapsing. Absorption may never cross conclusioni <-> dispositivo at any size: what a party asked for and what the court decided, and a recorded label is a weaker guarantee than refusing the merge. NOTE: this release MOVES CHUNK BOUNDARIES for anyone using `it-legal`. Co-Authored-By:
-
- Oct 13, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Reported from the Digesta side as "the pinned pod image boots, but coderai calls its vLLM endpoint with a token the pod rejects — every page 401, every page empty". The 401 was real and the pages were really lost. It was not the image, and not a mismatch between what coderai holds and what the pod wants — the pod answers 200 to its recorded token, before and after. It was the ORDER of two reads. `RunpodBackend.load_model` snapshotted `pool.api_key` into `self._headers`, and `_acquire_endpoint()` — which is what triggers provisioning, and therefore what runs `_adopt_shared_pod` and REPLACES `pool.api_key` — runs after it. A pod for this pool was already running (ours, from before a restart), so the pool adopted its token rather than renting a second A40: 22:27:15 [runpod] adopting pod sncrmkqdnd7c57's bearer token 22:27:15 [runpod] reusing the pod already running … instead of renting another 22:27:15 Inference error: Error code: 401 - Invalid API key. Every request that crossed that moment carried the token from before it. All 64 pages in flight failed; the same lease answered 200 minutes later, which is why it looked like a property of the run rather than of one instant in it. `_auth()` now reads the pool at send time, and the three sites that used the snapshot go through it (chat completions, streaming, openai_endpoint). Serverless keeps the snapshot: it has no pool and no adoption event. The placement candidate also reads the key AFTER acquiring rather than before, since acquiring is the thing that can change it. Behind that, the OCR gateway now retries a 401/403 once against a freshly chosen upstream. A pod only answers to the token it was launched with, so a stale credential is not recoverable downstream — and throwing a page away for a credential that was already correct by the time the error arrived is the expensive half of this bug. A second refusal is returned as itself, not retried forever. Also fixes a regression in 0.3.56: `register_pod` re-stamped `created` on every write, but an adoption re-registers an existing pod — so a pod adopted hourly looked newly born hourly, and the reaper's ledger attribution (which 0.3.56 computes from `created`) would have under-counted its hours. An existing `created` is now preserved. 9 tests in tests/test_pod_token_adoption_race.py, including the ordering pinned against the source so no send site can quietly go back to the snapshot. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/models now carries `max_model_len`, under vLLM's own name so a client written against a vLLM server reads it unchanged. It is published because a client cannot otherwise plan: anything that sends a growing conversation -- a continued generation, a resumed OCR page, a long tool loop -- has to know when the window runs out, and the only alternative is finding out from a 400 after the work is paid for. coderai's own OCR page resumption was the first casualty, and the way it failed is the point. The 0.3.50 clamp probed /v1/models for this field; coderai did not publish it, and the surya worker talks to coderai rather than to raw vLLM. So the clamp was INERT from the day it shipped, and said nothing: the log printed "? of the context window left" and every resumption ran with no ceiling. Two further details made it invisible -- `max_model_len` is not in the OpenAI model schema, so the client parks it as an EXTRA field and reading attributes alone found nothing; and the probe only logged on an exception, so a server that answered WITHOUT the field was indistinguishable from one that worked. - The number comes from the LOADED backend first, config second. The two disagree and only the served one is true: `ctx: 18432` is configured on one install while `--max-model-len 32768` is what the pods actually serve, so a client trusting config would plan around a window that does not exist. - VllmBackend.context_length() asks its own upstream /v1/models, cached: a served model cannot change shape under us. - Resolution handles BOTH manager shapes. list_models is a method of the MULTI manager, which has no .backend of its own -- the per-model manager does -- so the first version of this could never have fired either. Found by writing the test, not by reading the code. - Several spellings read (max_model_len/n_ctx/ctx/context_length, plus the vllm/runpod/llamacpp blocks), because several backends own this number. - None means UNKNOWN, never unlimited, and the worker now says out loud when a server named no window, naming ocr.surya_context_tokens as the answer. Also, so that "which build is this pod running" stops being an inference: the pod boot log prints its coderai version and the short git sha of the app layer (new GIT_SHA build arg). A mutable `:latest` is a fine way to publish the current release, but it leaves two pods started minutes apart looking identical in an account listing -- and on 2026-10-13 that cost two pods being restarted that already had the fix, because nothing in their logs named their version. Tests: 27 new. They pin what the resolver PREFERS (live over config), what it tolerates (a stray string, a list-wrapped number, a non-dict backend block), that another model's loaded backend is not consulted, that the listing actually calls it, and that a server naming no window produces a message rather than a silence. Full suite: 25 failures, all pre-existing and identical to baseline. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The Digesta dev's 400 on page resumption, and a bug behind it that was worse than the 400. The vLLM backend built its outgoing request from an ALLOW-LIST of fields it recognised. Resumption asks for two beyond an ordinary request -- continue_final_message, so the model carries on the exact string, and add_generation_prompt: false, so no new turn is opened after the prefilled text -- and neither was ever forwarded, by this or any previous version. What reached vLLM was a conversation ending in an assistant turn with add_generation_prompt left at its default TRUE. surya-ocr-2's chat template then appends a fresh <|im_start|>assistant after the text so far, so the model was asked to START A NEW ANSWER about the same image. It did. And the failure hid behind the guard built to catch it: the new answer repeated the beginning of the page, the no-progress check saw text it already had, logged "came back repeating itself" and discarded the segment. 253 of 262 resumptions on the live install -- read for a day as a model that degenerates on dense pages, when it was a model being asked the wrong question, answering correctly, and having the answer thrown away. It also explains the dev's table exactly: at max_tokens=8000 resumption is barely exercised and OCR holds 93%, while at 4000 and 2000 nearly every page resumes, each resumption is a full generation that produces a restart and is then discarded, and the batch times out. Fixed by forwarding BY EXCLUSION, which is the only form that stays correct: coderai names the fields it CONSUMES (_CHAT_CONSUMED_FIELDS) and everything else goes to the server, which is the thing entitled to decide what it supports. An allow-list means every field a server gains after the list was written is born broken -- the same reasoning as forwarding page diagnostics by exclusion. It reaches only a backend whose signature accepts it, so a backend that cannot carry an unknown field is unaffected rather than broken, and model/messages/stream are never overridable this way. Also: a 4xx explains itself, if nobody discards it. raise_for_status() reports "400 Client Error: Bad Request for url: ..." -- the symptom and nothing else -- while vLLM puts the reason in the BODY. A whole diagnosis cycle was spent guessing at a 400 that had already been explained and thrown away. The error now carries the server's own words. Made tolerant of a response object that reports no status_code, rather than turning an odd shape into an AttributeError inside the request path. Two findings for the operator, not code: the pods serve max-model-len 32768 while the model entry asks for ctx 18432, so per-model vllm settings still are not reaching the pod; and this fix must be in the POD image, because the proxy that dropped the fields runs there. Tests: 16 new (flags reach the payload, an unknown field is forwarded, no passthrough changes nothing, structure not overridable, exclusion at the API layer, signature-probed delivery, the error carries the server's words). Full suite: 25 failures, all pre-existing and identical to the stashed baseline. Co-Authored-By:
-
- Oct 12, 2026
-
-
Stefy Lanza (nextime / spora ) authored
A QLoRA trained for a client sat on disk for ten days while a second service rented its own A40 every morning, pushed the adapter over SSH and proxied to it. The path that should have carried it was written but had never been run end to end, and five things were in the way. * `ensure_text_loras` DROPPED a PEFT directory — "publish it to HuggingFace" — which is the shape every trainer writes. It now travels as one deterministic tar (sorted names, fixed uid/gid/mtime, so the content id is stable) and is unpacked by the blob store, because vLLM loads an adapter from a directory: adapter_config.json carries the rank and target modules, and a bare .safetensors is not loadable. * the adapter was re-read and re-hashed ON EVERY REQUEST. A Target is rebuilt per request, so `lora_entry` was set again each time; the bytes were never re-sent (the store is content-addressed) but 159 MB was read and hashed, and the pod asked, only to learn it already had it. Cached per (pod, path, size+mtime), so a retrained adapter at the same path is still sent again. * `_lora_args` passed a CONTENT ID straight through, producing `--lora-modules name=sha256:…`, which vLLM cannot open. On a pod that is the only form the adapter has. * an engine launched for a base model carried none of its adapters. vLLM registers them at launch and one vLLM serves one base, so the pod a shared pool rented for the base alone could never serve an adapter of it — the whole point of sharing the pool. Every model naming the same base now contributes its adapter under its own model id, which is the name requests ask for, so two versions can be A/B'd on one GPU with no reload. * `--max-loras` was never set. At its default of 1, requests naming different adapters cannot share a batch and serialise instead: the cost lands on latency, not on an error. Extracting a bundle is a write from network bytes, so member names are checked for escapes before anything is unpacked, and tar's data filter is used as well. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A fifth OCR engine. Chandra (datalab-to/chandra-ocr-2) is from the same lab as Surya and is driven the way olmOCR is — one page image over an OpenAI chat endpoint, three serving modes (model | vllm | server) — but its `ocr_layout` prompt asks the model to answer as one <div> per layout block carrying data-bbox (normalised 0-1000) and data-label. So it is the only engine here that returns text, boxes, labelled regions, tables AND page metadata in one pass, with no separate detector. Why it was worth adding now: it is a DIFFERENT MODEL FAMILY from surya-2 (Qwen3-VL's tokenizer, against surya-2's own 65k WordLevel vocabulary) reached over the same transport, at the same pixel ceiling, from the same rasterised page. That makes it a cheap controlled experiment on the U+FFFD accent loss: accents intact under chandra and lost under surya points at the model; both losing them points back at the document's pixels, and then pdfimages -list is the next thing to read. Implementation notes: - NO new pip dependency. The two prompts are mirrored verbatim from chandra/prompts.py (down to the Python repr of the allowed-tag list, which is trained-on wording), the input scaling is chandra's own scale_to_fit reimplemented, and the reply is split with stdlib html.parser rather than pulling bs4 and markdownify into the main venv for a tag split. - Chandra's pixel cap is 3072x2048 = 6,291,456 — the SAME numbers as surya-2 — and it snaps to a 28-pixel grid for the Qwen vision tower. So chandra_dpi defaults to 250 for exactly the reason surya_dpi does. Boxes are de-normalised against the image the MODEL saw, and the size the caller rasterised is reported as meta.source_size when they differ: de-normalising against the pre-scaling size would put every box on a scaled page in the wrong place. - A repetition loop answers 200. A VLM that loses its place repeats the last thing it saw until the token budget runs out, and the result looks like a successful page. chandra_max_retries (2) re-asks with a raised temperature — a greedy decode falls into the same loop twice — and a page that still loops is returned flagged with meta.repeat_detected rather than handed on silently. - The parse never loses page. No blocks at all, content outside any block, an unclosed final div (routine at the token limit), a bbox that will not parse: each keeps the text. A bbox that cannot be read yields NO region rather than a full-page one, because a wrong box looks like a real answer. - Table cells get a space, not a newline, so a table does not flatten to "VoceImporto". License: narrower than Surya's GPL. The code is Apache-2.0 but the WEIGHTS are a modified OpenRAIL-M — research, personal use and companies under $2M funding/revenue, no use competing with Datalab's API. So it is gated on chandra_accept_license, and the pod path differs from surya's on purpose: asking a pod for surya by name IS the GPL acceptance, while chandra's is only ever inherited from a decision already recorded in this install's config. A pod cannot accept a licence on an operator's behalf. lang= is accepted and does not steer it (language_aware: false), like surya-2 and olmOCR: 90+ languages, no language input. So it keeps ONE pool whatever is asked — a language-keyed pool is a second set of loaded instances holding VRAM. Wired through: engine registry, licence gate, per-engine DPI, the vLLM stop path (its VRAM is the subprocess's, not a pool's), the capability catalogue (which now reports boxes/regions/tables from chandra_layout, because with layout off there genuinely are none), model-name aliasing, the automatic fallback chain, pod planning + env seeding, the frontproxy's local-OCR check, models.json capability advertisement, the pipeline step's engine list, admin save/load and the Settings OCR card. tests/test_ocr_chandra.py: 61 tests. 788 passed across the OCR/config/admin/ runpod selection with the same 10 pre-existing failures (broker_config x5, studio_composed_surfaces x3, whisper_server x2). test_ocr_fanout's catalogue test asserted a frozen engine set; it now asserts against the registry, which is what "lists every engine" was always supposed to mean. Also fixed while in there: the Settings page used `parseInt(v) || default` for numbers whose zero is meaningful, so a deliberate 0 saved as the default. New _num() helper only falls back for an empty or unparseable field. Co-Authored-By:
-
- Oct 11, 2026
-
-
Stefy Lanza (nextime / spora ) authored
RunPod reclaimed an A40 mid-document today. The pool went on reporting it healthy, so every page chose the same corpse: the in-flight count climbed, the pod was never dropped, and nothing drained. From the client that reads as "requests neither complete nor time out", and it is what wedged the test. RunpodBackend has always handled this — _unreachable() drops the pod and the request goes again. The fan-out gateway never did: it returned 502 and left the dead pod in the pool as healthy. So a lease can now report its upstream dead, which drops the pod, clears the cached candidate set and releases the slot; the page is retried ONCE against whatever replaces it, and a third attempt would just be a retry loop against a real outage. Only connect-level failures count as dead. A read timeout is a model taking its time over a dense page, and discarding a healthy pod for that would be an outage of its own. Also: the first version of that handler could not work. It used a metaclass with __instancecheck__ so `except _DEAD_UPSTREAM` would match httpx's errors — but `except` matches on the class hierarchy at C level and never consults __instancecheck__, so it caught nothing. A test caught it; the check is now an explicit isinstance() where it cannot lie. And the registry path again: 0.3.15 keyed the fallback off XDG_CONFIG_HOME, which the container launcher exports only into the processes IT starts. A diagnostic script entering the container another way printed a THIRD registry path and created a second pod TLS CA. The fallback now reads CODERAI_CONFIG_DIR, the variable the launcher actually sets, so every process agrees however it was started — and since pod_tls.tls_dir() derives from the registry path, the CA follows. Co-Authored-By:
-
