- 25 Oct, 2026 13 commits
-
-
Stefy Lanza (nextime / spora ) authoredThe layout pass read its per-page option with `_opt("layout", True)`. The accessor is `_page_override`. A pod answered 500 {"detail": "surya ocr failed: name '_opt' is not defined"} after the image was built, pushed, signed and booted, because nothing executes these modules until a page reaches a pod: `codai/ocr/workers/*` run OUTSIDE the package -- the surya worker in its own venv, forbidden from importing codai -- so no test had ever run a line of them. The test I wrote for the layout pass asserted the SOURCE STRING of that call and proved only that the text was there. So the fix comes with a guard that would have caught it: every `_`-prefixed name a worker module loads must be bound somewhere in that module. Those are the module's own helpers, so an unbound one is a typo and not a global from elsewhere. Pure ast + builtins -- no linter, nothing added to requirements -- and it is checked against the bug itself, because a checker that passes everything is worse than none. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
-
Stefy Lanza (nextime / spora ) authored
`pod_ocr_engine` rewrote EVERY whole-request OCR forward to docTR. The premise was that a pod image cannot carry a VLM's model server, which was true of `coderai-vllm` and never true of `coderai-vllm-ocr`: that image carries surya, the rasteriser and its own vLLM -- it IS the pod-side pipeline -- and it carries no docTR at all, which is why its pods launch with CODERAI_OCR_FALLBACK_ENGINES=off. So the rewrite asked a pipeline pod for the one engine it cannot possibly have, and the pod answered "OCR engine 'doctr' is not enabled". The question cannot be asked without the image, and the gateway was not passing one. Found by sending a page through a pod, not by reading the code: the id resolved, the pool provisioned, the pod booted and went ready, and the request still came back 400 naming an engine nobody had asked for. A named configuration (ocr.engine_configs) also resolves to its base engine here. The ids are an orchestrator-side naming layer and a pod is a pre-configured instance of one -- its env already carries the overrides -- so telling it the id names an engine it has never heard of. Omitting the image keeps the old answer, so callers that cannot know it are unchanged. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A model id is a configuration, not a set of weights. Image models have worked that way for a long time -- one checkpoint, several published ids, one copy in VRAM -- and five things that should have followed from it did not. PER-MODEL LoRA STACKS. A diffusion pipeline could carry an acceleration LoRA and per-request LoRAs, with nothing in between: no way to say "this id is that base plus this style LoRA at 0.8". `loras` on a model entry says it, in the shape text models already use. `lora_path` is honoured too -- it had been in the config and the admin page for years while nothing on the diffusers image path read it. And `set_adapters` replaces the ACTIVE SET, so the image path's request-LoRA call was switching the pipeline's own adapters off: a 4-step distilled preset running un-distilled, and an id published as a fine-tune serving its base checkpoint. 200 OK, a plausible picture, nothing in the logs. The video path had always re-included them. A per-model LoRA that will not load now FAILS THE LOAD. The acceleration path logs and continues because the cost there is speed; here it is identity. SINGLE-FILE CHECKPOINTS. `from_single_file` was wired up only for GGUF components, so the one .safetensors Civitai ships for every community fine-tune could not be opened at all. A model path may now name a file, or `repo:file.safetensors` -- community repos hold every version side by side, so the repo id alone does not say which to serve. A SAMPLER IS A CLASS PLUS ITS OPTIONS. "DPM++ 2M SDE Karras" is DPMSolverMultistepScheduler and two settings; naming the class alone gave plain DPM++ 2M, a different sampler, silently. `scheduler_config` carries them. RASTERISATION, PER REQUEST. A scan is normally handed to the model as the scanner wrote it -- 0.1 ms and 147 KB against a render plus ~1.8 MB of PNG -- but only between the target DPI and `page_image_max_dpi_factor` times it, and only install-wide. `native_dpi` removes the ceiling for one request; the floor stays, because a page passed through BELOW the target is a coarser page, which is what cost surya-ocr-2 its Italian accents once already. An uploaded image now carries its own bytes too: it was decoded and re-encoded to arrive at the picture the caller already had. SEVERAL IDS FOR ONE OCR ENGINE. `ocr.engine_configs` gives OCR what image models had: named configurations of one engine, each its own id, its own catalogue row, its own pool. An override naming a setting that does not exist is refused rather than quietly serving the base settings under another name. Which forced the last one into the open. `/v1/ocr/engines` advertised `surya: {regions: true, tables: true}` while the surya worker built every page with `"regions": [], "tables": []` hardcoded. surya ships a layout predictor and a table recogniser; they were never run. A client that chose surya BECAUSE the catalogue said it returned regions got empty arrays and no error. Both halves are now true by construction: the row is read from `surya_layout` / `surya_table_rec`, and those load the predictors. Regions come back with surya's own reading order (LayoutBox.position, explicit as `order`, -1 where an engine states none) and tables as cells with row/column ids. Loading is configured; spending is per request. `layout=true` chooses whether to pay for the pass on an engine that HOLDS the predictor -- it cannot make one acquire a model mid-page, which is how a footprint grows past what the VRAM accounting reserved. Asked for where none is loaded, it is refused with the configuration to use instead; answering 200 with empty regions would be the same lie wearing a request field. Pods differ from their siblings too: a pool whose entry names an `ocr_engine_config` boots pods with that configuration's settings, and a named config overriding `<engine>_model_id` reaches its own pods rather than sharing its sibling's. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
set_image_model keys config by path: self.config[f"image:{model_name}"] = config or {} so a second image_models entry for the same checkpoint overwrote the first and said nothing. Two configs of one checkpoint is not an odd thing to want -- the same base served with a different transformer, quantization or step count -- and text models have supported it for a long time, addressing the second under its alias and skipping an unaliased duplicate with a message. Image models had none of that. The symptom is worse than losing an entry, because nothing looks lost: both entries still appear in /v1/models and both register their aliases, so a request for EITHER is answered with whichever config was registered last. Found while testing two configs of Qwen-Image-Edit-2511, one with a GGUF transformer and one with a fine-tuned transformer quantized at load: a request for the GGUF config was served by the 4-bit one, and the only clue was "Registered image model: Qwen/Qwen-Image-Edit-2511" appearing twice in the startup log. So the image path now does what the text path does: a sibling sharing a checkpoint is registered under its alias, which gives it a config key of its own, and is NOT aliased back to the shared path -- that mapping is what collapses the two again. An unaliased duplicate is skipped with the same explanation text models give. `path` now travels in every image config, because a sibling is addressed by an alias and an alias is not something from_pretrained can open: the config has to be able to say which checkpoint it is a config OF. _load_diffusers_pipeline prefers it when it differs from the name it was called with, and for a primary entry the two are the same string. Needs a restart to take effect, registration being a startup act: a collision that has already happened cannot be undone in the running process's config dict. Full suite 4558 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.4.96 added gguf>=0.10.0 to requirements.txt, which was half right. Two things about how this image is built that the change missed: * requirements-oci.txt has carried `gguf>=0.9.0` since June, so the dependency was not undeclared after all -- only unsatisfiable. diffusers' GGUFQuantizationConfig refuses anything below 0.10.0 ("requires both PyTorch and GGUF>=0.10.0"), so a 0.9.x install satisfies the line and still cannot load a GGUF pipeline component. Pinned to 0.10.0. * Dockerfile.oci COPIES requirements.txt, requirements-nvidia.txt and requirements-vulkan.txt into the build and then deletes them unused: the RUN installs `-r /tmp/requirements-oci.txt` and nothing else. So for the container image, requirements-oci.txt is the only list that counts. requirements.txt still matters for a host build.sh install, and the in-image upgrade reads BOTH, which is why the entry there is not wasted. No rebuild needed to ship it: coderai-upgrade's delta install takes requirements.txt plus requirements-oci.txt, installs what is new, and the autoupgrade commits the result onto the image tag. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Requested for a client that edits photos by instruction -- "make the dress red", "add a necklace" -- which plain img2img cannot do at any strength, because `strength` is the fraction of the source it destroys before re-imagining it. Serving Qwen-Image-Edit / FLUX.1-Kontext class models turned up two latent bugs on the way. The instruction path. /v1/images/edits now shapes its call to what the pipeline accepts, from the signature, rather than branching on the model: QwenImageEditPlus / QwenImageEdit / FluxKontext image ✓ strength ✗ SDXL-Img2Img / QwenImageImg2Img / FluxImg2Img image ✓ strength ✓ Taking an `image` with no `strength` IS the distinction, and it generalises, so no list of class names has to be maintained. `strength` is dropped where it does not exist -- a caller cannot know which kind of model it is addressing, and these pipelines carry no **kwargs to swallow it, so passing it through would be a hard TypeError. `guidance_scale` becomes `true_cfg_scale`, the knob they actually read. Steps default to 4 at cfg 1.0 for them instead of inheriting img2img's 30 at 7.5, which would waste a minute an image and overcook the result. Up to three images may be given (`image` plus `images`) for a composed edit, conditioned at one shape; more is a 400. `instruction_edit` is reported in /v1/models so a client can hide the strength slider and treat the prompt as an instruction. Verified to classify the four Qwen/Kontext/FireRed ids and to leave Pony, Z-Image and Illustrious as img2img. component_models is new: a map from a component to ANOTHER repo, loaded and handed to the pipeline as a prebuilt object. Several Qwen-Image-Edit derivatives ship a transformer and nothing else -- the base supplies the VAE, the text encoder and the scheduler -- and there was previously no way to express that, so they could not be served at all. Its entry in component_quantization still applies, so a transformer too large for the card is quantized as it loads. Two latent bugs, both of which anyone using GGUF image components would have hit: * `gguf` was never a declared dependency. build_gguf_pipeline_components has existed through several releases and could not have worked: diffusers needs gguf>=0.10.0 for GGUFQuantizationConfig and only says so mid-load, as "Unable to load weights from checkpoint file". Now in requirements.txt, which build.sh installs. * from_single_file was called with no `config`, so it INFERRED the architecture -- and its fallback is Stable Diffusion 1.5. A Qwen-Image, Flux or Wan transformer therefore failed with "stable-diffusion-v1-5 does not appear to have a file named config.json". It now passes the base pipeline's own component subfolder, which was already at hand. Measured on an RTX 3090 after the fixes: the Q4_K_M transformer binds as QwenImageTransformer2DModel (13.23B params, 85s) and the assembled QwenImageEditPlusPipeline is 22.68 GiB resident with a 4-bit text encoder -- which fits 24 GB but leaves 1.32 GiB and OOMs on 390 MiB of activations. That is a deployment setting, not a code one, and it is why the figure is measured here rather than asserted. Full suite 4558 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Fixes a NameError shipped in 0.4.94: `name 'model_key' is not defined` on every /v1/images/edits request. Replacing that endpoint's load block with the shared _pipeline_for_task call removed the line defining model_key, while a line further down still used it to look up the model's config. The inpaint endpoint got its replacement definition; this one did not. Editing was broken from the moment it was deployed, and a client reported it from production. 187 tests covered this code and none of them caught it, because every one read the source as text -- is this guard present, does this call precede that one. A structural test cannot see an undefined name on the first line of the request path. That is a flaw in how the work was tested, not bad luck. So the endpoints are now executed. tests/test_image_edit_endpoints_actually_run.py calls create_image_edit and create_image_inpaint with their collaborators stubbed -- the manager, image decode, the response writer, the pipeline -- so the whole body between the request arriving and the pipeline being called actually runs. Checked the way a regression test has to be: with the bug reintroduced it fails with the production message verbatim, and passes with the fix. One that passed either way would have been worth nothing. The tests assert what the body is for, not just that it returns: that the SOURCE IMAGE arrives at the pipeline at the size requested, that an omitted size keeps the photo's aspect instead of squaring it, that the mask matches the image's shape, that a malformed size is a 400 from the endpoint rather than a 500 from inside it, and that a model's prompt_prefix survives every layer in between. Also noted for whoever reaches for a linter next: a stdlib undefined-name scan was tried first and would NOT have caught this one, since it cannot tell a function-local assignment from a module-level name. Real scope analysis means pyflakes, which is not a dependency here. Full suite 4558 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A checkpoint is a UNet, a VAE and its text encoders; which task it performs is a property of the pipeline class wrapped around them, not of the weights. Both tasks already worked after 0.4.93, but each endpoint loaded its OWN pipeline under its own key -- so a client that creates a character with a model and then edits it with the same one held an 18 GB SDXL fine-tune twice, and the second load evicted the first. Alternating between the two thrashes the card continuously. Reading the two loaders side by side turned up the other half of it: _load_img2img_pipeline ignored the model's configuration altogether -- dtype hardcoded from global args, no offload, no quantization, no VAE override. Only _load_diffusers_pipeline honours models.json. A 4-bit model was therefore served at 4 bits for generation and bf16 for editing, out of two copies. So the model is loaded once, through the configured loader, under the manager's own key -- its VRAM accounting and eviction now see one model -- and each task gets a view built with AutoPipelineFor*.from_pipe around the components already resident. /v1/images/generations gets the same treatment in reverse, so a model an edit loaded can create a character without reloading. The views are cached on the base pipeline, not as extra models-dict entries. Deliberately: they share the base's tensors and so must die exactly when it does, and a second models-dict key per task is what let a cache key reach from_pretrained in 0.4.93. The img2img:/inpaint: keys are gone from the module entirely. Task detection asks diffusers' own tables rather than a signature, because a ControlNet text-to-image pipeline requires an `image` -- the control hint, not a source -- and a signature check would file it as img2img. Tested. Two behaviours worth knowing: the views share the UNet, so a LoRA or IP-Adapter applied through one task is visible to the others (same weights, by design); and SD, SDXL, Flux and Z-Image all have entries for all three tasks, so this is not an SD-only arrangement. I had assumed Z-Image was unmapped and wrote a test asserting it; the test caught that and both it and a code comment are corrected. 28 tests here, 187 across the image files; full suite 4549 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Reported from a client against nesaorg/PonyRealism-Uncensored as three symptoms -- "the edit result is garbage", "size is ignored", and a 500. Two defects, neither of which announced itself. The edit was never an edit. _load_img2img_pipeline picked its class by looking for 'xl', 'sdxl', 'stable-diffusion' or 'sd' in the MODEL NAME, and "nesaorg/ponyrealism-uncensored" contains none of them, so it fell through to the generic DiffusionPipeline -- which faithfully honours the checkpoint's model_index.json, and that one declares StableDiffusionXLPipeline: the text-to-image class. Every diffusers __call__ ends in **kwargs (it carries the deprecated callback pair), so `image` and `strength` were accepted, dropped, and the pipeline rendered a fresh picture from the prompt alone at its default 1024x1024. One defect, all three symptoms: garbage (a prompt saying "add X, keep everything else" rendered from scratch), size ignored (1024x1024 is the text-to-image default, not the source's shape), and 200 OK throughout. stable-diffusion-xl-base-1.0 worked only because its NAME contains "xl". Both loaders now ask AutoPipelineForImage2Image / AutoPipelineForInpainting, which read the declared class and substitute the one for this task. A guard backs it up, on cached pipelines as well as freshly loaded ones -- an install already holding a text-to-image pipeline would otherwise keep using it until it restarted. A pipeline with no `image` input is converted with from_pipe (the same components, no second copy in VRAM) or refused with 501. A 200 carrying a picture of something else is the one outcome worth ruling out. The 500 was a cache key used as a repo id. These endpoints keep task-specific pipelines in MultiModelManager.models under "img2img:<repo>", and the manager's fuzzy already-loaded pass matched one by its last path segment, then assigned the whole key to resolved_name -- so from_pretrained was asked for a repo called "img2img:nesaorg/PonyRealism-Uncensored" and answered "Repo id must use alphanumeric chars" about a model that loads perfectly well. That pass now skips task namespaces, which the endpoints look up by exact key anyway. Writing the test for that found two more namespaces with the identical latent bug -- depth: and segment: -- so _TASK_NAMESPACES covers four, and an invariant test asserts every task prefix the endpoints invent is declared. A future endpoint cannot reintroduce this quietly. Not fixed, because it is not a bug: /v1/images/edits is plain img2img, so "add X and keep everything else exactly the same" cannot work however correct the pipeline is. That needs a mask through /v1/images/inpaint, or an instruction-edit model exposed with a capability clients can prefer. 16 tests; full suite 4537 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
clip_skip was mapped onto diffusers' argument of the same name, and that argument counts something else. Its SDXL path reads hidden_states[-2] # clip_skip is None hidden_states[-(clip_skip + 2)] # otherwise so diffusers' clip_skip is the number of layers skipped BEYOND the penultimate one SDXL already uses. Every model card, A1111, ComfyUI and InvokeAI mean something simpler by the name: 1 is the final layer, 2 the penultimate, n the n-th from the end. So a request asking for 2 -- which is what the booru-tag SDXL families are trained at, and what their model cards all say -- read hidden_states[-4] instead: two layers too deep, the equivalent of clip-skip 4, which on a Pony-family model is the difference between a photograph and mush. It was inconsistent as well as wrong: the same 2 meant [-4] on SDXL and [-3] on SD 1.x, two depths from one number. The mapping now happens here, once, in the sense users mean, and clip_skip 2 reaches the same layer whichever family the model belongs to. "Mirrors diffusers exactly" was the mistake in 0.4.89: mirroring is the wrong goal for a field a caller fills in. SDXL's pooled conditioning vector is also handled properly now. The UNet cannot run without it and diffusers refuses prompt_embeds that arrive without one, so handing it half the set is a ValueError raised in check_inputs -- an instant 500, before any denoising. Where a pipeline does not offer the vector, it is rebuilt the way CLIP produces it (pooled at the end-of-text token, then projected through text_projection) rather than giving up the weighting and the window count this module exists for; only when even that is impossible does the encoder stand down and let the pipeline encode the prompt itself. The invalid combination is no longer constructible: either all four keys or None. A 500 from /v1/images/edits, /inpaint or /outfit now logs its traceback. The cause used to reach the caller alone, so the log showed the request and then the status, and a failure there could not be diagnosed afterwards at all. Test note: test_a_400_is_not_relabelled_as_a_server_fault scanned a fixed 250-character window for the HTTPException pass-through above each catch-all; the new log line pushed that to 316-382 and broke a passing check on correct code. It now asserts structure instead of proximity -- the pass-through must be the nearest handler inside the same function, with the logging between it and the raise -- which also catches it drifting into a different function. 58 tests in the prompt file; full suite 4521 passed with the same 15 pre-existing failures as 0.4.91 (diff empty). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two asks that turned out to be about the same thing: a request taken as meant. A model fine-tuned on a quality ladder needs that ladder on every request, and a client speaking the plain OpenAI image API sends a sentence with nowhere to put it. CoderAI adds no tags of its own and still does not: prompt_prefix and negative_prompt_prefix on the MODEL's config entry are comma-joined in front of whatever the caller sent, and both default to empty, so an install that configures nothing sends exactly what was written. A request overrides either, and an empty string switches one off for that call -- what a caller writing its own complete prompt needs. Applied at most once: a prompt that already opens with the ladder is left alone rather than given it twice, which would only dilute the rest. The prefix resolves once, before the encode, and the same text feeds the embedding cache key. Keyed on the prompt as SENT, two models with different prefixes would share one entry and the second would be served the first's text. The archive keeps the prompt that ran plus the one requested when they differ. On OCR: there is no per-page byte limit, and there never was. A page over a megabyte is ordinary -- an A4 scan at 300 DPI is several -- and both a 3 MB and a 50 MB body upload completely and are rejected only at authentication, with the bundled nginx at 4096m and the generated /v1/ocr routing not lowering it. What did exist was a hardcoded 100 MB ceiling on one document, which a 300 DPI colour filing passes without being unusual and which nothing could raise. It is ocr.max_upload_mb now (0 = no ceiling here), in Settings, and its 413 names the setting and says the document was NOT read -- the loud refusal ocr.max_pages already makes, instead of "file too large (max 100 MB)". The 1 MB figure that makes a large page fail in the field is nginx's own client_max_body_size default, in a reverse proxy added in front. That is exactly the shape of "anything over a megabyte fails", so it is recorded in the code, the settings hint and docs/ocr.md rather than left to be rediscovered. Also documented there: what IS capped is pixels, not bytes -- Surya-2 and Chandra top out near 6.29 MP, an A4 page at about 255 DPI, and above that the page is silently downscaled to fit. 37 tests, including a real 1.25 MB single-page PDF built from noise and read back through pdfium as one page. Full suite 4512 passed with the same 15 pre-existing failures as 0.4.90 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three things the image endpoints accepted and then quietly did something else with. size. A size that did not parse fell through to the default, so "1024" or "1k x 1k" generated a square nobody asked for and the response looked like a success. A side that was not a multiple of 8 went to the pipeline as-is, where the VAE cannot represent it. Nothing capped it either, so 4096x4096 was accepted and came back as a CUDA OOM -- which reads like a busy GPU rather than a size the caller chose. Now: a 400 quoting what was sent, a 400 for a zero dimension (snapping it up would answer an uninitialised client variable with an 8x8 image), snapping to the grid with a log line, and a 400 naming image.max_side / image.max_pixels when it is over the ceiling. "auto" is accepted as in the OpenAI API, and the dead `width != width` NaN guard on a value that was already an int is gone. The archive records the size actually generated alongside the one requested. /v1/images/inpaint no longer defaults to 1024x1024, which squashed every non-square photo and pulled the mask out of alignment with it -- worse than the same default on /v1/images/edits, where it was fixed in 0.4.86. Omitted now keeps the source's shape. Textual inversions. easynegative, bad-hands-5 and the rest are vectors living in CLIP's embedding table under a word of their own, and until the file is loaded that word is an ordinary unknown token -- so a large part of the ecosystem's negative prompting asked for nothing at all. They load from the request (embeddings: [...]), from the model's config (textual_inversion), or by filename from image.embeddings_dir, which is how A1111 works and what a prompt copied off a model card assumes. Two details decide whether this works at all: an SDXL-format embedding holds a clip_l and a clip_g tensor that must go into the matching tokenizer/encoder pair, and a multi-vector embedding occupies several tokens (tok, tok_1, tok_2...) that the prompt has to be expanded into -- without it an eight-vector embedding contributes one vector and appears inert. A name the caller gave that will not load is a 400; a bad file in the directory is a line in the log, because one file should not break every request. /v1/images/outfit goes through the same encoder now, so the weight grammar and clip_skip reach it too. Its per-frame video path encodes the prompt once for the clip instead of once per frame. And a 400 is no longer reported as a failed backend. On the first request for a model that is not loaded yet, the generate call sits inside the try that falls back from diffusers to sd.cpp, so a rejected request was caught as a load failure and answered with a report about which backends failed. The three edit endpoints did the same through their catch-alls. All four re-raise HTTPException untouched. Found by the tests while writing them: the embeddings directory was searched by the renamed trigger word rather than the filename, so "easynegative:myneg" never resolved, and "768X512" did not parse. 64 new tests; full suite 4475 passed with the same 15 pre-existing failures as 0.4.89 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A CLIP text encoder reads 77 tokens, two of them markers. Handing a longer prompt to a diffusers pipeline does not fail: the tokenizer truncates it and the model renders what fits. The booru-tag fine-tune families (Pony, Illustrious, NoobAI) open with a run of quality tags, so the part that got dropped was the subject -- and the request answered 200 with an image, which is why it was easy to miss. Image prompts are now encoded here rather than by the pipeline, which is what makes the rest possible: * as many 75-token windows as the prompt needs, concatenated -- no limit * the A1111 weight grammar: (tag:1.3), [tag], bare (tag) = 1.1, nesting multiplies, \( escapes. Per-token scaling with the window mean restored, so emphasis does not drag the brightness along with it * BREAK, which closes the current window instead of blending tags meant for different subjects * clip_skip, mapped to diffusers' own convention -- SDXL counts from the penultimate layer, SD 1.x from the final output and needs the layer norm applied by hand. Getting that backwards shifts every image Three places were dropping the negative prompt outright. The sd.cpp path hardcoded negative_prompt='', so for every GGUF image model whatever the caller asked to avoid stopped at the API -- half the idiom, since the quality tags have a negative half (score_4, score_5, score_6). /v1/images/edits and /v1/images/inpaint had no field for one at all, so an edit could only ever be steered positively. All three now pass it, and take clip_skip and prompt_weighting with it. clip_skip and prompt_weighting are also per-model and server-wide settings, because a client speaking the plain OpenAI image API has no field to ask for clip_skip and a model trained at 2 needs it on every request. Request beats model beats global. Weighting defaults on, so a literal '(' in prose now weighs 1.1 -- the ecosystem's convention, with \( and prompt_weighting:false as the way out. What that cannot be allowed to change is prompts that already worked: against a real CLIPTextModel a short unweighted prompt encodes bit-identically to what diffusers produced, because CLIP pads with its own end-of-text token, so [bos]+toks+[eos]+pad... and [bos]+toks+pad...+[eos] are the same ids. The test records that reasoning, so a tokenizer where it stops holding is caught. Flux, SD3 and Z-Image carry a T5 with its own limit and conventions and keep their pipeline's native text path. 49 tests; full suite 4410 passed with the same 15 pre-existing failures as 0.4.88 (diff empty). Co-Authored-By:
-
- 24 Oct, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored
A deployment has to point HF_HOME somewhere before any config is read, so a container bakes it into the image. models.hf_cache_dir was then applied in exactly one place -- inside the server's main(), by mutating os.environ, which reaches no other process. So the front used the configured cache and everything else used the image's: supervisord's other children, and the isolated-venv workers, which were handed HF_HUB_CACHE and HUGGINGFACE_HUB_CACHE but not HF_HOME or TRANSFORMERS_CACHE, enough for a worker to re-fetch its model into the directory the image named. Measured on one install: 22 GB in the image's directory, 13 GB of it a byte-for-byte duplicate of the configured cache, three copies of one model, and the whole of it on the smaller of the two disks -- which was 99% full. The configured value is now applied from codai/__init__.py, before anything can import huggingface_hub, transformers or torch and read these variables once at import time. Every coderai process imports the package, so every one of them agrees without the launcher being involved. All four variables, since leaving any behind splits the cache again. main() keeps an authoritative pass against the config it actually loaded (--config may name another directory), the venv workers get the same four through child_env(), and the OCR fleet supervisor -- stdlib-only on purpose -- reads the same key from the config.json it already opens, instead of building the path from CODERAI_CACHE_DIR. Nothing is invented: with no hf_cache_dir configured the environment is left exactly as found, and CODERAI_NO_HF_CACHE_ENV opts a process out. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
POST /v1/files/upload exists so a client sends a file once and then names it: inlining a 40 MB clip in every request is the waste it was built to remove. But it stored its content-addressed files in the output directory and answered 501 when there was none configured -- so on such an install the only way to send media was the inlining the endpoint replaces, and a client that had already switched to uploading saw its photo edits fail outright. An install always has a cache directory, so uploads fall back to it (media.uploads_dir()). With an output directory nothing moves: an upload still lands there beside the generated files. Both stores stay in the lookup path, so an install that gains --file-path later does not lose track of what it already holds, and /v1/files/<name> serves from the cache store only for up- names -- the cache directory does not quietly become a second public output directory. 501 now means there is nowhere at all to write; a non-writable store answers 507 rather than a 500 out of makedirs. Composing a new video still needs an output directory, since there has to be somewhere to write the result. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Four ways an edit could answer 200 while quietly doing the wrong thing. The enhancement paths decode a clip to frames and encode a new video, so the audio was left behind: a clip that arrived with speech came back silent. Only the ffmpeg minterpolate fallback -- the path nobody takes by default -- ever kept it. _mux_original_audio() puts the track back on all three in-process paths, stream-copied where MP4 accepts the codec and re-encoded to AAC where it does not. Both paths already preserve the source duration, so nothing is stretched. A silent source, or a remux PyAV cannot do, returns the frames unchanged rather than losing a finished GPU job. The image and video editing endpoints took base64 (and, for video, an http URL), so a caller that had already uploaded a file had to re-send it on every edit -- megabytes a call for a file on the same disk. They now resolve through the compose resolver, so a /v1/files path or a sha256: upload id works everywhere, and our own files are read off disk instead of fetched back over a loopback that would need a token. Responses carry the relative path beside the url, as compose does, so a caller need not depend on the public hostname. Routing through that resolver would have tightened what callers may send: it validates base64 strictly, which rejects the 76-column wrapping that MIME, the base64 tool and encodebytes all produce. Whitespace carries no meaning in base64, so it is now stripped rather than refused. /v1/images/edits defaulted size to the string "1024x1024", so omitting it squashed every photo that was not square. Omitted now keeps the source's shape at the ~1 MP a diffusion model wants, on the 64-grid it needs. /v1/images/upscale fell through to PIL LANCZOS for an unrecognised model and still answered 200, charging for an AI upscale that never ran and returning something merely soft. That is a 400 naming the models that would work. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The line was printed on the way INTO a re-read, so an install with `retry_degraded` off -- or on "auto" behind `fallback_engines: off`, which is the same thing -- produced no record anywhere of a degraded page. The engine answers 200, nothing raises, and the only other copy of the finding is in the reply, which belongs to the client: a batch could come out almost entirely truncated and leave the server log looking like a clean run. That is how ~97% of one run went unnoticed. Finding a ruined page is now separate from being able to fix it, and when the chain is empty the log says so and names the setting that emptied it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
With `ocr.workers` set, the workers dispatch every page and never rent -- the controller owns the fleet -- but the controller only ever decided to grow inside its own `acquire()`, which it never calls while the workers serve the pages. The one signal the workers did send was keyed by POD, which is exactly what a cold fleet has none of: a worker holding pages with zero pods reported `{}`, and the controller read an idle install. So a cold OCR fleet never booted for any load, and a warm one never grew past `min_pods` -- "scale to 128 pods" was unreachable whenever the workers were on. Observed live: `workers: 2`, `min_pods: 0`, `pod_pipeline: require`, a page waiting 5.8 minutes for a pod that nothing would rent; the same request rented one within 20 seconds with `workers: 0`. Demand now travels on its own, keyed by pool rather than by pod, and the scaler pass acts on it -- single-threaded with the shedding it must not race, and still through `_grow_reason`, so max_pods and the parallel-boot caps are unchanged. Second, independent fix in the same area: under `ocr.pod_pipeline: require` a fallback chain now keeps only engines that can run on a pod. `auto` orders the chain fewest-ways-to-fail first, which put paddle and docTR -- neither of which has a pod-side pipeline -- at the front of it on an install whose whole rule is that this box never reads. The documented way out was `fallback_engines: off`, and that ALSO switched off the degraded-page recovery, which reads `auto` as "the fallback order": an install protecting its orchestrator thereby lost the mechanism written for the run that came out ~97% truncated with no error anywhere. Found live, in exactly that state. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
vLLM enables async scheduling -- CPU scheduling overlapped with GPU execution -- by default, and lists plain "ngram" among the methods it cannot run with, so asking for ngram silently turns it off: the draft is paid for with a throughput loss somewhere else, and the log says so in a warning nobody reads twice. "ngram_gpu" drafts by the same prompt-lookup, needs no model and no VRAM either, and keeps async scheduling. Both spellings work; the field now suggests the one that does not take something away. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Decode on one GPU is bound by how fast the weights can be read: measured at 30 tok/s against a ~28-30 tok/s bandwidth ceiling, it is already at the wall and no setting moves it. Prefill is not -- it is compute-bound, and on a long prompt it is nearly all of the request. So the levers that matter are the ones that avoid work rather than hurry it, and each of them now has a field of its own. Prefill batch (--max-num-batched-tokens). vLLM's default is 2048, so a 60k-token prompt is prefilled in ~30 sequential passes of small GEMMs that leave the tensor cores idle. The cost of raising it is a higher activation peak, taken out of the KV cache, so the field says to check the boot report afterwards rather than implying it is free. Prefix caching (--enable-prefix-caching), as a tri-state. vLLM's own default is per model -- on for an ordinary decoder, off for a hybrid-attention one, where it is supported but still opt-in upstream -- so "leave it alone" has to be expressible and has to be the default. For those models this switch is the only way to get it, and where the same document is asked about repeatedly it turns a whole prefill into almost nothing. Speculative decoding (--speculative-config). "ngram" is spelled out as a word because it needs no draft model, no second checkpoint and no VRAM: it drafts by finding the current text earlier in the prompt, which pays off precisely on answers that quote their source. Anything else is passed through as the JSON vLLM documents, so a real draft model needs no code change. Quantization and dtype, which were accepted by the save route but had no field anywhere. The hint says to leave quantization blank: it DECLARES how the weights on disk are packed and converts nothing, vLLM reads that from the checkpoint and picks its fastest kernel, and a repo whose name says AWQ can still be compressed-tensors -- so naming a format can only lose speed or stop the engine booting. Thinking, per model. A thinking model generates its thought first and charges it against max_tokens, so on extraction work it is most of the latency and none of the answer; on a tight budget it spends the whole allowance before the answer starts and returns an empty one. Until now only a request could turn it off, which an install does not control -- its clients do. Three states, since unset must never be read as "do not think" for a model whose answers depend on it, and an explicit request value still wins either way. The vLLM card is also shown for a model served by a vLLM POD, which is where these settings matter most, and that fixes a silent data loss: the card was hidden for `backend: runpod`, so the collector sent `vllm: {}` and the save route read an empty block as "this model has no vLLM settings" and deleted the stored one. Merely opening such a model and saving it dropped its block -- kv-cache dtype and the cache floors with it -- with no error, and nothing changed until the next pod booted in a different precision. An engine block is now sent only when its card was on screen to be edited; otherwise the key is left out, which the route ignores. The regression test fails on the previous code. Co-Authored-By:
-
- 23 Oct, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
Measured against the production pod: a reasoning model asked for 64 tokens spent all 64 inside its thought, returned NO content, and the response said finish_reason "stop". max_tokens=64 finish=stop content=0 chars '' reasoning=262 max_tokens=1500 finish=stop content=13 chars 'XMU-5635-FB' reasoning=448 Two layers were each sure the other knew better. The proxy hardcoded finish_reason = "stop" and never read choices[0].finish_reason from the remote; OpenAIFormatter then hardcoded it again, so even a caller that worked the reason out had it overwritten on the way to the client. vLLM had been saying "length" all along. It matters because "length" is the one reason a client can act on. An answer cut off at the limit is half a sentence, and told it stopped normally the client has nothing to retry on and no sign anything is missing. On a reasoning model the thought is generated FIRST and spends the same budget, so a tight limit returns an EMPTY answer that claims to have finished of its own accord -- which is how this surfaced, as a retrieval test scoring zero for reasons that had nothing to do with what it was measuring. The remote's word is now carried through the per-call sink beside reasoning, tool_calls and logprobs (get_last_finish_reason, buffered and streamed), the formatter accepts the caller's reason and keeps "stop" only as the default, and a tool call still wins -- that describes the answer's shape, and a truncation after the call was formed must not relabel it. Fixed at all five response exits, not just the one it was found at: streamed chat (both terminal-chunk paths), buffered chat, and streamed and buffered /v1/completions. A local backend is never told a reason, so the budget decides: generation that used every token it was allowed was cut off, not finished. Drawn only when a limit was actually set and reached -- never from a token count alone, or every long completion would be mislabelled. Tests verified against the reverted files: assert None == 'length' for the backends, TypeError: unexpected keyword 'finish_reason' for the formatter. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The pods page drew one bar -- host RAM -- and printed VRAM underneath it as a line of small text. On a page whose job is "what is this rented A40 doing", the GPU was the one reading an operator had to take differently from everything else on the row. Both now get a bar, each labelled, each scaled to its own total. The colours differ on purpose: host RAM keeps the green/amber/red bands, because that is the ceiling that has actually killed pods (an OCR pod was OOM-killed at 128 in-flight with the GPU never the constraint). VRAM gets a neutral bar and says "reserved", because vLLM claims its gpu_memory_utilization share at boot and so reads ~90% from the first second on every healthy pod -- run that through the load bands and every pod is permanently red, an alarm that is always on, which an operator learns to ignore and then ignores the real ones with it. Either bar is omitted when its telemetry is missing: a booting or unreachable pod has none, and an empty bar reads as "idle" when the truth is "unknown". Tested by running the page's own _memCell() in node over real payload shapes. Co-Authored-By:
-
- 22 Oct, 2026 18 commits
-
-
Stefy Lanza (nextime / spora ) authored
boot.sh has printed `profile=… image=…` since the day a pod reported a dependency missing that had already been published under its tag. It prints it before the app exists, though -- to the container's stdout and the pod's own boot log -- and the orchestrator collects a pod's log FROM THE APP, so the file an operator reads begins at "lifespan start" and the build identity is not in it. A pod serving a cached `:latest` could not be told from one serving a fresh image using the log we actually have, which is the question those variables were baked in to answer. Tonight that cost a direct answer: an FP8 KV cache proved the pod was on the new image only by inference -- FP8 cannot initialise without the precompiled kernels the old image lacks -- which is sound but is not the same as reading the tag. The app says it itself now, in its first lines, and stays silent on an install where nothing was baked. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The fleet image and the OCR image were coupled in three places, not one: the profile-parity test, and -- less visibly -- the OCR image's own build check, which read the FLEET's vLLM pin and failed the build on a mismatch. That one meant the OCR image could not be built on a different vLLM at all. They may diverge now. What replaces the coupling is the thing that actually failed last time: not that the images differed, but that nothing said so, and someone had to already suspect it. So each profile must PIN vllm and the torch stack with `==` (no floating version, so two builds of one image cannot differ either), each build check holds its own venv to its OWN pin -- which still catches the resolver substituting a version nobody asked for, a failure the cross-image check could never see -- and a choice the engine would otherwise make by version is named by the image that depends on it. That needed a mechanism, so profiles gain `<profile>.env`: KEY=VALUE baked into the image and exported by boot.sh, which PRINTS each variable it applies and lets the pod's own environment win. The OCR profile uses it to pin VLLM_ATTENTION_BACKEND=FLASH_ATTN -- vLLM's own default on Ampere, named rather than inherited, because FlashInfer is the one backend that image cannot use (its kernels compile on first use and it ships no CUDA toolkit) and which backend gets selected is a property of the vLLM VERSION. A future bump can now change throughput, never whether the pod starts. Also takes one install's model names out of three shipped files. The measurement they illustrate is what matters -- a registration overwritten, a backend fallen through to "cpu" -- not whose models they were. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
They cost ~2.9 GiB and the OCR models never ask for an FP8 KV cache, which is the only thing they are there for -- and that image is already large. So the fleet image carries them and this one does not. The parity test that caught the divergence stays, but now guards the ENGINE rather than the file: every pin deciding which vLLM (and which torch under it) runs must be identical, with this one difference named explicitly and the reasoning recorded beside it. Verified by drifting the OCR vLLM pin and watching the test fail. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
FP8 KV cache halves bytes-per-token, which is the one change that materially raises concurrency on a 48 GB card: FP16 KV on an A40 gave 212,321 tokens for this 27B hybrid (64 KiB/token), so a 128k window fitted 1.62 times. Asking for it failed. vLLM accepted the dtype and resolved it to torch.float8_e4m3fn, then the engine core died during init: FP8 KV on Ampere routes attention through FlashInfer, which JIT-compiles its prefill kernel on first use and calls nvcc. No image here ships the CUDA toolkit (the runtime, yes; and PyPI's nvidia-cuda-nvcc-cu12 wheel turns out to contain only ptxas), so flashinfer/jit/cpp_ext.py get_cuda_path() RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' flashinfer-jit-cache is upstream's package of precompiled modules for exactly this flashinfer and CUDA pair -- 959 of them, 320 with an e4m3 KV cache, head dims 64/128/256 -- and jit/env.py prefers it over compiling, so the engine never looks for a compiler. No toolkit in the image and no build stage; e5m2 is NOT in it, which is why `fp8` (= e4m3fn) is the dtype to ask for. It costs ~2.9 GiB, and a pod pulls the image on every cold start. Added to the OCR profile as well, which does not need FP8: FlashInfer is in the attention path, so prebuilt in one image and JIT in the other is two different engines on the same weights -- the invisible difference that already cost three rounds of blaming that image, and what tests/test_ocr_accent_repair.py exists to prevent. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The orchestrator already talks to a pod in parallel, and nothing in the path serializes: the front admits 256, acquiring a model instance is a ref-count, the pool hands the same pod to every caller and only GROWS at the in-flight threshold, the HTTP client keeps 64-256 connections per pod without blocking, and the blocking call runs in a 256-worker executor (measured on the production install; Python's own default there would have been 20). What was not parallel-safe is everything a backend cannot return in its text. `generate_chat` returns a string, so usage, logprobs, the tool calls a remote parsed and the thought it separated travel beside it and are read AFTER the call through `get_last_*`. Those were attributes of the backend -- and one backend object serves every concurrent request for its model. Each call began by resetting them, so a second request arriving between the first one's answer and its read blanked the first one's tool call, or handed it the second one's. That is the same shape as the empty answers seen in production on 2026-10-21, and under real parallelism it is not a rare interleaving but the common case. The store is per CALL now: a context variable, because a request is an asyncio task with its own context and `asyncio.to_thread` copies it, so the worker thread writes into the very dict the awaiting request will read. A stream captures the store when it opens, since a thread inherits no context variables, and accumulates its own thought instead of appending to a shared attribute. The backends keep their attributes as the fallback, so every caller that never opens a store -- the OCR pipeline, a script, a test -- behaves exactly as before. Tests drive two deliberately interleaved calls, and two interleaved streams, through ONE backend and assert neither sees the other's tool call, thought or usage. The suite's autouse fixtures gain one more: a store is request-scoped state, and tests share a context, so it is cleared between them like the template cache. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The two numbers that decide how much KV cache the share learner must leave an engine were constants in the module that uses them. They are now `vllm.kv_target_gb` and `vllm.kv_min_concurrency`, global or per model in its `vllm` block, on both the Settings page and the model's vLLM section — because the right answer differs per model by an order of magnitude: 2 GiB is correct for a page-at-a-time OCR model and leaves an LLM serving a 128k window unable to hold a single request. Setting the count to 0 opts out and sizes by GiB alone, which is what an OCR engine wants. The cluster page gets the same treatment the RunPod page just had: full-width wrapper and the dense table class, across all nine of its tables. The widest is ten columns, where the default 1rem of cell padding alone was 320px. Long values (a node URL, a capability list) now wrap inside their cell instead of widening the table past the window. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The share learner proposed 0.900 -> 0.460 the moment qwen38 booted on an A40: "13.27 GiB of cache on a 0.900 share leaves room to spare; 0.460 would still give ~2.0 GiB — using it from the next start". 2 GiB of this model's cache is ~32,000 tokens, measured from its own boot report (13.27 GiB bought 212,321 tokens), and the engine was serving a 131,072-token window. That is not one full request, and a vLLM whose cache cannot hold max_model_len refuses to start at all -- so the next start of that model would have failed, climbed the escalation ladder over several six-minute boots, or come up unable to serve the window it advertises. The flat 2 GiB target is right for a page-at-a-time VLM, which is what it was measured for. It is now a FLOOR, raised to whatever holds two full-length requests of the window this engine is actually serving. Bytes-per-token come from the boot report rather than the config, because they depend on the model's attention shape -- this one runs linear attention on most of its layers, at ~64 KiB/token -- and the token count is captured as it streams, like the GiB figure beside it, since both scroll out of the tail before the server is ready. With no report to read the flat target stands: a floor invented from a guess would refuse to shrink anything for ever, which is the waste this feature exists to prevent. A VLM serving 4096-token pages still hands its memory back. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Full width alone was not enough: the default cell padding is 1rem a side, which is 2rem per column and ~384px of pure padding across twelve columns -- by itself enough to push the table into a horizontal scroll on a 1280px laptop. A `.table--dense` class takes that to .4rem/.5rem. And one column was carrying nothing: "Pool" rendered "spot" or "on-demand", which the GPU cell already marks with "(spot)", while the pool's identity is in the Model column (a shared pool's rows carry the pool key there). Dropping it takes the table from twelve columns to eleven. Measured budget: ~1458px of content+padding before, ~1165px after, against a 1232px viewport once the page gutters are taken off -- so it now fits a 1280px screen instead of needing 1500px. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Twelve columns of live fleet state do not fit a 1100px reading column, so the one table an operator scans across had to be scrolled sideways to see cost, health and logs -- and the Served column added to it made that worse. It now uses the same full-width wrapper the Models page already uses, and the two meters give back 2rem each (7rem still shows "100.0/48 GB"). The header row stops wrapping, so a figure is never split across two lines. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Both of these surfaced the moment a config said 131072 while the pod was still serving the 32768 it had been launched with -- a disagreement that is normal for a few minutes after an edit, and which compaction has to survive. **The summarizer was sized from the configured window.** It chunks the history it is about to summarize to fit the summarizing model's context, and that number came straight from the backend's configured value: 131072 against a pod serving 32768, so every summarization request overflowed too. The slicer then caught the failure, left the oversized message exactly as it was, and compaction reported that it could not help. It now prefers a window an engine has already refused against, like the plan itself does. **The head's size cap was waived for its first message.** The intent was to keep at least one anchoring turn, but it meant a conversation opening with a pasted 45,000-token document kept that document verbatim as the head, dropped the rest of the history, and overflowed anyway -- the one shape where compaction matters most. An opening turn too big to keep belongs to the slicer, which summarizes its middle. Also says on the model page what the RunPod "Max context" field actually is: the window the pod is LAUNCHED with, which the general context field above it does not reach, which takes effect on the next pod rather than the running one, and which the first model to create a shared pool sets for every model on it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
With compaction on, a long conversation's streamed reply began with coderai's own progress log -- "
🗜 Compacting context (~45406+300 tok > 120% of 32768) using 'summarize'…", the slice notice, the per-chunk summarizer lines -- and only then the answer. Those went into `delta.content`, which is the text a client stores, displays and builds its next turn on. Observed on the install the moment compaction was switched on. Compaction can take a minute, so the client does need to hear something. The wait keepalive had already answered where that goes: `wait_status_mode`, silent | invisible | visible, per model or globally. Compaction now follows the same setting instead of inventing a second one -- invisible by default (the notice rides the reasoning channel when thinking is on, else an empty-content chunk with `x_compaction` metadata), silent emits an SSE comment that holds the connection without giving the client anything to render, and visible is the old behaviour for someone watching a chat UI. The same block also reported a failure as a successful answer: when compaction could not make the request fit, the error was emitted as assistant content with `finish_reason: "stop"`. The client was told the request had succeeded and handed the error as the model's reply. It is a structured error event now, like every other failure on that path. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Both of these came out of probing the live install rather than the test fixtures, and neither was visible from the code. **A streamed overflow arrived as a successful empty answer.** A remote serving an OpenAI API answers 200 the moment the stream opens, so anything that fails afterwards can only be said inside the stream -- coderai does exactly that, and deliberately, so a client treats it as an error rather than as assistant text. But that event carries no `choices`, and the proxy's SSE reader looked only at `choices[0].delta`: the event parsed to "no text", was dropped, and the stream ended. The client got HTTP 200, `finish_reason: "stop"`, the PREVIOUS request's usage and no content. This was not specific to overflow -- every mid-stream failure from a pod arrived as silence. Both proxies now raise RemoteStreamError, on "present and not None", because `{"error": {}}` is a failure while some servers put a literal `"error": null` in every ordinary chunk. **The refusal parser was written against the wrong wording.** The hand-written fixture used the older OpenAI phrasing ("you requested N tokens"); vLLM 0.27 says "you requested 300 output tokens and your prompt contains at least 32469 input tokens, for a total of at least 32769 tokens", so nothing matched and the message that was supposed to lead with the numbers led with "the request is longer than the context window" instead. The refusal a production pod actually sent is now a test fixture, wrappers and all, alongside the parameter clause ("parameter=input_tokens, value=N") that a build stating only the limit still appends. Verified on the install: `usage.context_size` now reports 32768 where it reported 2048 before, thinking still arrives as its own channel (182 chars of reasoning at 0.3s, content at 2.7s), and a 181,546-character conversation is refused with both token counts inside the first 90 characters. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Requests to a pod-served model were refused for exceeding the context window. Auto-compaction has shipped for a while, but three things stood between it and doing its job -- and one of them would have made enabling it worse than leaving it off. **The window was wrong.** A pod-served model's `ctx` lives in its runpod block and becomes the pod's `--max-model-len`, but the backend applied the generic `n_ctx` kwarg last -- and that kwarg carries the node-wide `vulkan.n_ctx` fallback, default 2048, whenever the entry sets no `n_ctx` of its own. So a model served with 32768 published 32768 on /v1/models (which reads the block) and reported 2048 in every answer's `usage.context_size`. Compaction compacts to a percentage of that number: switching it on would have crushed every conversation to ~1300 tokens and looked like a model with no memory. The block now wins, for pods and for vLLM, and a per-model `n_ctx` reaches the launch so the reported window and the served one cannot disagree. **The summary described the wrong messages.** It was built from "everything but the last two turns", which includes the turns kept verbatim, so the kept head and most of the kept tail were summarized AND kept -- paying twice in a window that was already too small. The split is now decided once, and exactly the dropped middle is summarized. The kept ends are configurable with it (head_messages/head_tokens/tail_tokens/target_pct, globally and per model), the head is reserved BEFORE the tail is filled (filling the tail first and adding the head after overshoots the target by the whole head), and a head that would end on an unanswered tool call is trimmed -- the mirror of the orphaned-result rule the tail already had. **The refusal was unreadable and final.** The numbers in it are the only exact ones in the pipeline -- coderai estimates prompts at chars/4, the engine has tokenized them -- and they were buried behind an apology, inside the pod's wrapper, inside the orchestrator's, each prefixing its own status line and 120-character proxy URL. A client truncating the result got three layers of prose and neither token count. Every layer now puts what it was told FIRST, the message leads with `context_length_exceeded:` and both integers, and a refusal is used rather than only reported: the real window is remembered for later requests, the estimator is corrected by the ratio the engine just measured, and the request is compacted against those numbers and sent once more (compaction.retry_on_overflow, on by default). Also fixes the layered fit, which mixed its units: the target was computed in real tokens and handed to a splitter that measures in estimate tokens, so a prompt the engine counts as too long could be judged already small enough and left exactly as it was. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The RunPod page showed only the instant: in-flight, KV occupancy, queue depth. Nothing said how much work a pod had done, so a healthy pod billed by the hour that had served nothing looked exactly like one carrying the fleet. Each pod now reports the requests it has finished and the prompt/generation tokens that went through it, read from the serving engine's own counters and shown as a "Served" column, with the live fleet's total beside the pod count. Two things had to be true for the number to be worth printing: * prometheus_client renames a Counter. vLLM DECLARES "vllm:num_preemptions" and serves "vllm:num_preemptions_total", alongside a "_created" sample whose value is a unix timestamp. Keying on the declared name alone silently dropped every counter -- so the preemption and prefix figures added in 0.4.69 never matched anything, and the page drew a dash that read as "not supported". Both spellings are accepted now, and "_created" is never read as a count. * vLLM's counters restart at zero with the engine process (model swap, crash, OOM kill), so a pod that had served four thousand requests would report twelve. Each drop is treated as a restart and the previous reading carried forward, which makes the column monotonic for the pod's lifetime. Also drops the prefix hit rate rather than fix its name: in vLLM 0.27.1 the prefix-cache hit counter is registered from the multimodal-cache counter, so a rate computed from it is not the prefix hit rate. A wrong number on an operator's page is worse than a dash. Verified against the shipped image's own prometheus_client, not a hand-written fixture: the exposition names and the "_created" timestamp in the tests are what ghcr.io/nextime/coderai-vllm:latest actually renders. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
digesta-alpha, reproducible on a fresh pod: [boot +0.8s] seeded models registered: qwen38-awq, diogene-alpha, digesta-alpha, qwen38-vision <- correct Resolving model: digesta-alpha [register] digesta-alpha registered in text_models <- teach, overwriting it [lora] added 'digesta-alpha' to the running engine — no reload Loading model on demand: digesta-alpha Warning: No GPU backend detected. Error loading model digesta-alpha: Unknown backend: cpu seed_model_env strips `backend` unless told the engine: a RunPod model carries `backend: "runpod"`, which is THIS install's placement and means nothing on a pod. The SEED path has passed the engine since the same fault bit diogene-alpha (the reason is written beside that call). teach_model never did. So the model was seeded correctly at boot and then RE-REGISTERED by teach -- which happens whenever an adapter has to be sent -- and the good entry was replaced by a backend-less one. It worked until something taught it, which is why it read as intermittent, and why diogene-alpha on the same pod was fine: nothing re-taught it. Orchestrator-side, so a pod already holding a broken registration can be re-taught in place rather than replaced. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The VRAM bar added in v0.4.67-68 reads ~90% on every pod from the moment it boots, because that is what it was told to do: vLLM claims its gpu_memory_utilization share up front and fills what is left after the weights with KV cache. Measured on an A40, 2026-10-22 -- 23.22 GiB weights + 3.49 activation + 2.61 CUDA graphs + 13.27 KV, out of 48, at utilisation 0.9. A bar pinned at 90% forever says nothing about load. (The model IS 4-bit; the quantization is partial by design -- the whole vision tower, lm_head, the linear_attn projections on ~48 of 63 layers and an MTP head stay FP16, which is why weights are 23 GiB and not 14.) What binds instead, and is now what the page shows: * HOST RAM, as the bar. The measured ceiling: an OCR pod went 43 GB free at 32 in-flight, 12.95 GB at 64, and was OOM-killed at 128 while the GPU was never the constraint. Read from CGROUP accounting first, because inside a container /proc/meminfo reports the HOST's memory -- a pod with a 60 GB limit on a 1 TB machine looks like it has hundreds of gigabytes free right until the kernel kills it. * KV CACHE occupancy, with running/waiting beside it: the GPU-side headroom, and what decides how many more requests can be admitted. * PREEMPTIONS, called out in red. The proof the cache ran out: vLLM evicts a running request and recomputes it later, so throughput collapses while every other counter still looks busy. Invisible in utilisation and in VRAM. Reserved VRAM stays as static text under the RAM bar -- context, not a gauge. Also removed a fiction: loaded_info reported `vram_gb: 60.58` and `device: "CPU offload 8.1 GB"` for this model. Nothing measured that -- coderai never loaded it, vLLM did, so it fell back to a config estimate for an unquantized load that never happened. A UI claiming 8 GB of CPU offload where there is none is worse than showing nothing; it now says "served by its own engine". Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
v0.4.67 asked each pod for /admin/api/gpu-stats. Pods answer 404: a pod runs the SLIM app — /healthz, /v1/models, /internal/engine-state — and no admin API at all. Built on an assumption; the live check caught it. Probe the pod before choosing its endpoint. /internal/engine-state already carries a `vram` block: used, free and total per card with the GPU's name, built context-free and cached 4s there. It is what the front health-polls, so every pod has it and it costs the pod nothing extra. Verified against a live A40: 43.8/48 GB with qwen38 resident. The pod reports FREE, not used — that is what a context-free query returns — so used is the difference. Inverting that would show an idle pod as full. Utilisation and temperature added AT THE SOURCE: nvidia-smi now asks for utilization.gpu and temperature.gpu in the same call as the memory query rather than paying for a second one (a saturated card answers slowly enough that one call matters), with the pynvml equivalent, and both ride into engine-state's vram.devices. Absent or "[N/A]" is left out, never zeroed: a zero draws a bar that reads as "idle" when the truth is "not known". So VRAM works on every pod already running; utilisation needs the next pod image, and until then that column shows a dash — which is the degradation the page was built for. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The page listed what each pod COSTS and nothing about what it is doing, so a saturated pod and an idle one looked identical next to the same $/hr. Read from the PODS, not from RunPod. A pod runs coderai, so it already serves /admin/api/gpu-stats -- per-card util, VRAM used/total and temperature, collected with nvidia-smi off its own event loop and cached there for 1.5s. RunPod reports utilisation too, but it lags by tens of seconds and every read spends the quota that renting and boot-polling need: a 429 there is what stopped the fleet scaling for 3.3h on 2026-10-19. Telemetry must never be able to cost a fleet its ability to rent. Utilisation and temperature are the BUSIEST card, not the mean -- on a multi-card pod the saturated card decides whether it can take more work, and averaging it with an idle sibling hides exactly that condition. VRAM is summed, being the pool a model is loaded into. Bounded everywhere, because this is the page an operator opens when something is already wrong: 2.5s per pod, all pods concurrently, a 4s TTL so several open tabs collapse onto one round, and a 3s cap on the round -- a slow pod drops out and the rest still render. The spend figures beside it stay right when the telemetry does not arrive, and a pod that is booting or unreachable shows a dash, never a zero: a zeroed bar reads as "idle" when the truth is "unknown". held_pod_endpoints() is deliberately separate from pods_status(): that snapshot is serialised into a browser and a pod's bearer must not be. Covered by a test. Co-Authored-By:
-