- Oct 28, 2026
-
-
Stefy Lanza (nextime / spora ) authored
`precision` was read by the loader and listed in main.py's _LOAD_AFFECTING, but it was in neither the admin save allowlist nor the model page, so the one setting that decides how much VRAM a card actually needs could not be configured anywhere. The dtype then came from a default derived from GPU detection -- and a card's precision is a property of the card, not of the hardware it happens to land on. The select sits with the quantization boxes because that is where it is read wrongly: 4-bit and 8-bit quantize the WEIGHTS, while the embeddings, lm_head, the norms, the activations and the KV cache all stay in this dtype, so a 4-bit model left in fp32 still pays fp32 for every one of them. The hint says so, and says that fp32 also rules out Flash Attention. "" is stored as null, so "not configured" stays distinguishable from a value and the loader's own default still applies. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZnLhydX9QqUcfBbqLu
-
Stefy Lanza (nextime / spora ) authored
Two things that were only reachable from a python -c. The lip-sync engines had no management surface at all: no status, no install, no removal. They are not models.json entries on purpose -- each runs in its own interpreter as a subprocess, so a row in a model section would put them in the VRAM ledger as something the manager can load, which it cannot -- but that is a reason to give them their own surface, not none. So: a tab on the models page, three admin endpoints behind require_admin like every other model action, and an install that runs in the background while the page polls. The log is kept because when an install fails, WHERE it failed is the whole answer: a wheel that will not build reads nothing like a 401. "Rebuild venv" keeps the weights, since a dependency change does not alter six gigabytes. `component_models` could only name a repo. A community fine-tune is very often ONE file -- a retrained transformer with no repo around it, the base supplying the VAE, the text encoder and the scheduler -- and from_pretrained cannot open one, so such a component could only be named if somebody had published a converted directory tree for it. It now accepts a single-file or civitai: reference. And the Civitai resolver picked the LARGEST file of a version. Real Dream's Krea 2 V3 publishes the same weights three times: fp8 13.46 GB, int8 13.18 GB, nvfp4 8.12 GB. Largest-wins lands on fp8 by a 0.28 GB margin nobody chose, and could as easily have landed on nvfp4, which needs Blackwell and cannot load on an Ampere card at all. A reference may now name the file, and without one the choice is made among the builds this GPU can run, by precision rather than size. Two smaller ones found by using it. urllib replays headers across a redirect, so Civitai's signed R2 URL received a bearer token as well and answered 400 -- which looks exactly like a bad token, and meant adding a token BROKE downloads that had worked without one. curl strips it; now so does this. And the cache has its own root (CODERAI_CIVITAI_CACHE) instead of living under the HF cache, which is how a second GGUF directory grew inside one. Co-Authored-By:
-
- Oct 25, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Two asks that turned out to be about the same thing: a request taken as meant. A model fine-tuned on a quality ladder needs that ladder on every request, and a client speaking the plain OpenAI image API sends a sentence with nowhere to put it. CoderAI adds no tags of its own and still does not: prompt_prefix and negative_prompt_prefix on the MODEL's config entry are comma-joined in front of whatever the caller sent, and both default to empty, so an install that configures nothing sends exactly what was written. A request overrides either, and an empty string switches one off for that call -- what a caller writing its own complete prompt needs. Applied at most once: a prompt that already opens with the ladder is left alone rather than given it twice, which would only dilute the rest. The prefix resolves once, before the encode, and the same text feeds the embedding cache key. Keyed on the prompt as SENT, two models with different prefixes would share one entry and the second would be served the first's text. The archive keeps the prompt that ran plus the one requested when they differ. On OCR: there is no per-page byte limit, and there never was. A page over a megabyte is ordinary -- an A4 scan at 300 DPI is several -- and both a 3 MB and a 50 MB body upload completely and are rejected only at authentication, with the bundled nginx at 4096m and the generated /v1/ocr routing not lowering it. What did exist was a hardcoded 100 MB ceiling on one document, which a 300 DPI colour filing passes without being unusual and which nothing could raise. It is ocr.max_upload_mb now (0 = no ceiling here), in Settings, and its 413 names the setting and says the document was NOT read -- the loud refusal ocr.max_pages already makes, instead of "file too large (max 100 MB)". The 1 MB figure that makes a large page fail in the field is nginx's own client_max_body_size default, in a reverse proxy added in front. That is exactly the shape of "anything over a megabyte fails", so it is recorded in the code, the settings hint and docs/ocr.md rather than left to be rediscovered. Also documented there: what IS capped is pixels, not bytes -- Surya-2 and Chandra top out near 6.29 MP, an A4 page at about 255 DPI, and above that the page is silently downscaled to fit. 37 tests, including a real 1.25 MB single-page PDF built from noise and read back through pdfium as one page. Full suite 4512 passed with the same 15 pre-existing failures as 0.4.90 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three things the image endpoints accepted and then quietly did something else with. size. A size that did not parse fell through to the default, so "1024" or "1k x 1k" generated a square nobody asked for and the response looked like a success. A side that was not a multiple of 8 went to the pipeline as-is, where the VAE cannot represent it. Nothing capped it either, so 4096x4096 was accepted and came back as a CUDA OOM -- which reads like a busy GPU rather than a size the caller chose. Now: a 400 quoting what was sent, a 400 for a zero dimension (snapping it up would answer an uninitialised client variable with an 8x8 image), snapping to the grid with a log line, and a 400 naming image.max_side / image.max_pixels when it is over the ceiling. "auto" is accepted as in the OpenAI API, and the dead `width != width` NaN guard on a value that was already an int is gone. The archive records the size actually generated alongside the one requested. /v1/images/inpaint no longer defaults to 1024x1024, which squashed every non-square photo and pulled the mask out of alignment with it -- worse than the same default on /v1/images/edits, where it was fixed in 0.4.86. Omitted now keeps the source's shape. Textual inversions. easynegative, bad-hands-5 and the rest are vectors living in CLIP's embedding table under a word of their own, and until the file is loaded that word is an ordinary unknown token -- so a large part of the ecosystem's negative prompting asked for nothing at all. They load from the request (embeddings: [...]), from the model's config (textual_inversion), or by filename from image.embeddings_dir, which is how A1111 works and what a prompt copied off a model card assumes. Two details decide whether this works at all: an SDXL-format embedding holds a clip_l and a clip_g tensor that must go into the matching tokenizer/encoder pair, and a multi-vector embedding occupies several tokens (tok, tok_1, tok_2...) that the prompt has to be expanded into -- without it an eight-vector embedding contributes one vector and appears inert. A name the caller gave that will not load is a 400; a bad file in the directory is a line in the log, because one file should not break every request. /v1/images/outfit goes through the same encoder now, so the weight grammar and clip_skip reach it too. Its per-frame video path encodes the prompt once for the clip instead of once per frame. And a 400 is no longer reported as a failed backend. On the first request for a model that is not loaded yet, the generate call sits inside the try that falls back from diffusers to sd.cpp, so a rejected request was caught as a load failure and answered with a report about which backends failed. The three edit endpoints did the same through their catch-alls. All four re-raise HTTPException untouched. Found by the tests while writing them: the embeddings directory was searched by the renamed trigger word rather than the filename, so "easynegative:myneg" never resolved, and "768X512" did not parse. 64 new tests; full suite 4475 passed with the same 15 pre-existing failures as 0.4.89 (diff empty). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A CLIP text encoder reads 77 tokens, two of them markers. Handing a longer prompt to a diffusers pipeline does not fail: the tokenizer truncates it and the model renders what fits. The booru-tag fine-tune families (Pony, Illustrious, NoobAI) open with a run of quality tags, so the part that got dropped was the subject -- and the request answered 200 with an image, which is why it was easy to miss. Image prompts are now encoded here rather than by the pipeline, which is what makes the rest possible: * as many 75-token windows as the prompt needs, concatenated -- no limit * the A1111 weight grammar: (tag:1.3), [tag], bare (tag) = 1.1, nesting multiplies, \( escapes. Per-token scaling with the window mean restored, so emphasis does not drag the brightness along with it * BREAK, which closes the current window instead of blending tags meant for different subjects * clip_skip, mapped to diffusers' own convention -- SDXL counts from the penultimate layer, SD 1.x from the final output and needs the layer norm applied by hand. Getting that backwards shifts every image Three places were dropping the negative prompt outright. The sd.cpp path hardcoded negative_prompt='', so for every GGUF image model whatever the caller asked to avoid stopped at the API -- half the idiom, since the quality tags have a negative half (score_4, score_5, score_6). /v1/images/edits and /v1/images/inpaint had no field for one at all, so an edit could only ever be steered positively. All three now pass it, and take clip_skip and prompt_weighting with it. clip_skip and prompt_weighting are also per-model and server-wide settings, because a client speaking the plain OpenAI image API has no field to ask for clip_skip and a model trained at 2 needs it on every request. Request beats model beats global. Weighting defaults on, so a literal '(' in prose now weighs 1.1 -- the ecosystem's convention, with \( and prompt_weighting:false as the way out. What that cannot be allowed to change is prompts that already worked: against a real CLIPTextModel a short unweighted prompt encodes bit-identically to what diffusers produced, because CLIP pads with its own end-of-text token, so [bos]+toks+[eos]+pad... and [bos]+toks+pad...+[eos] are the same ids. The test records that reasoning, so a tokenizer where it stops holding is caught. Flux, SD3 and Z-Image carry a T5 with its own limit and conventions and keep their pipeline's native text path. 49 tests; full suite 4410 passed with the same 15 pre-existing failures as 0.4.88 (diff empty). Co-Authored-By:
-
- Oct 24, 2026
-
-
Stefy Lanza (nextime / spora ) authored
vLLM enables async scheduling -- CPU scheduling overlapped with GPU execution -- by default, and lists plain "ngram" among the methods it cannot run with, so asking for ngram silently turns it off: the draft is paid for with a throughput loss somewhere else, and the log says so in a warning nobody reads twice. "ngram_gpu" drafts by the same prompt-lookup, needs no model and no VRAM either, and keeps async scheduling. Both spellings work; the field now suggests the one that does not take something away. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Decode on one GPU is bound by how fast the weights can be read: measured at 30 tok/s against a ~28-30 tok/s bandwidth ceiling, it is already at the wall and no setting moves it. Prefill is not -- it is compute-bound, and on a long prompt it is nearly all of the request. So the levers that matter are the ones that avoid work rather than hurry it, and each of them now has a field of its own. Prefill batch (--max-num-batched-tokens). vLLM's default is 2048, so a 60k-token prompt is prefilled in ~30 sequential passes of small GEMMs that leave the tensor cores idle. The cost of raising it is a higher activation peak, taken out of the KV cache, so the field says to check the boot report afterwards rather than implying it is free. Prefix caching (--enable-prefix-caching), as a tri-state. vLLM's own default is per model -- on for an ordinary decoder, off for a hybrid-attention one, where it is supported but still opt-in upstream -- so "leave it alone" has to be expressible and has to be the default. For those models this switch is the only way to get it, and where the same document is asked about repeatedly it turns a whole prefill into almost nothing. Speculative decoding (--speculative-config). "ngram" is spelled out as a word because it needs no draft model, no second checkpoint and no VRAM: it drafts by finding the current text earlier in the prompt, which pays off precisely on answers that quote their source. Anything else is passed through as the JSON vLLM documents, so a real draft model needs no code change. Quantization and dtype, which were accepted by the save route but had no field anywhere. The hint says to leave quantization blank: it DECLARES how the weights on disk are packed and converts nothing, vLLM reads that from the checkpoint and picks its fastest kernel, and a repo whose name says AWQ can still be compressed-tensors -- so naming a format can only lose speed or stop the engine booting. Thinking, per model. A thinking model generates its thought first and charges it against max_tokens, so on extraction work it is most of the latency and none of the answer; on a tight budget it spends the whole allowance before the answer starts and returns an empty one. Until now only a request could turn it off, which an install does not control -- its clients do. Three states, since unset must never be read as "do not think" for a model whose answers depend on it, and an explicit request value still wins either way. The vLLM card is also shown for a model served by a vLLM POD, which is where these settings matter most, and that fixes a silent data loss: the card was hidden for `backend: runpod`, so the collector sent `vllm: {}` and the save route read an empty block as "this model has no vLLM settings" and deleted the stored one. Merely opening such a model and saving it dropped its block -- kv-cache dtype and the cache floors with it -- with no error, and nothing changed until the next pod booted in a different precision. An engine block is now sent only when its card was on screen to be edited; otherwise the key is left out, which the route ignores. The regression test fails on the previous code. Co-Authored-By:
-
- Oct 23, 2026
-
-
Stefy Lanza (nextime / spora ) authored
The pods page drew one bar -- host RAM -- and printed VRAM underneath it as a line of small text. On a page whose job is "what is this rented A40 doing", the GPU was the one reading an operator had to take differently from everything else on the row. Both now get a bar, each labelled, each scaled to its own total. The colours differ on purpose: host RAM keeps the green/amber/red bands, because that is the ceiling that has actually killed pods (an OCR pod was OOM-killed at 128 in-flight with the GPU never the constraint). VRAM gets a neutral bar and says "reserved", because vLLM claims its gpu_memory_utilization share at boot and so reads ~90% from the first second on every healthy pod -- run that through the load bands and every pod is permanently red, an alarm that is always on, which an operator learns to ignore and then ignores the real ones with it. Either bar is omitted when its telemetry is missing: a booting or unreachable pod has none, and an empty bar reads as "idle" when the truth is "unknown". Tested by running the page's own _memCell() in node over real payload shapes. Co-Authored-By:
-
- Oct 22, 2026
-
-
Stefy Lanza (nextime / spora ) authored
The two numbers that decide how much KV cache the share learner must leave an engine were constants in the module that uses them. They are now `vllm.kv_target_gb` and `vllm.kv_min_concurrency`, global or per model in its `vllm` block, on both the Settings page and the model's vLLM section — because the right answer differs per model by an order of magnitude: 2 GiB is correct for a page-at-a-time OCR model and leaves an LLM serving a 128k window unable to hold a single request. Setting the count to 0 opts out and sizes by GiB alone, which is what an OCR engine wants. The cluster page gets the same treatment the RunPod page just had: full-width wrapper and the dense table class, across all nine of its tables. The widest is ten columns, where the default 1rem of cell padding alone was 320px. Long values (a node URL, a capability list) now wrap inside their cell instead of widening the table past the window. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Full width alone was not enough: the default cell padding is 1rem a side, which is 2rem per column and ~384px of pure padding across twelve columns -- by itself enough to push the table into a horizontal scroll on a 1280px laptop. A `.table--dense` class takes that to .4rem/.5rem. And one column was carrying nothing: "Pool" rendered "spot" or "on-demand", which the GPU cell already marks with "(spot)", while the pool's identity is in the Model column (a shared pool's rows carry the pool key there). Dropping it takes the table from twelve columns to eleven. Measured budget: ~1458px of content+padding before, ~1165px after, against a 1232px viewport once the page gutters are taken off -- so it now fits a 1280px screen instead of needing 1500px. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Twelve columns of live fleet state do not fit a 1100px reading column, so the one table an operator scans across had to be scrolled sideways to see cost, health and logs -- and the Served column added to it made that worse. It now uses the same full-width wrapper the Models page already uses, and the two meters give back 2rem each (7rem still shows "100.0/48 GB"). The header row stops wrapping, so a figure is never split across two lines. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Both of these surfaced the moment a config said 131072 while the pod was still serving the 32768 it had been launched with -- a disagreement that is normal for a few minutes after an edit, and which compaction has to survive. **The summarizer was sized from the configured window.** It chunks the history it is about to summarize to fit the summarizing model's context, and that number came straight from the backend's configured value: 131072 against a pod serving 32768, so every summarization request overflowed too. The slicer then caught the failure, left the oversized message exactly as it was, and compaction reported that it could not help. It now prefers a window an engine has already refused against, like the plan itself does. **The head's size cap was waived for its first message.** The intent was to keep at least one anchoring turn, but it meant a conversation opening with a pasted 45,000-token document kept that document verbatim as the head, dropped the rest of the history, and overflowed anyway -- the one shape where compaction matters most. An opening turn too big to keep belongs to the slicer, which summarizes its middle. Also says on the model page what the RunPod "Max context" field actually is: the window the pod is LAUNCHED with, which the general context field above it does not reach, which takes effect on the next pod rather than the running one, and which the first model to create a shared pool sets for every model on it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Requests to a pod-served model were refused for exceeding the context window. Auto-compaction has shipped for a while, but three things stood between it and doing its job -- and one of them would have made enabling it worse than leaving it off. **The window was wrong.** A pod-served model's `ctx` lives in its runpod block and becomes the pod's `--max-model-len`, but the backend applied the generic `n_ctx` kwarg last -- and that kwarg carries the node-wide `vulkan.n_ctx` fallback, default 2048, whenever the entry sets no `n_ctx` of its own. So a model served with 32768 published 32768 on /v1/models (which reads the block) and reported 2048 in every answer's `usage.context_size`. Compaction compacts to a percentage of that number: switching it on would have crushed every conversation to ~1300 tokens and looked like a model with no memory. The block now wins, for pods and for vLLM, and a per-model `n_ctx` reaches the launch so the reported window and the served one cannot disagree. **The summary described the wrong messages.** It was built from "everything but the last two turns", which includes the turns kept verbatim, so the kept head and most of the kept tail were summarized AND kept -- paying twice in a window that was already too small. The split is now decided once, and exactly the dropped middle is summarized. The kept ends are configurable with it (head_messages/head_tokens/tail_tokens/target_pct, globally and per model), the head is reserved BEFORE the tail is filled (filling the tail first and adding the head after overshoots the target by the whole head), and a head that would end on an unanswered tool call is trimmed -- the mirror of the orphaned-result rule the tail already had. **The refusal was unreadable and final.** The numbers in it are the only exact ones in the pipeline -- coderai estimates prompts at chars/4, the engine has tokenized them -- and they were buried behind an apology, inside the pod's wrapper, inside the orchestrator's, each prefixing its own status line and 120-character proxy URL. A client truncating the result got three layers of prose and neither token count. Every layer now puts what it was told FIRST, the message leads with `context_length_exceeded:` and both integers, and a refusal is used rather than only reported: the real window is remembered for later requests, the estimator is corrected by the ratio the engine just measured, and the request is compacted against those numbers and sent once more (compaction.retry_on_overflow, on by default). Also fixes the layered fit, which mixed its units: the target was computed in real tokens and handed to a splitter that measures in estimate tokens, so a prompt the engine counts as too long could be judged already small enough and left exactly as it was. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The RunPod page showed only the instant: in-flight, KV occupancy, queue depth. Nothing said how much work a pod had done, so a healthy pod billed by the hour that had served nothing looked exactly like one carrying the fleet. Each pod now reports the requests it has finished and the prompt/generation tokens that went through it, read from the serving engine's own counters and shown as a "Served" column, with the live fleet's total beside the pod count. Two things had to be true for the number to be worth printing: * prometheus_client renames a Counter. vLLM DECLARES "vllm:num_preemptions" and serves "vllm:num_preemptions_total", alongside a "_created" sample whose value is a unix timestamp. Keying on the declared name alone silently dropped every counter -- so the preemption and prefix figures added in 0.4.69 never matched anything, and the page drew a dash that read as "not supported". Both spellings are accepted now, and "_created" is never read as a count. * vLLM's counters restart at zero with the engine process (model swap, crash, OOM kill), so a pod that had served four thousand requests would report twelve. Each drop is treated as a restart and the previous reading carried forward, which makes the column monotonic for the pod's lifetime. Also drops the prefix hit rate rather than fix its name: in vLLM 0.27.1 the prefix-cache hit counter is registered from the multimodal-cache counter, so a rate computed from it is not the prefix hit rate. A wrong number on an operator's page is worse than a dash. Verified against the shipped image's own prometheus_client, not a hand-written fixture: the exposition names and the "_created" timestamp in the tests are what ghcr.io/nextime/coderai-vllm:latest actually renders. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The VRAM bar added in v0.4.67-68 reads ~90% on every pod from the moment it boots, because that is what it was told to do: vLLM claims its gpu_memory_utilization share up front and fills what is left after the weights with KV cache. Measured on an A40, 2026-10-22 -- 23.22 GiB weights + 3.49 activation + 2.61 CUDA graphs + 13.27 KV, out of 48, at utilisation 0.9. A bar pinned at 90% forever says nothing about load. (The model IS 4-bit; the quantization is partial by design -- the whole vision tower, lm_head, the linear_attn projections on ~48 of 63 layers and an MTP head stay FP16, which is why weights are 23 GiB and not 14.) What binds instead, and is now what the page shows: * HOST RAM, as the bar. The measured ceiling: an OCR pod went 43 GB free at 32 in-flight, 12.95 GB at 64, and was OOM-killed at 128 while the GPU was never the constraint. Read from CGROUP accounting first, because inside a container /proc/meminfo reports the HOST's memory -- a pod with a 60 GB limit on a 1 TB machine looks like it has hundreds of gigabytes free right until the kernel kills it. * KV CACHE occupancy, with running/waiting beside it: the GPU-side headroom, and what decides how many more requests can be admitted. * PREEMPTIONS, called out in red. The proof the cache ran out: vLLM evicts a running request and recomputes it later, so throughput collapses while every other counter still looks busy. Invisible in utilisation and in VRAM. Reserved VRAM stays as static text under the RAM bar -- context, not a gauge. Also removed a fiction: loaded_info reported `vram_gb: 60.58` and `device: "CPU offload 8.1 GB"` for this model. Nothing measured that -- coderai never loaded it, vLLM did, so it fell back to a config estimate for an unquantized load that never happened. A UI claiming 8 GB of CPU offload where there is none is worse than showing nothing; it now says "served by its own engine". Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The page listed what each pod COSTS and nothing about what it is doing, so a saturated pod and an idle one looked identical next to the same $/hr. Read from the PODS, not from RunPod. A pod runs coderai, so it already serves /admin/api/gpu-stats -- per-card util, VRAM used/total and temperature, collected with nvidia-smi off its own event loop and cached there for 1.5s. RunPod reports utilisation too, but it lags by tens of seconds and every read spends the quota that renting and boot-polling need: a 429 there is what stopped the fleet scaling for 3.3h on 2026-10-19. Telemetry must never be able to cost a fleet its ability to rent. Utilisation and temperature are the BUSIEST card, not the mean -- on a multi-card pod the saturated card decides whether it can take more work, and averaging it with an idle sibling hides exactly that condition. VRAM is summed, being the pool a model is loaded into. Bounded everywhere, because this is the page an operator opens when something is already wrong: 2.5s per pod, all pods concurrently, a 4s TTL so several open tabs collapse onto one round, and a 3s cap on the round -- a slow pod drops out and the rest still render. The spend figures beside it stay right when the telemetry does not arrive, and a pod that is booting or unreachable shows a dash, never a zero: a zeroed bar reads as "idle" when the truth is "unknown". held_pod_endpoints() is deliberately separate from pods_status(): that snapshot is serialised into a browser and a pod's bearer must not be. Covered by a test. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
From ONE pod, ONE set of weights, behind two aliases: qwen38 -> clean answer diogene-alpha -> "The user is asking to list hearings from 2026-10-21 to..." The entire chain of thought went to the client as the answer. Four breaks: 1. THE GATE ASKED THE NAME. _reasoning_active was a regex (qwen3|qwq| deepseek-r1|...) over the request's model name: `qwen3` matches "qwen38" and not "diogene-alpha", so one alias of one model separated its thought and the other did not. It is read from the template now -- whose generation prompt ends with '<think>\n', so the model starts INSIDE the thought and emits a bare </think>, the form that leaks when nothing looks for it. The regex stays as a last resort for a template that cannot be read. 2. THE BUFFERED FORMATTER TOOK NO `reasoning`. Where the thought WAS separated it was then dropped: no leak, but no thinking either. litellm's Message has no such field, so it is attached after the dump rather than lost -- under both spellings, since Qwen's own template reads `reasoning_content`. 3. THE PROXY HOP CARRIED NEITHER FORM. Buffered, the remote sends reasoning_content beside an already-cleaned content, which the string contract cannot hold: get_last_reasoning(), like get_last_tool_calls(). Streamed, it sends reasoning DELTAS, and dropping them meant a client saw nothing at all until the answer began -- so the <think>/</think> tags go back on as they arrive and the caller's existing reasoning gate splits them live. An all-thinking stream leaves the tag open on purpose: the gate flushes an unclosed thought as reasoning, and there is no content for a close tag to precede. 4. THINKING COULD NOT BE TURNED OFF. enable_thinking defaulted to False, so "unset" and "explicitly off" were the same value, and the field sat in _CHAT_CONSUMED_FIELDS while no backend forwarded it. It defaults to None now: unset sends nothing and the model keeps its own default (a thinking model thinks), an explicit value travels as chat_template_kwargs. That needed the POD backend to gain passthrough at all -- the local vLLM backend has forwarded unknown fields for releases, a pod got none of them, so chat_template_kwargs could never reach a pod-served model. ChatMessage silently dropped `reasoning_content`, so a previous turn's thinking never reached the template that re-renders it inside <think>. Accepted and forwarded now. The chat page hardcoded stream:false, so thinking could never arrive live there, and it read only message.content. It has `stream` and `thinking` toggles (both on), renders the thought in a block that fills AS IT ARRIVES and collapses when the answer starts, and keeps it in the history it replays. Streaming is the request's choice, not forced: cleared, it takes the buffered path exactly as before. Co-Authored-By:
-
- Oct 19, 2026
-
-
Stefy Lanza (nextime / spora ) authored
/v1/runpod/status is the one surface an operator has for seeing WHY a server is busy. During a pod boot wave it was both unanswerable and, when it did answer, wrong about the only number anyone was watching. IT COULD NOT ANSWER. Measured on a live orchestrator: /healthz 200 in 2.9 ms (no thread needed) /metrics 200 in 180 ms (no thread needed) /v1/runpod/status no answer in 25 s (needs a thread) /v1/runpod/scale no answer in 25 s (needs a thread) Nothing was broken. Those two are the only endpoints that hand work to a thread, and they drew it from the DEFAULT executor -- the pool the request path also draws on. 25 s of silence reads as a dead server, and it also meant a client lease could not be RELEASED while the fleet was busy, so pods kept billing with no way to stop them but the console. Three changes, each closing one way that happened: * a CONTROL pool (16 threads, fixed) that nothing in the request path may use -- status, scale and release run there. Its whole value is being empty when everything else is full, and a test pins that no request-path module touches it. * the PDF split moves to a two-wide PDFIUM pool. pdfium admits exactly one caller, so hundreds of concurrent documents each took a general thread and then queued on the same global lock: the serialisation was unchanged while the threads they held were the control plane's. Queueing two-wide costs nothing that was not already being paid. * the snapshot's three sources now run concurrently, each with its own 5 s deadline, and whatever is late is reported as `degraded` rather than waited for. pools_schedule_status() was the half never bounded in v0.4.21 -- pods_status() gave up on a busy pool and this one waited for it -- and the ledger snapshot reads without the lock it shares with the writer that fires on every termination. Enforcement still takes that lock properly: only the snapshot may degrade. IT COULD NOT BE BELIEVED. `live_hourly_usd` summed only HEALTHY pods. RunPod bills from creation and a pod spends ~12 minutes booting without ever being healthy, so a fleet entirely in paid boot reported as free: reported $0.59/hr actual $4.72/hr reported $0.00/hr actual $7.67/hr (13 pods, 8 serving) A client polling it concluded nothing was being spent and kept feeding work. It now counts every pod in the listing, and reports the serving and booting halves separately -- their ratio is how much of the bill is boot, which is the figure that says a ramp is too steep, and it did not exist before. Both admin pages show it; "0 pod(s) · $7.67/hr" was a real reading and is self-contradictory. 29 tests. Full suite 3741 passed, 15 pre-existing failures unchanged. Co-Authored-By:
-
- Oct 16, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Three things, and the third is the reason the first two were impossible. **A mixed fleet was distributed as if every pod were the same.** `PodPool._pick` was `min(free, key=inflight)` and growth was `load / scale_up_inflight_per_pod` -- both card-blind. An A100 and an A40 were handed the same number of pages, so the fast card drained its share and idled while the slow one straggled; and a document completes only when its LAST page lands, so the slowest pod set the latency the client saw. The scaler had the matching fault in reverse: a fleet of bigger cards was still sized as if each were the preferred one, which is the same arithmetic that reached 94 pods at 10 % utilisation. Now: weighted least-connections, an in-flight cap scaled per pod, and growth that divides demand by the fleet's real capacity. **`gpu_limits`: per-card numbers, nothing multiplied.** "gpu_limits": { "NVIDIA A100-SXM4-80GB": {"parallel": 186, "queue": 744, "scale_up": 93}, "NVIDIA A40": {"parallel": 64, "queue": 256, "scale_up": 32} } A scaled single number cannot stay right: the model-level figures describe ONE card, and which card that was came from the first entry of `gpu_types` -- so changing the RENTAL preference silently re-scaled the concurrency, and a figure measured on an A40 came to describe an A100 by a factor nobody had measured. A table states what was measured, per card. The share of load is DERIVED from it (two cards with depths 32 and 93 are given work in that ratio) rather than configured twice, with `gpu_weights` and then the VRAM ratio as fallbacks for cards nobody has measured yet. Editable per model in the admin UI; a row with a card name and no numbers is dropped, because "not measured yet" must not be saved as a limit of zero. `parallel`/`queue` are applied AFTER the candidate card is chosen. The launch plan is built before, so its env carried the model-level number whatever card won -- and the gate is env, set once, so a pod rented on a 3x card spent its whole life admitting a 1x card's work. **An adopted pod recorded `gpu="(adopted)"`.** A pool that cannot name a pod's card cannot size it, weight it, or report the fleet: a whole day's status output read `by_gpu: {"(adopted)": 32}`. `list_pods`/`get_pod` now ask RunPod for `gpuCount` and `machine { gpuDisplayName gpuTypeId gpuType { memoryInGb } }` (every field probed against the live schema first), the registry stores the card and its size so adoption needs no API call, and an unreadable card costs a weight of 1.0, never a pod. **And a catalogue entry could crash an engine.** Adding a `vision_models` row for a model served on RunPod quarantined the cpu engine for ninety minutes: its id is not a path, not a URL and not a repo id, so resolution fell through to "assume a repo id", then to "assume a URL", and `requests.get("qwen38-vision")` raised MissingSchema -- five exits in 120 s, QUARANTINED, taking /v1/ocr and every pool with it while `/healthz` still answered `ok: true` because the FRONT was up. Two independent guards: * a model pinned to a backend that serves it ELSEWHERE is never resolved, downloaded or cached here -- which is also what lets one row run on rented pods and load locally on a box with a GPU, the pin being the whole difference and the section not overriding it; * a name that is none of the three resolvable kinds is refused with a sentence naming all three and pointing at `backend`. The pin does NOT make local weights useless: `resolve_model_source` is configurable (`auto | hf | url | upload`) and reads the entry directly, so an id still means "the pod downloads it" and a local path still means "push it". Tested, so a later change cannot couple the two. 43 new tests. 3411 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
**The accent regression, reported from production hours after 0.4.7 shipped.** Italian accents stopped surviving surya again. The cause was this afternoon's scan passthrough, not the old detokenisation fault: its guard used an ABSOLUTE 150-DPI floor instead of the DPI the page would have been rendered at. surya's own default is 250 (`surya_dpi`), and Aruba leaves it unset, so every scanned page holding a 150-200 DPI JPEG went to the model at native size -- 25-40 % less linear resolution than the render it replaced -- and surya-ocr-2 is measured to read badly below 250. The reasoning error is the part worth keeping: passing a scan through was always meant to be "the same pixels, without the work". Fewer pixels is not a saving. The floor is now the TARGET DPI, and `page_image_min_dpi` (default 0) can only RAISE it. Verified against a 250-DPI engine: a 150 or 200 DPI scan is rendered up as before; 250 and 300 DPI scans still pass through, which is where the shortcut was always legitimate. It also honours the REQUEST. `ocr_document` resolves `use_dpi = int(dpi) if dpi else engine_dpi(engine)` and hands that one value to the pod forward, the local render and (on the pod) this guard -- so "is this scan detailed enough" is asked about what the caller asked for, never about a default they overrode. Pinned, because a future change reading the engine default would silently hand a 200-DPI scan to a request that wanted 300. **And the rasterisation hop now encodes, instead of shipping raw pixels.** Measured on A4 at 250 DPI (6.04 Mpix, surya's own default): raw RGB over the pipe 18.1 MB + 109.7 ms for the parent to encode it for the engine anyway PNG level 1 in the worker 0.72 MB 104.7 ms, in the WORKER the parent's lazy open 0.12 ms The encode is not added, it MOVES: off the single parent process onto the raster pool, which is parallel. The pipe carries 25x less, the second encode disappears, and the parent holds 0.72 MB per page in flight instead of 18.1 MB -- at 48 pages that is 35 MB instead of 870 MB, which is the quantity this whole subsystem keeps getting into trouble over. PNG, not JPEG, unless the operator asked: a page about to be read must not pass through a lossy codec by default. Raw RGB stays the fallback, so the protocol still reads on an older parent. Seven existing tests had encoded the wrong floor and were corrected; one of them hardcoded pixel counts that assumed A4 while the fixture is 609 pt, so it measured 244 DPI while claiming 250. 11 new tests. 3368 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.4.8 moved the forward off the default 32-thread pool and the measured result on the live orchestrator was 33 -> 483 pages in flight, 1.65-2.2 -> 7.99 pages/s, and a control-plane request that used to time out at 40 s answering in 12 ms. But a thread is still the wrong unit for the wait: ~60 s per page against a real corpus means 15 documents/second at ~3.5 pages each is ~3,200 outstanding forwards, against a pod-pool ceiling of 2048. Threads cannot get there, and should not try. So the hot path is a coroutine. `pod_http` gains `apost`/`aget` on httpx -- already in the stack (codai/cluster/nodes.py, codai/main.py), so no new dependency -- with everything the sync path learned kept intact: * the pinned TLS context, unchanged: a direct-TCP pod serves HTTPS with a certificate signed by THIS install's CA at an ip the certificate cannot name, so the CA is trusted and the hostname check is off. NOT verify=False, which would accept anybody; * one client per event loop AND per pinned-ness, because a client belongs to the loop its sockets were opened on and the two trust settings must never share one; * connection limits from the same `_pool_size()` the sync session uses; * a proxied pod (http://, or https:// to a name) stays an ordinary request. `_apost_page_to_pod` and `_apod_still_alive` mirror the sync twins, and both twins now share ONE request builder and ONE response reader -- two copies of that field list is how an engine name, a dpi or a token budget comes to apply on one transport and not the other. Every distinction 0.4.2-0.4.6 established is preserved and tested on the new path: an ANSWER (even 503) is a verdict about the pod and belongs on the refusal path, nothing-came-back is PodUnreachable, a dead pooled socket is retried on a fresh one, and a timeout is never retried -- resending a page the pod is still working on is a second card doing the same work and a double bill. `ocr.pod_http_async` (default on, admin API + settings control) puts the forward back on the pod thread pool. That path is kept working deliberately, not as dead code: this is a new transport on the hottest path in the subsystem, and an operator who hits something odd should change one setting rather than wait for a release. Four existing test files had to be repaired, and the reason is worth keeping: they stubbed only the SYNC twin, so the moment the async default landed they stopped testing their own stubs and began dialling 10.0.0.1 for real -- 300 s hangs where there should have been failures. Their helpers now stub both twins via `_stub_page_post`, and the policy tests say `pod_http_async: False` out loud because they stub `pod_http.post` specifically. A test that quietly measures something else is worse than one that breaks, and this change produced four of them. 21 new transport tests. 3345 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Measured on A4 at 200 DPI (3.4-3.9 Mpix), best of three: pdfium page.render() 21-46 ms PNG encode, PIL default (level 6) 86-353 ms 1846 KB PNG encode, level 1 (lossless) 123 ms 2117 KB JPEG q95 8.4 ms 596 KB the page's own embedded JPEG 0.1 ms 147 KB So most of what is called rasterisation is not rasterisation, and on the surya path it was paid TWICE per page: once in subprocess_engine to hand the page to the isolated-venv worker, once inside surya's own client to build the data URI (surya/inference/backends/openai_client.py:32, with "image/png" hard-coded into the URI it builds). Against roughly 5 s on the GPU per page -- 12 pages/min/pod at Running: 0-3 reqs, measured on two live pods -- that is CPU the pods need in order to hold 48 pages in flight. Two ways out, both shipped, both default: * A page that already IS an image is sent as the PDF holds it. A scan is usually one full-page DCTDecode object, and extract_embedded() takes it in 0.1 ms with no render, no encode and no resample. Every guard in there is a way those bytes would have been a WRONG page rather than a slower one: one image object and nothing else (a stamp or a text layer must not be dropped), DCTDecode alone (Flate is raw pixels, CCITT and JBIG2 are bitonal, JPX is JPEG 2000), grey or RGB only (a CMYK JPEG out of a PDF decodes inverted), /Rotate 0 and an axis-aligned positive matrix (otherwise the bytes are a sideways page, which reads as gibberish rather than as an error), 98 % coverage anchored at the origin, and an effective DPI within [min_dpi, max_dpi_factor x target]. * When a page must be encoded, it is encoded once, at zlib level 1. Lossless, 2.6x faster, and the bytes are carried forward so the request reuses them. The carrier is an attribute on the PIL image, recorded with the size it was true for. Any PIL transform returns a NEW object and so drops it, which is the design: a missed passthrough costs milliseconds, a wrong one shows the model a plausible page at the wrong resolution. chandra and olmOCR therefore scale BEFORE converting, since convert() was quietly invalidating every page. End to end, one page from PDF bytes to data URI: a page that must be rendered 610 ms -> 234 ms (jpeg: 68 ms) a scanned page 610 ms -> 6.9 ms, 1895 KB -> 147 KB New: ocr.page_image_format (auto | png | jpeg), page_image_jpeg_quality, page_image_min_dpi, page_image_max_dpi_factor, with the admin API and the settings form. `png` always re-encodes, which is what a controlled comparison needs; `jpeg` is lossy and stays opt-in, because whether it costs accuracy is a question about a corpus and not about a codec. The rasteriser worker loads page_image by path: it may not import codai.ocr (377 MB via litellm), and the guards must be the SAME ones the parent applies or a pool outage would change what the model is shown. ocr_worker.py may not import even that -- it runs in the engine's own venv -- so its copy of the carrier convention is tested against page_image's to keep the two from drifting. 42 new tests; 3306 passed with the 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pooled keep-alive connection that a pod closes while idle fails on its next use: pod OCR at https://.../v1/ocr is unreachable: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) _post_page_to_pod turned every transport failure into OcrError(503), 503 is in _POD_CANNOT_RUN_PIPELINE, so one dead TCP connection was read as "these pods cannot run the OCR pipeline" and set the whole pool aside for five minutes. It happened twelve times in four hours on a production run: the GPU-less orchestrator rasterised every page of every document for those windows and climbed to 40 GB at ~0.8 GB/min, while sixteen rented A40s sat idle and willing. An ANSWER may condemn an image; a socket may not. So: * PodUnreachable, a distinct error for "nothing came back", which never sets the pool aside. A pod that ANSWERS 404/501/503 still does. * One page is retried on a fresh connection (nothing was sent, so there is no side effect) and then on another pod — but a pod is dropped from the pool only after it fails to answer /healthz. Writing a pod off because an idle connection was reused has cost seven healthy cards before. * A timeout is NOT retried: the pod may be ninety seconds into a 130-second read, and sending the page again doubles the work on that card. * An active cooldown is no longer extended. It used to be, silently: every further refusal pushed the expiry out while only the first one printed, so a five-minute window could last twenty with nothing in the log. Two more ways to end up local, both of which were SILENT: * The gate compared the request's engine against the three VLM engine names. /v1/ocr has always also accepted the MODEL a caller knows in place of the engine driving it — resolve_engine maps the Surya-2 checkpoint to 'surya' and the catalogue advertises those ids as aliases — so a caller naming the model got no pod-side pipeline and no message. The switch has to be read the way a READ reads it, for the same reason it has to be read the way a LAUNCH reads it. * Serve mode not 'model', an empty model id, an engine with no pod pipeline, a standing refusal: each returned "" or None without a word. The log of the run that prompted this holds 8,278 successful pod reads and nothing at all explaining the thousands of documents in between. Every one of them now says which way the decision went, rate limited rather than said once, because a cooldown is minutes and a run is hours. And, since reading a page here means rasterising here, ocr.pod_pipeline: prefer (default) wait up to ocr.pod_pipeline_wait_s for a pod, then read it here — a pool with nothing free is usually one still booting. A timed-out wait is not paid again for five minutes, so the ceiling cannot become a per-page tax. auto read it here at once (0.4.1-0.4.5 behaviour). require never rasterise on this box: 503, saying why. Tests: 32 new (tests/test_ocr_pod_pipeline_policy.py) covering the alias, all four silent routes, the three policies and the transport/answer line. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
New `ocr.max_pages`. A document with more pages than the ceiling is refused with a 413 naming its page count and the limit, instead of being read. Checked off the page tree, before anything is rendered, split or sent -- so an over-long document costs milliseconds rather than an hour of time, memory, tokens and (where pages are read on rented cards) money, only to die in a client's timeout anyway. Length is the one cost knowable before any work, and the one that scales every other cost at once. Shipped OFF (0 = no ceiling). Page count is not a fault: an install that batch-reads book scans is as legitimate as one that reads two-page letters, and this product does not know which it is talking to. Settable from the OCR settings page. Refused LOUDLY and with both numbers. This subsystem already carries the trap that a degraded page answers 200; a skipped document that looked like an empty read would be the same mistake with a bigger blast radius. The message names the setting so an operator can find it, and says the document was NOT read so a caller can route it elsewhere rather than file an empty result. `max_pages` at exactly the ceiling passes -- "more than N" means N is fine. A nonsense value fails OPEN: refusing every document would be worse than the problem. Co-Authored-By:
-
- Oct 15, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Adoption existed, but only ON DEMAND: it runs inside `acquire()`, when a request needs a pod. Nothing reconciled the registry at boot. So after every restart this deployment's surviving pods sat unowned until either a request happened to want that pool or the reaper's 20-minute grace expired — and an hourly unattended upgrade could pay for a fleet twice, once for the pods it abandoned and again for the ones it rented beside them. Observed 2026-10-16. A forced upgrade at 00:01 orphaned three pods. At 00:04 the surya pool rented a NEW pod while a healthy surya pod from before the restart was still running and still in the registry. The three were reaped at 00:21 having billed $1.11 between them: "not tracked by any pool and its owner stopped reporting". THREE FIXES. 1. A POD THAT WAS BOOTING COULD NEVER BE ADOPTED. register_pod writes the url only once the pod is ready — `# url once ready` — and find_shared_pod skipped a url-less entry in SILENCE. These images take 6-14 minutes to boot, so a restart has a good chance of landing inside one, and the pod it lands on is unadoptable for the rest of its paid life. The pod id is enough to ask RunPod where it went, so an entry with no address now recovers one. 2. TWO OF THE THREE REFUSALS SAID NOTHING. No url and a failed health probe both `continue`d without a word, so "rented beside a pod it could have adopted" and "had nothing to adopt" produced identical logs. That is why tonight's case cannot be pinned down: the evidence was never written. Every refusal now names the pod and the reason. 3. NOTHING RECLAIMED AT STARTUP. `reclaim_registered_pods()` walks the registry when the scaler starts and takes this deployment's pods back into their pools — before `reap_orphans`, deliberately: a pod we are already paying for should rejoin its pool, not be terminated and rented again a minute later. It takes back EVERY pod of a pool, not one; the first version of this reclaimed one of sixteen and left fifteen billing, which is the same bug with extra steps. Bounded by how many the registry holds for that key, since adoption removes nothing by itself and the loop must end whatever it answers. A `pool:` key is left to the on-demand path: its shape comes from whichever model loads first, and guessing would be worse. Also: `scale_up_max_parallel_boots` is now editable in the admin model form (and survives the save). It decides how many GPUs start BILLING together when a pool grows — a floor of 16 fills in waves of 4 at the default — which is exactly the kind of thing an operator should not have to edit models.json for. 17 tests. Suite: 3183 passed, the same 13 known-stale failures. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
From last night's run: ~97% of a 1,000-document batch came out truncated, with no error anywhere. The engine answered 200 every time. Its decode had fallen into a loop, run to the token cap emitting the same fragment, and the resumer correctly refused to continue a reply that was already repeating itself -- so the page was flagged, returned, and indexed. `fallback_engines` never fired, because it only fires on an exception. The same pages read cleanly on another engine (38,055 characters against a truncated stub), so the recovery was there the whole time and nothing reached for it. `ocr.retry_degraded` (default "auto") re-reads ONLY the degraded pages on another engine and keeps the replacement only if it is not degraded in turn. Bounded by construction: a clean document costs nothing, a ruined one costs at most a second pass over its ruined pages, and `retry_degraded_max_pages` caps even that. The page keeps its own index and carries `degraded_retry` saying who re-read it and why; the response reports `recovered_pages`. Degraded means looping or truncated -- the two the engine itself attests. NOT a short page, a blank page or an unreadable one: those have their own meaning and re-reading them spends a card to be told the same thing. Which leaves the other half of the report: olmOCR returned 4,046 characters where chandra read 30,983 on the same page, ending cleanly well short of the cap, flagging nothing. The only evidence an early stop leaves is the company it keeps, so `retry_short_pages_below` compares a page with the DOCUMENT'S OWN median rather than a character count -- a threshold in characters would need re-tuning for every language, DPI and layout. Default 0, OFF, and deliberately: this signal is inferred where the others are attested, and a genuine title or signature page looks identical. A short page is also only replaced by a LONGER one. Off on pods: only the orchestrator can reach another engine's pods, so a pod would re-read on the engine that just produced the flag. Co-Authored-By:
-
- Oct 14, 2026
-
-
Stefy Lanza (nextime / spora ) authored
An OCR VLM needs a model server AND the surya pipeline that drives it. Which machine runs the second one is now chosen by the pod IMAGE, and by nothing else: coderai-vllm the pod serves the model; the ORCHESTRATOR rasterises every page and sends a ~1.06 MB base64 PNG per A4 page at 250 DPI -- and again for every resumption round. Unchanged. coderai-vllm-ocr the pod carries surya's client venv and pypdfium2, so it is sent a ONE-PAGE PDF (~474 KB, 1.7 ms to split here against ~105 ms to rasterise) and reads it on the card with its own vLLM over loopback. A separate `pod_ocr_pipeline` flag was built first and removed. Two switches that can disagree is the invisible mismatch that cost three rounds of blaming a pod image for orchestrator bugs; choosing the image IS choosing the mode, it already lives in models.json and on the model page, and an image that cannot run a pipeline can no longer be asked to. PAGE-LEVEL FAN-OUT IS UNCHANGED. One page is one request in both modes -- `placement.choose` picks an upstream per page -- so every pod in the pool shares the work and none owns a document. Sending a whole PDF to one pod would turn an 8-way parallel read into a serial one, which is why the split is per page. THE TUNING TRAVELS, not just the wiring. surya_dpi, the page budget, the continuation count, the repeat-detector thresholds, langs, the page window, the resident cap. Every one was set in response to a measured failure, and a pod reading pages at surya's own defaults would be indistinguishable from the "pre-fix pod image" that was wrongly blamed three times. A test cross-checks every CODERAI_OCR_* the plan sends against what config.py actually reads: a variable sent and never read is a setting that silently does not apply, which is the exact shape of two bugs found earlier today. FALLBACK, and what it refuses to hide: * no pod rented -> the WHOLE document is read locally, not half each way (a document read by two pipelines is the hardest kind of result to explain); * a pod answers 500 -> raised, not quietly re-read here; a broken fleet must stay visible; * the image lacks the pipeline -> the orchestrator does what it always did. The image, built and verified (14306 MB, /healthz OK): venvs/surya 1.6 GB torch 2.14.1+cpu cuda=None + ultralytics 8.4.177 venvs/vllm 11 GB main venv: pypdfium2, and no torch at all (light core intact) A CUDA torch in the surya venv would add ~4.2 GB AND put surya's small local line detector on the card serving the model; the venv check asserts torch.version.cuda is None so a resolve cannot quietly undo it. 35 tests. Two flaws of my own that they caught: four passed alone and failed in the suite because they patched sys.modules while `from codai.models import placement` resolves the name as an attribute of the already-imported package -- now patched on the real module, verified in both orders; and _plan_env used a hand-rolled stub that tested the stub, now built from a real RunpodModelConfig. Suite at the 25 pre-existing failures. NOT YET EXERCISED AGAINST A REAL POD: the orchestrator path has only run against fakes, and the image is local -- not pushed or signed. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.3.66 serialised every pdfium call behind a lock, because pdfium permits one caller at a time and calling it concurrently corrupts its state into "PDFium: Data format error" on valid documents. Correct, and it made rasterisation single-threaded for the whole install -- a ceiling of about one core. pdfium IS safe one-document-per-process, so `ocr.raster_workers` (0 = auto at half the cores capped at 4, 1 = in-process under the lock) hands documents to worker processes. Measured on 12 concurrent 47-page PDFs at 150 DPI: 21.9s -> 10.3s, 2.13x. Not 4x -- the parent is single-threaded receiving the pixels and rebuilding the images, and ~4.5 GB crosses the pipes. The ceiling is the parent. Three things this had to get right, each of which cost a bug first: - A worker is borrowed per CHUNK, not per document. The caller OCRs each window between pulls, which takes far longer than rasterising it, so holding a worker across that would idle it for most of its tenure and cap the install at `raster_workers` concurrent documents -- worse than the lock it replaces. - A pool that cannot start must cost nothing. The first version waited 300s for a worker that could never arrive (missing script), stalling every document for five minutes: the optimisation becoming an outage, which is the exact failure its test was written for. Creation failure is now remembered rather than retried per document, and a busy pool falls back in-process after 30s. - Workers are executed, never imported. `import codai.ocr` costs 377 MB and 1.5s via codai.models.manager -> litellm; paying that per worker, in the subsystem whose other half exists to bound memory, would be self-defeating. The worker imports pypdfium2 and PIL only, ~25 MB. This is why it is not a ProcessPoolExecutor: spawn pickles a function by qualified name and re-imports the package to find it. Also fixes a cost 0.3.65 introduced and nothing but a stopwatch would surface: pricing a document called `pdf[i].get_size()` per page, which LOADS each page object -- 593 ms for a 47-page document, paid UNDER THE LOCK, so every document in the install queued behind every other being priced. `pdf.get_page_size(i)` reads the same numbers from the page tree by index. page_estimate: ~600 ms -> 9.6 ms. Every failure path degrades to a correct in-process render: no pool, a dead worker mid-document, a missing script, an unusable module, an abandoned document. Correctness never depends on the optimisation working. 87 tests across the budget and pool suites. Full suite at the 25 pre-existing failures. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`ocr.max_concurrency` bounds how many pages are BEING READ. Nothing bounded how many were RESIDENT, and those differed by a factor of the document length: ocr_document rasterised every page up front and held the list until the response was written, so pages queued behind the semaphore were in memory too. The real figure was (documents in flight x their page count), which no setting expressed. A production install was OOM-killed twice inside forty minutes -- 90.7 GB with max_concurrency at 1024, then 92.7 GB with it at 128. Eight times smaller, half a gigabyte worse, which is the proof the semaphore was never holding the memory. Two bounds, on the two things that grow: - `ocr.page_window` (0 = follow max_concurrency, capped at 32) reads a document a window of pages at a time -- rendered, read, measured, detected on, dropped, then the next window. Residency is O(window) not O(document). Measured on a 40-page A4 PDF at 250 DPI: 40 pages held = +974 MB RSS, 32 = +768 MB, 8 = +215 MB. - `ocr.max_resident_mb` (0 = derive it) is a ceiling on the bytes in-flight documents may hold as pages, charged before a page exists: page_estimate() prices a PDF from its page tree without rendering it. Derived from the smaller of the cgroup limit and host RAM -- the container limit first, since a 2 GB container sizing itself from a 128 GB host has no ceiling at all. The estimate carries a measured overhead factor (PAGE_OVERHEAD = 1.6): the RGB pixel arithmetic under-predicts real residency by 40-55%, and a budget that under-charges admits half again as much as it meant to. A document larger than the whole budget still runs, clamped and alone -- a gate that can refuse forever is a deadlock with a polite error message. Reservations are returned on cancellation too, or a disconnected client shrinks the budget permanently. Also: rendering now runs off the event loop (a long PDF used to stall every other request in the process), the engine is sticky across windows so a document is not read by a different engine in each one, and pages already read keep their results when a later window falls back. The comment in config.py that reasoned "64 in flight is ~1.2 GB" is corrected in place -- it assumed resident == admitted. 59 tests. Not bounded, and not measured: /v1/ocr/batch still buffers every uploaded file; source bytes are ~1% of rasterised pages, so that was not the term behind the kills. Co-Authored-By:
-
- Oct 13, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Raised from the Digesta side: the one high-leverage way to make surya fall through less often is to make its repeat-detector and retry tunable. Half right, and the other half turned out to be the bigger lever. Checking the installed surya rather than the description: `_should_retry` stops at `max_retries` and RETURNS the looping text — surya does not return empty. The empty page comes later, because looping markup parses to no blocks. So the symptom is real and the mechanism is not what it looked like. **The lever is that surya never said it looped.** chandra has reported `repeat_detected` since it shipped, with the text returned anyway because a looping transcription usually has a correct beginning. surya reported nothing, so a short or empty page was indistinguishable from an unreadable scan — and those need opposite handling: one is a cascade candidate, the other means the document is bad. surya now flags it at the moment the verdict is made (the retry hook's last call for the page), with `repeat_chars`, the length of the reply that was still looping: "looped and produced four thousand characters" has a usable beginning, "looped and produced nothing" does not. **And the detector is now retunable** — `ocr.surya_repeat_max_repeats`, `surya_repeat_window`, `surya_repeat_scaling`, 0 on each leaving surya's own (4, 500, 3.0). Hooked like `_should_retry` already was, since the vLLM backend passes neither. A false positive is expensive: the page is re-run several times at rising temperature and then parsed from a reply nobody wanted, usually yielding less text than the first one had. Operator settings, deliberately not per-request — they describe the corpus (tables, dotted leaders and form fields repeat short strings legitimately), not the page. An explicit argument from surya's own code still wins, so its `cut_from_end` call keeps working. 11 tests in tests/test_surya_loop_reported.py, including that a loop surya then rescues is not recorded as a verdict, and that the thresholds must be in place BEFORE the hook is installed (it patches once per process). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Resumption was surya's alone. chandra and olmOCR stopped at their configured cap and returned whatever they had — and what they had is the worse half of the problem: chandra emits HTML, so a cut-off reply is unclosed markup that parses to a fragment of the page or to nothing, and olmOCR's reply opens with YAML front matter, so one cut off before the body closes parses to metadata and NO TEXT AT ALL. In the finished page both are indistinguishable from a model that could not read it, which is the confusion this whole area exists to remove. The rules now live once, in codai/ocr/resume.py, and they are the ones surya learned the hard way rather than a fresh guess: * both vLLM flags or neither — `continue_final_message` AND `add_generation_prompt: false`. With the second left at its default the chat template appends a fresh assistant turn and the model answers again about the same image, which reads exactly like a model repeating itself and did for a whole day; * never resume a reply that is already looping: it did not run out of room, it ran out of page, and resuming can only extend the loop; * the context window is the ceiling, not the budget. Resumption re-sends everything so far as INPUT, so the room shrinks by what was just produced, and a segment that no longer fits means the cap is too BIG, not too small; * progress has to be real: a segment identical to the previous one, or one that comes back looping, ends the loop; * `finish_reason` is authoritative both ways — "stop" with a full budget is a complete page and must not be resumed, which a token count alone cannot tell. Both engines' ask paths now return a `resume.Reply` carrying the finish reason and the usage counts (they were discarding both), take a `prefix` that goes back as the prefilled assistant turn, and report `model_attempts` in the same shape surya produces — including the budget in force, so a caller can tell a request that landed from one that was ignored. Being in-process, they have no thread boundary to cross: `page_options()` reaches them directly. New settings, in config.json, the admin API and the settings page: chandra_max_continuations / chandra_context_tokens and olmocr_max_continuations / olmocr_context_tokens. The UI parses them as numbers rather than falling back on `||`, because 0 is a real setting here (resumption off) and not "use the default". chandra's `detect_repeat_token` moves to being shared rather than copied: it is a port of chandra's upstream util and answers "has this reply fallen into a repetition loop", which is not chandra-specific. 42 tests: tests/test_ocr_resume_shared.py (28, the rules) and tests/test_ocr_resume_engines.py (14, each engine's own ask path). The existing chandra/olmocr/confidence/pod-TLS harnesses were updated to the new ask contract — it changed deliberately, and the stubs say why. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod ties a network volume to a single data centre, and a pod elsewhere cannot attach it. So `network_volume_id` made `_data_centers()` answer with the volume's region and throw the `data_centers` fallback list away — which is correct for one volume and still forced a choice nobody should make: rent wherever a card is free and re-download the weights onto every cold pod, or keep the storage and wait for capacity in one region. On a contended card that is a lease that fills versus one that waits. `network_volumes` is a map {REGION: volume_id} on the account block and on a model's. The region is chosen first — the scaler already paired every candidate card with every allowed region — and the volume now follows the region that candidate was ranked for. A region with no volume in the map is dropped from the list with a line saying which, because a pod rented there would silently have no storage and boot minutes slower; if nothing is left, provisioning refuses and names both sets rather than placing the data somewhere the operator never listed. A model's map replaces the account's whole rather than merging: listing regions is a statement about where that model's data may live. `network_volume_id` keeps working unchanged and a one-entry map behaves identically to it, which is how this ships without touching any install. The global volume was supposed to make all of this unnecessary and cannot: the type is not creatable through the public API (POST /v2/network-volumes takes only STANDARD|HIGH_PERFORMANCE and demands a dataCenter), and object storage — no locking, no atomic rename, eventual consistency — is wrong for anything but read-mostly weights. `global_volume_id` stays supported; it stops being the plan. Seeding a new region needs no new pod shape: POST /admin/api/runpod/seed-volumes boots one ordinary pod per region with that region's volume attached, which IS the download because the caches already point there, then terminates it. Sequential, admin-only, under the usual spend caps. The venv needs nothing — venv_on_volume already builds it under a lock with .ready written last, so the first pod in a region builds it and every pod after finds it, which is what makes the dependency-free slim image usable in every region. Also here: _data_centers() upper-cases region ids, since the map's key matching the candidate's region is the whole mechanism; pools_schedule_status carries the map; both admin surfaces gained a "Per-region volumes" field. Everything is offline-tested only — 25 new tests in tests/test_runpod_regional_volumes.py. The first multi-region install is the real proof, and the first thing to check there is that RunPod's dataCenterId and the map's key are spelled the same. test_runpod_placement's region test counted three copies of `sel.get("_dc") or self._data_center()`; the region is now resolved once into a local because the volume lookup needs it too, so it checks that one resolution and its uses instead of duplicates of it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two defects that a throughput collapse on a live install made visible. With surya_max_tokens raised to 12288, 262 resumption segments spent 1.6M generated tokens in 80 minutes, 122 of them running the full cap dry, while 253 of those 262 ended in "came back repeating itself". Resumption re-sends the text so far as INPUT, so the model's context window -- not the continuation budget -- is the real ceiling. The budget did not know that: at a 12288-token cap, the third segment of a page asked for 12288 more on top of ~28,500 already in a 32,768-token window and was rejected outright. The page was then reported as one that could not be resumed AT ALL, discarding text already paid for. - surya_context_tokens, or 0 to read the server's own max_model_len from /v1/models. Configuration wins: an operator may serve a shorter window than the checkpoint advertises. - Each segment asks for the smaller of the cap and the room left, measured from what the server said the prompt cost (usage.prompt_tokens), not guessed; the guess covers the first resumption only. - Out of room stops, KEEPS what it has, and says so -- naming the fix that helps, which is a SMALLER max_tokens, not a bigger budget. - A refusal for length is not retried: the fallback resumption adds a user turn, so it is longer than the request just refused. - Not knowing the window changes nothing and guesses nothing. The second defect: a reply that trips surya's repeat detector hit the cap because it is LOOPING, not because the page is long. Resuming it can only extend the loop, and surya re-runs the whole page anyway -- so one page on that path cost four full generations. It is no longer resumed, and it is not marked truncated either: that is surya's retry to make, and "truncated" would send the caller looking for a bigger budget it cannot use. A fixed token cap is also an accidental repetition circuit-breaker, which is why raising it slowed the LOOPING pages by the ratio of the caps rather than every page. Documented in docs/ocr.md with the window arithmetic (2048 -> ~13 segments, 12288 -> 2). Tests: 8 new (window clamp, stop-one-segment-early, length refusal not retried, unknown window, configuration over server, a cap below the 256-token floor is never mistaken for a full window, a looping reply is not resumed, a genuinely long page still is). 62 pass in the surya/blank-page suites; the floor-versus-cap bug was caught by the existing 44. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two things, from one sample document. RESUMING A CUT-OFF PAGE. Raising surya_max_tokens only moves the cliff: a dense enough page overflows any fixed number, and incomplete block markup parses to a fragment or to nothing. `ocr.surya_max_continuations` (default 2) resumes instead -- the text so far goes back as a prefilled assistant turn with vLLM's `continue_final_message`, so the model carries on the exact string rather than answering afresh about the same image. A server rejecting the flag (llama.cpp) gets a plain "continue" instruction; a server rejecting both leaves the page partial and honestly marked, since a page that cannot be finished is still a page. Logprobs are concatenated in step with the text, because surya_repair_accents rebuilds characters from the model's own per-token bytes and a later segment paired with earlier bytes would corrupt exactly the accented characters the repair exists to fix; if they cannot be aligned the repair is given up on rather than fed a mismatch. Temperature 0 -- the opposite of the retry path, which raises it to break a repetition. A page still unfinished after its budget carries "complete": false so a caller can refuse a partial rather than store it as read. On by default: returning a page KNOWN to be incomplete is a correctness failure, not a cost trade-off. IS THERE ANYTHING ON THE PAGE AT ALL. A four-page court PDF came back with 47 characters and was reported as an OCR failure. It was not. Every page was a ~900-byte CCITT Group 4 image -- what an EMPTY page costs, a couple of bytes per all-white row -- carrying 0.13% ink, all of it a diagonal scan edge and a diagonal "copia non ufficiale" watermark. The 47 characters WERE the watermark. The engine had read the page perfectly. "The model could not read it", "the reply was cut off" and "the page is blank" were indistinguishable in a short result, and they lead to completely different places -- a better engine, a bigger budget, or a document fixed upstream. Any page whose text is shorter than ocr.blank_page_max_chars (200) now carries the fraction of its pixels that are not background, from an image already in memory, with blank_pages listed at the top of the response. The opposite verdict is as useful as the blank one: ink above the floor says there IS something there, so the engine's failure is real and worth chasing. It applies to every engine, because the measurement is of the image rather than of the model, and it is a diagnostic only -- nothing is skipped, refused or altered because of it. Verified against the sample: all four pages 0.13-0.33% ink, correctly called blank against the 0.5% floor. Two bugs found while testing: `rounds` counted resumption ATTEMPTS rather than successes, so a failed resume would have reported the page as whole; and three assertions matched a source window that moved, one of them for a sentence that is wrapped across f-string lines and never appears contiguously. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three defects, found from a live batch that reported ~98 pages of about 71 characters, 100 with NO text at all, and 13 reading perfectly. 1. `model_attempts` could not reach anyone. `_page_from_dict` copied the worker's diagnostics onto page.meta BY NAME and knew only `unreadable_trace` and `accents_repaired` — so the field added in v0.3.45 was dropped at the subprocess pipe, and was being hunted for in a live batch that could not have contained it. The same reassembly had already dropped `accents_repaired` once, reporting null for a repair that worked. It now forwards by exclusion: every key that is not an OcrPage field becomes meta. A key-by-key copy means each new diagnostic is born broken and nobody finds out until they need it. 2. An empty page and a 71-character page are the same bug. surya-ocr-2 emits block markup — a category and a bbox per block, BEFORE the text — so a reply stopped at its token cap is incomplete markup, and incomplete markup parses to a few blocks or to none. The cap is surya's `max_tokens_default=2048`, which its vLLM backend never overrides, and the same batch averaged 2,644 generated tokens per request: above it. `ocr.surya_max_tokens` raises it (0 = surya's own), opt-in because a higher cap costs tokens on every page. What is not optional is telling the cases apart, so a page that hit the cap now carries tokens, cap and `truncated: true` with an explanation that it is a cut-off reply and NOT a page the model could not read. Counts are per THREAD, like the trace, so one lane never reports another lane's verdict. 3. The fleet could grow by at most one pod per BOOT. `_should_grow` opened on a boolean held for the whole provision, and a provision is ~12 minutes — so a burst needing four pods took forty minutes to be served by four, whatever max_pods said (measured: 64 pages at 16 per pod got two pods, ready ten minutes apart; reported as "no real scale-out"). The grower also could not see the SIZE of the demand: `min(inflight) >= threshold` answers "is the fleet full?", so the only request it could formulate was "one more", and the callers blocked in acquire() — the actual measure of unmet demand — were invisible. Both are counts now: `_booting` (booting pods ARE capacity, so one demand is never provisioned for twice) and `_waiting`. Growth runs while pods + booting < demand, bounded by max_pods and by the new `scale_up_max_parallel_boots` (default 4) — a real limit, because the pods the demand justifies are the GPUs that start billing together. The original single-flight anticipatory trigger is KEPT beside it: max_inflight_per_pod defaults to unlimited, so requests are often never blocked and a merely saturated fleet would otherwise never grow at all. A prediction gets one pod; measured demand may get several. Found while testing (3): acquire() ignored its own timeout when a sibling process was booting a pod for the same pool — it waited a flat 15s and looped back into the same decision without re-checking the deadline, for as long as the sibling's registry entry survived (up to 15 minutes). A test hung on it. 42 tests across tests/test_runpod_scale_out.py (new) and tests/test_surya_retry_budget.py. Two of my own test harnesses were wrong and are fixed, not worked around: a sys.modules ledger stub that `from codai.api import runpod_ledger` ignores once anything has imported the real module, and a static count of increments against decrements, which cannot express "one increment, four mutually exclusive exits" — replaced by runtime balance tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Read from the pod's own drained log, ten minutes of batch OCR on 2026-10-13: Running: 15-19 reqs, Waiting: 0 reqs, GPU KV cache usage: 5-7% Avg generation throughput: ~1,900-2,100 tokens/s, every minute 524 x "POST /v1/chat/completions" 200 OK, no aborts 2,644 generated tokens PER REQUEST (second pod: 4,010) MM cache hit rate: 25-27% Three things in those numbers, and only the third is a defect. The GPU was never the bottleneck: 5-7% of the KV cache and `Waiting: 0` for ten solid minutes. So a card to itself cannot be the fix, and the co-loaded pyannote cost nothing measurable — the limit was never VRAM or batch slots. Oversubscription converts throughput into latency, not into stalling. 64 pages into max_pods x scale_up_inflight_per_pod = 32 slots queues correctly, but every page's latency rises by the queue, and a client counting finished DOCUMENTS sees nothing until a document's LAST page lands: spread 64 pages over sixteen documents and all sixteen advance while none completes. The pool now says so at build time instead of leaving it to be inferred. The defect: 2,644 generated tokens for a page whose text is ~400. surya re-runs a page whose generation repeated itself (`_should_retry`) up to three more times, each at a rising temperature and each able to reach its token cap — four full generations to arrive at the same unreadable answer — and its repeat detector cannot tell "the model broke" from "this page really does repeat", which a tax table does. The 25-27% multimodal cache hit rate is the direct evidence: a quarter of all requests re-sent an image the pod had already read. None of it was visible: a page generated four times over and a page that was merely slow look identical from outside. `max_retries` is unreachable from surya's settings — the vLLM backend calls chat_completions_batch without it, so the default 3 was the only value there has ever been. So the decision is hooked, beside the token-trace hook already in that module, and `ocr.surya_max_retries` is a CEILING (asking for 10 where surya allows 3 gets 3): 1 by default, 0 to disable, -1 to hand it back. Every retry is logged, and a page that needed more than one call carries `model_attempts` in its result — always, not only under surya_token_trace. A page read first time stays byte-identical to what it always returned. The counter is per THREAD, like the trace, or one lane would be blamed for another lane's retries. Also pins SURYA_INFERENCE_PARALLEL=1 in the served modes: a lane is our unit of concurrency, and left unset surya sizes its own thread pool from a built-in GPU table and fans out inside the worker, so instances x lanes would stop being the real number of pages in flight. 19 tests in tests/test_surya_retry_budget.py; settings carry the new knob. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The worker protocol was one request, one reply, strictly in order, so pages in flight WAS the process count — and a process costs ~600 MB of imported torch. That is the whole reason `surya_instances: 64` meant 64 processes, 37 GB and a nine-minute stall with the rented GPU idle at 2% KV-cache usage. In the served modes that memory buys nothing: the recognition VLM is on the pod, the local predictor holds no weights (1.7 MB of tokenizer and config on disk — surya-2 returns its own boxes, so the detection model is never even fetched), and the process rasterises a page and then waits on a socket. So a lane is a worker OBJECT inside the process that already exists, with its own predictor and its own page, sharing the one torch import. Effective concurrency is instances x lanes; `surya_lanes` defaults to 4, served modes only, built on demand. Three things make one process safe to drive concurrently: - every reply carries the id of the request it answers. Out-of-order replies are the point — a short page must not queue behind a dense one — and attributing one page's text to another request would be worse than losing it, so an unknown id is dropped rather than guessed at; - the token trace and the accent repairs are per THREAD. surya's hook is module-level, one _generate_one per process, and its sink was a list belonging to whichever worker object installed it last. Shared between lanes it would have filed one page's logprobs, and surya_repair_accents' rewrites, under a different page; - a worker that dies fails its pages instead of leaving them parked on a reply that is never coming, and killing the engine wakes everything waiting. A single-lane engine sends no ids and reads replies in order exactly as before, which is how paddle, docTR and surya-`local` are still driven. `lanes()` is asked for politely (_engine_lanes), because an engine is anything with recognize_image and a pool must not refuse to build over a capability an engine simply does not have. Tested against the REAL worker protocol with a stubbed engine: four 0.25s pages finish in under 0.75s, on four distinct lane objects, and six pages over two lanes never exceed two. Co-Authored-By:
-
- Oct 12, 2026
-
-
Stefy Lanza (nextime / spora ) authored
MEASURED VERDICT. The byte trace on a page of crisp synthetic Italian, served through the Aruba install on vLLM 0.27.1: unreadable: count=29 blame='the serving path' page 0: calls=1 with_logprobs=1 conclusive=True kinds=None why: the byte stream decoded cleanly, so the characters were lost AFTER the model: the fault is in the serving path, not the model surya-ocr-2's own byte stream was valid UTF-8 and every line came back at conf=0.9605. The model read the page correctly; something after it destroyed 29 accented characters. What that does NOT implicate: surya-ocr-2 and chandra-ocr-2 are the SAME architecture (Qwen3_5ForConditionalGeneration) on the same vLLM, and chandra's accents survive. So this is not vLLM mishandling qwen3_5 -- it is vLLM mishandling SURYA'S CUSTOM 65,425-entry BYTE-LEVEL vocabulary, in which one accented letter is two tokens and either byte alone decodes to a marker. It was also a single model call, which kills the concurrent-batching hypothesis I had been carrying. OPTION 1, shipped and on by default: ocr.surya_repair_accents rebuilds the reply from the per-token bytes in the same response. Applied in the backend wrapper BEFORE surya parses the HTML, so blocks, boxes, text and confidences are all built from correct characters; repairing further down would mean fixing the same text in several places and missing one. Rewriting a model's output is serious, so it refuses unless all three hold: strict UTF-8 decode, no marker in the result, and an ASCII skeleton identical to the original's. The third is what makes it a repair and not a substitution -- it can only ever change non-ASCII characters, so a truncation, a different block or a shifted offset all leave the original alone, markers and all. Clean text is untouched, not even reassembled, and a repaired page reports `accents_repaired: {blocks, recovered, how}` because the text is no longer byte-for-byte what the server sent. Default ON because the alternative is shipping text measured to be corrupt. Set false to see exactly what the server sent. OPTION 2, prepared: a `coderai-vllm-ocr` pod image pinned to vLLM 0.20.1 -- the version surya-ocr itself spawns (surya/settings.py:96) -- so the model can be served on the stack its authors validate, the repair can be checked against a reference rather than trusted, and the upstream bug report has a known-good comparison. Selected per model with `runpod.engine: "coderai-vllm-ocr"`, built with `build_capability_image.sh vllm-ocr`. The pin set was verified to exist before the profile was written, because a profile that cannot build is worse than none: wheels.vllm.ai/0.20.1/cu129 is live, and torch 2.11.0 / torchvision 0.26.0 / torchaudio 2.11.0 (vLLM 0.20.1's own metadata pins) all have +cu129 cp313 wheels. Qwen3_5ForConditionalGeneration is in 0.20.1's registry, so it can serve the model. No torchcodec pin: 0.20.1 does not require one, and pinning a package nothing needs is its own hazard -- the file says what to add if a build ever surprises us. NOT a drop-in for `coderai-vllm`: everything else on it would be pinned back seven minor releases. tests: test_ocr_accent_repair.py, 29 tests -- the repair itself, every guard, and the image profile. 833 passed across the ocr/runpod/config/admin/front selection with the same 8 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`ocr.surya_token_trace` (off by default) reports the per-token bytes at every place a surya page lost a character. It exists to answer one question that the finished text cannot: whose fault is it? invalid-utf8 the reassembled stream does not decode -- a lone UTF-8 lead byte with nothing to complete it. The MODEL emitted a broken sequence; the text faithfully renders a wrong answer. model-emitted the stream decodes and a token IS the replacement character. The model chose it. no findings the bytes were sound and the markers appeared anyway, so the loss happened AFTER them -- OURS, in the serving stack. That last case is why an empty trace is a result, not a failure. CORRECTION, and the reason this was needed. v0.3.21 claimed U+FFFD could not be a byte-level artefact, on the grounds that `a`-grave was a single token so one marker had to mean the model chose it. That was wrong. Measured: 'a-grave' utf8=[195, 160] ids=[2131, 2258] pieces=['A-tilde', 'l-bar'] decode([2131]) = U+FFFD <- a lone lead byte, exactly ONE marker decode([2131, 2104]) = 'e-grave' surya-ocr-2's vocabulary is BYTE-level, so one marker per accented letter is precisely the signature of a byte-level failure -- the thing it was claimed to rule out. The vocab lookup was misread too: `vocab['a-grave'] = 2160` does not mean the letter is token 2160; in a byte-mapped vocabulary that key is the glyph standing for byte 0xA0, which alone also decodes to one marker. The "one marker vs two" discriminator does not exist. Digesta's charset instinct was closer to the mark than the rebuttal. base.UNREADABLE and docs/ocr.md now carry the correction rather than the refuted argument. What IS established, measured on the Aruba install at the same DPI through the same transport on perfect synthetic Italian (crisp vector glyphs, 34pt, 250 DPI): surya 29 U+FFFD markers, zero accents surviving chandra 0 markers, all 12 accents intact So the document is exonerated. Also notable: the em dash survived surya intact while every two-byte accent died, and surya emitted U+1EEA (Vietnamese) in place of one -- no decoding bug can fabricate a character that was never in the input. Implementation notes: - surya DROPS this data. Its recognition predictor does pass it on (`BlockOCRResult(..., raw_logprobs=out.logprobs)`) but the model never declares the field and pydantic defaults to extra="ignore", so it is constructed and silently discarded -- their own comment calls it "phase 2 use". Verified by construction: the attribute does not exist on the result. So the value is taken one layer earlier, wrapping `_generate_one`, which still has it. The wrapper only reads and swallows every error: a diagnostic that could break a page would be worse than no diagnostic. - The scan reassembles the WHOLE byte stream and walks it as UTF-8 rather than judging tokens pairwise. Per token is meaningless here -- one character legitimately arrives as two or three tokens, each "invalid" alone -- and the first version of this did judge them pairwise and flagged every accent and every em dash, the exact false positive that would make it worthless. - Capped at 40 findings per page, two tokens of context each side, because a lead byte is only wrong in the light of what followed it. - Clean pages are byte-identical to before: the key appears only when the text actually contains a marker, and it is reported ONCE document-wide under `unreadable.tokens` rather than repeated in every page's meta. - The flag is in _pool_params, because the hook patches surya's backend at LOAD -- without that, toggling it would appear to save and take effect never. A third discriminator needing no code: serve the same weights through llama.cpp (`datalab-to/surya-ocr-2-gguf`, `surya_serve: "llamacpp"`). A different detokenizer on identical weights would convict vLLM outright. tests/test_ocr_token_trace.py: 30 tests. 818 passed across the OCR/config/admin/ runpod selection with the same 10 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A fifth OCR engine. Chandra (datalab-to/chandra-ocr-2) is from the same lab as Surya and is driven the way olmOCR is — one page image over an OpenAI chat endpoint, three serving modes (model | vllm | server) — but its `ocr_layout` prompt asks the model to answer as one <div> per layout block carrying data-bbox (normalised 0-1000) and data-label. So it is the only engine here that returns text, boxes, labelled regions, tables AND page metadata in one pass, with no separate detector. Why it was worth adding now: it is a DIFFERENT MODEL FAMILY from surya-2 (Qwen3-VL's tokenizer, against surya-2's own 65k WordLevel vocabulary) reached over the same transport, at the same pixel ceiling, from the same rasterised page. That makes it a cheap controlled experiment on the U+FFFD accent loss: accents intact under chandra and lost under surya points at the model; both losing them points back at the document's pixels, and then pdfimages -list is the next thing to read. Implementation notes: - NO new pip dependency. The two prompts are mirrored verbatim from chandra/prompts.py (down to the Python repr of the allowed-tag list, which is trained-on wording), the input scaling is chandra's own scale_to_fit reimplemented, and the reply is split with stdlib html.parser rather than pulling bs4 and markdownify into the main venv for a tag split. - Chandra's pixel cap is 3072x2048 = 6,291,456 — the SAME numbers as surya-2 — and it snaps to a 28-pixel grid for the Qwen vision tower. So chandra_dpi defaults to 250 for exactly the reason surya_dpi does. Boxes are de-normalised against the image the MODEL saw, and the size the caller rasterised is reported as meta.source_size when they differ: de-normalising against the pre-scaling size would put every box on a scaled page in the wrong place. - A repetition loop answers 200. A VLM that loses its place repeats the last thing it saw until the token budget runs out, and the result looks like a successful page. chandra_max_retries (2) re-asks with a raised temperature — a greedy decode falls into the same loop twice — and a page that still loops is returned flagged with meta.repeat_detected rather than handed on silently. - The parse never loses page. No blocks at all, content outside any block, an unclosed final div (routine at the token limit), a bbox that will not parse: each keeps the text. A bbox that cannot be read yields NO region rather than a full-page one, because a wrong box looks like a real answer. - Table cells get a space, not a newline, so a table does not flatten to "VoceImporto". License: narrower than Surya's GPL. The code is Apache-2.0 but the WEIGHTS are a modified OpenRAIL-M — research, personal use and companies under $2M funding/revenue, no use competing with Datalab's API. So it is gated on chandra_accept_license, and the pod path differs from surya's on purpose: asking a pod for surya by name IS the GPL acceptance, while chandra's is only ever inherited from a decision already recorded in this install's config. A pod cannot accept a licence on an operator's behalf. lang= is accepted and does not steer it (language_aware: false), like surya-2 and olmOCR: 90+ languages, no language input. So it keeps ONE pool whatever is asked — a language-keyed pool is a second set of loaded instances holding VRAM. Wired through: engine registry, licence gate, per-engine DPI, the vLLM stop path (its VRAM is the subprocess's, not a pool's), the capability catalogue (which now reports boxes/regions/tables from chandra_layout, because with layout off there genuinely are none), model-name aliasing, the automatic fallback chain, pod planning + env seeding, the frontproxy's local-OCR check, models.json capability advertisement, the pipeline step's engine list, admin save/load and the Settings OCR card. tests/test_ocr_chandra.py: 61 tests. 788 passed across the OCR/config/admin/ runpod selection with the same 10 pre-existing failures (broker_config x5, studio_composed_surfaces x3, whisper_server x2). test_ocr_fanout's catalogue test asserted a frozen engine set; it now asserts against the registry, which is what "lists every engine" was always supposed to mean. Also fixed while in there: the Settings page used `parseInt(v) || default` for numbers whose zero is meaningful, so a deliberate 0 saved as the default. New _num() helper only falls back for an empty or unparseable field. Co-Authored-By:
-
