- 12 Oct, 2026 17 commits
-
-
Stefy Lanza (nextime / spora ) authoredA QLoRA trained for a client sat on disk for ten days while a second service rented its own A40 every morning, pushed the adapter over SSH and proxied to it. The path that should have carried it was written but had never been run end to end, and five things were in the way. * `ensure_text_loras` DROPPED a PEFT directory — "publish it to HuggingFace" — which is the shape every trainer writes. It now travels as one deterministic tar (sorted names, fixed uid/gid/mtime, so the content id is stable) and is unpacked by the blob store, because vLLM loads an adapter from a directory: adapter_config.json carries the rank and target modules, and a bare .safetensors is not loadable. * the adapter was re-read and re-hashed ON EVERY REQUEST. A Target is rebuilt per request, so `lora_entry` was set again each time; the bytes were never re-sent (the store is content-addressed) but 159 MB was read and hashed, and the pod asked, only to learn it already had it. Cached per (pod, path, size+mtime), so a retrained adapter at the same path is still sent again. * `_lora_args` passed a CONTENT ID straight through, producing `--lora-modules name=sha256:…`, which vLLM cannot open. On a pod that is the only form the adapter has. * an engine launched for a base model carried none of its adapters. vLLM registers them at launch and one vLLM serves one base, so the pod a shared pool rented for the base alone could never serve an adapter of it — the whole point of sharing the pool. Every model naming the same base now contributes its adapter under its own model id, which is the name requests ask for, so two versions can be A/B'd on one GPU with no reload. * `--max-loras` was never set. At its default of 1, requests naming different adapters cannot share a batch and serialise instead: the cost lands on latency, not on an error. Extracting a bundle is a write from network bytes, so member names are checked for escapes before anything is unpacked, and tar's data filter is used as well. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
-
Stefy Lanza (nextime / spora ) authored
Triton and torch.inductor JIT-compile vLLM's kernels at run time and need a full build toolchain for the stubs they emit. Each piece was found on a live olmOCR pod, one layer at a time, because from outside the failure is identical every time: vLLM dies before registering and the pod answers 502. gcc the C compiler g++ the launcher Triton builds is C++, so gcc alone is not enough ninja-build inductor shells out to ninja for its extensions Only olmOCR hits this, and the reason is worth recording: it is FP8, so vLLM's quantisation kernels go down the Triton path. surya-2 is dense bf16 and served fine on images that had none of these, which is why "the image lacks a compiler" looked for a while like it could not be the problem. Applied to vllm.apt AND vllm-ocr.apt, which is copied from it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
It shipped at 4, which made it the binding limit for every install. The limiter is meant to be each engine's own `*_instances` -- the pool hands out one instance per page and an engine is not reentrant -- while a RunPod fleet can hold max_pods x scale_up_inflight_per_pod pages in flight: 64 x 16 = 1024 on the Digesta install. A ceiling of 4 capped that at a quarter of ONE pod. Measured on one A40, same page, same install: N surya req/s p50 chandra pg/s p50 1 0.93 1.1s 0.21 4.6s 8 3.79 2.0s 1.31 6.1s 16 7.11 2.2s 1.88 8.4s <- both peak 24 5.89 4.1s 2.09 10.6s 32 5.44 +13 err 3.5s 1.44 +21 err 7.5s 48 ALL FAILED Both peak at 16 and both fail outright by 32 -- a cliff, not a degradation. Default raised 4 -> 64: four pods' worth at the measured knee, far above what a single-box install can feed, so it stops being the limiter while staying finite. NOT higher, because every admitted page holds a rasterised bitmap (~18 MB for A4 at 250 DPI): 64 in flight is ~1.2 GB, 1024 would be ~18 GB of page images before any engine does work. Trading a throughput limit for an OOM is not an upgrade. docs/ocr.md carries the table and spells out the three settings that bound this, which are easy to confuse: the per-model scale-out threshold, the per-engine instance count (the real limiter, and for the SUBPROCESS engines it is what the box can host -- a surya worker is a separate interpreter at ~750 MB), and this ceiling. Also recorded there: surya's req/s is BLOCKS, not pages, since it calls once per layout block; and its worker protocol is synchronous, so a worker handles one page at a time and concurrency is the process count. In served mode that worker only scales an image and makes an HTTP call, so making it concurrent -- or skipping the subprocess entirely, as chandra does -- is what would let surya use a large fleet. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
SURYA READS ITALIAN AGAIN. Measured on the live install, the same synthetic page that scored 29 markers all morning: MARKERS : 0 (29 baseline) ACCENTS : AEEIOUaeeiou (all twelve) La societa` ha facolta` di piu` attivita`: percio` e` cosi` che l'unita` perche' gia` oltre. ...but it reported `accents_repaired: null` while visibly having fixed the page, because _page_from_dict carried `unreadable_trace` across the worker boundary and never carried `accents_repaired`. The fix worked; only its receipt was dropped. That matters more than a missing field: the text is no longer byte-for-byte what the server sent, and a caller comparing the two has no way to know why. A silent fix is nearly as bad as a silent corruption. tests: +2 (47 in test_ocr_accent_repair.py) for the receipt crossing the boundary and for staying absent when nothing was repaired. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
ROOT CAUSE, measured in surya-ocr-2's own tokenizer: id 2160 decode -> U+FFFD vocabulary piece -> 'a-grave' id 2168 decode -> U+FFFD vocabulary piece -> 'e-grave' id 2169 decode -> U+FFFD vocabulary piece -> 'e-acute' id 2172 decode -> U+FFFD vocabulary piece -> 'i-grave' id 2178 decode -> U+FFFD vocabulary piece -> 'o-grave' id 2185 decode -> U+FFFD vocabulary piece -> 'u-grave' The model emits the RIGHT token -- 2160 really is a-grave -- and the decode table cannot render it. Every decoder in the chain inherits that: surya's own, and vLLM's, which reports `decoded_token: None` and therefore `bytes: []` and writes U+FFFD into the text. So the accents were never lost in transport. Eliminated by measurement, each in turn: DPI (a crisp synthetic page lost them identically), the document (chandra reads the same pixels perfectly), the vLLM version (0.20.1 cannot load the model at all), --trust-remote-code (applied, `trust_remote_code=True` in the engine config, no change), and the byte channel (vLLM reports `bytes: []` precisely because the decoder failed). THE FIX: when the decoder answers with the replacement character for an id, use the VOCABULARY's piece for that id. The vocabulary is the authority on what a token means. Decoding is per id, because a bulk decode cannot be corrected after the fact -- once flattened to text, the id that failed is unidentifiable. Verified end to end against the live pod: decoder alone : 'nt ? pi? facot? perci? cos?' with vocab : 'nt a-grave piu-grave facota-grave percie-acute cosi-grave' This needs the ids, which vLLM only reports when its server is launched with --return-tokens-as-token-ids (the per-REQUEST field is ignored on the chat path: chat_completion/serving.py:1179 falls back to the server setting only when the request field is None, and the id form is reached through _get_decoded_token). That flag is now in Aruba's global vllm.extra_args, which is what a pod receives -- the per-model `vllm` block in models.json does NOT reach a pod, which cost two test cycles to discover. An id the vocabulary also cannot name keeps its marker, and the ASCII-skeleton guard still decides whether the repair as a whole is safe to apply. tests: +3 (45 in test_ocr_accent_repair.py) pinning the vocabulary fallback, the unknown-id case, and that decoding stays per id. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`return_token_ids` added to the vllm entry of _ENGINE_FORWARDED_FIELDS, so a rented pod's vLLM backend can be asked for raw vocabulary ids -- without which the v0.3.31 repair can never fire on a RunPod-served model, which is every OCR model on the Digesta install. Inert by default: engine_config_for_pod skips falsey values, so adding the field cannot change an install that has not set it. Set on Aruba for `datalab-to/surya-ocr-2` ONLY, through its models.json `vllm` block rather than globally. The flag changes what the `token` string inside logprobs looks like and that install also serves qwen38-awq and diogene-alpha on vLLM pods; nothing in coderai reads that field except the OCR repair, but a client might, and there is no reason to find out the hard way. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The byte channel is a dead end on 0.27.1 -- measured: `content` comes back with one U+FFFD per accented letter AND the per-token `bytes` come back EMPTY (257 of them on one page, no byte >= 0x80 anywhere), so neither output carries the character. But the token IDS are not lossy, and surya's OWN tokenizer decodes them correctly: decode([2131, 2104]) == 'e-grave', measured in its venv. So the repair now asks for ids and decodes them itself, bypassing vLLM's detokenizer entirely. Both halves, because each alone does nothing: 1. THE WORKER reads vLLM's `token_id:{id}` form, decodes with surya's tokenizer (lazily loaded and cached -- the worker must start while the venv is building, and a clean page must not pay for a tokenizer), and keeps the byte route as the fallback for any stack that reports bytes honestly. Ids are tried FIRST because on 0.27.1 the bytes lead nowhere. All-or-nothing on the id list: a partial one would decode to a partial page. The ASCII-skeleton guard still applies, so a decode of the wrong call cannot be substituted. 2. THE PROXY stops dropping the request. `return_tokens_as_token_ids` is a per-request field on vLLM's ChatCompletionRequest, but sending it to a pod changed nothing -- because codai/backends/vllm.py::_chat_payload builds an explicit whitelist and silently discarded it. That is exactly the failure its own neighbouring comment warns about ("a proxy that edits the request decides for it"), five lines further up. Now forwarded, gated on `vllm.return_token_ids`, and only alongside logprobs since it describes how logprob tokens are rendered. `vllm.return_token_ids` is off by default (it changes what the `token` field looks like, and only this one consumer reads it) and is in VLLM_FIELDS, so the single checkpoint with a byte-level vocabulary can ask for ids without changing what every other vLLM model reports. A config field rather than relaunching the server with --return-tokens-as-token-ids: same effect, no pod rebuild. This is a fix ON 0.27.1. It makes the coderai-vllm-ocr image (vLLM 0.20.1) a CONTROL that confirms the diagnosis rather than the only way out. tests: +11 in test_ocr_accent_repair.py (36 -> 47) covering the id form, the all-or-nothing rule, ids-before-bytes, the fallback when a decode fails, the skeleton guard on the id route, and the proxy forwarding with its logprobs gating. 912 passed across the ocr/front/config/admin/runpod/vllm/backend selection; the 9 failures are pre-existing (broker x6, studio, whisper x2) plus test_cluster's RPC check, which depends on llama_cpp being installed in this venv and touches nothing changed here. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
CONFIDENCE. chandra hard-coded `conf=0.0` and olmOCR reported nothing at all. chandra's code even argued for it: "the model reports none, and inventing one would be read as a measurement". The first half was right and the conclusion was wrong -- the number was obtainable all along, by the route surya already uses. Both are served over an OpenAI chat endpoint, so the request now asks for logprobs and the reply's mean exp(logprob) is the confidence. It is the same computation surya's own client does, so the three VLM figures are comparable with each other instead of being three different notions of "confidence". engine scope source paddle line recogniser score doctr line recogniser score surya block mean token probability chandra page mean token probability olmocr page mean token probability The SCOPE is reported too (`/v1/ocr/engines` and `meta.confidence_scope`) and it is not cosmetic: a per-line score can single out the line to re-read, a page-level one can only say the page went badly. surya calls per layout block so its figure is genuinely per-block; chandra and olmOCR generate a whole page in one call, so chandra puts the one page figure on every line and region (which is what `unreadable`'s per-spot conf reads) and says the scope is "page" rather than letting it pass for per-line precision. A server may decline logprobs, and then conf is 0.0 -- which must NOT read as "the model was hopeless". `meta.confidence_measured` separates the two. That is the same distinction the unreadable-glyph trace needed and twice got wrong in this subsystem: a zero meaning "not measured" and a zero meaning "measured as bad" are different facts, and conflating them produces confident nonsense. coderai's in-process chat path surfaces no logprobs, so it reports not-measured rather than a 0.0 posing as one. g++. Digesta got an olmOCR pod past the TLS problem and straight into a 502: Triton JIT-compiles vLLM's kernels at run time and the launcher it builds is C++, so it invokes g++ -- and `vllm.apt` shipped only gcc. With only gcc the image looks right, vLLM starts, and then the engine core dies before it registers. Added to `vllm.apt` AND to `vllm-ocr.apt`, which was copied from it and had inherited the same gap before ever being used. BOTH vLLM pod images therefore need a rebuild to carry it: coderai-vllm (which olmOCR uses, so this is what unblocks Digesta) and coderai-vllm-ocr (pushed an hour ago and affected identically). One at a time, as ever. tests: test_ocr_confidence.py, 20 tests -- the arithmetic (exp then mean, not the geometric mean), None-not-zero for an unreporting server, the page-vs-block scope, the rotated-retry carrying the retry's figure, and a registry check so a new engine cannot arrive with no answer to "what does conf mean here". Ten existing fakes updated for the new (text, confidence) contract. 858 passed across the ocr/front/config/admin/runpod selection with the same 8 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
CORRECTION. v0.3.26-0.3.28 reported `blame: "the serving path"` with `conclusive: true` and "the byte stream decoded cleanly", on data that was not there. Measured through the live gateway today: result.raw (what surya reads) : 3 markers 'nt ? propriet? della propriet? d' reassembled from logprobs : 0 markers 'nt propriet della propriet ...' entries with bytes=[] : 257 any byte >= 0x80 in stream : False vLLM 0.27.1 reports `bytes: []` for the byte-level tokens of surya-ocr-2's custom vocabulary, and not one byte >= 0x80 appears anywhere in the stream. The accent bytes are not corrupted in the logprobs, they are ABSENT from them. _reassemble accepted an empty byte list as valid data, so it built a string with the accents silently missing, found no invalid UTF-8 in it, and called the result clean. That is reading absence as cleanliness -- the exact failure the envelope in v0.3.26 was built to prevent for MISSING logprobs, walked into one level further down. Fixed: an empty byte list counts as missing. The trace now counts `tokens_without_bytes` and, when any are present, reports `conclusive: false` with a `why` that says the loss is inside the serving stack (it produced neither usable bytes nor usable text) while stating plainly that this CANNOT say whether the model's own output was sound. `unreadable.blame` is then "unknown" rather than an accusation the data does not support. Consequence for surya_repair_accents: it CANNOT work on vLLM 0.27.1 -- there is nothing to rebuild from, and that is not a tuning problem. It is kept because its guards refuse correctly: the ASCII-skeleton check catches the mismatch, so a page comes back with its markers rather than with the accents quietly deleted, and the repair will work on any stack that does report the bytes. So on vLLM 0.27.1 the real options are `chandra` (reads the same pixels perfectly) or the `coderai-vllm-ocr` image pinned to the vLLM surya validates against. tests: +5 in test_ocr_token_trace.py (42 total) pinning that empty bytes are refused by both the reassembly and the repair, that the blame goes to "unknown" rather than the serving path, and that a FULLY reported clean stream still produces the genuine verdict. 643 passed across the ocr/front/config/admin selection with the same 8 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
MEASURED VERDICT. The byte trace on a page of crisp synthetic Italian, served through the Aruba install on vLLM 0.27.1: unreadable: count=29 blame='the serving path' page 0: calls=1 with_logprobs=1 conclusive=True kinds=None why: the byte stream decoded cleanly, so the characters were lost AFTER the model: the fault is in the serving path, not the model surya-ocr-2's own byte stream was valid UTF-8 and every line came back at conf=0.9605. The model read the page correctly; something after it destroyed 29 accented characters. What that does NOT implicate: surya-ocr-2 and chandra-ocr-2 are the SAME architecture (Qwen3_5ForConditionalGeneration) on the same vLLM, and chandra's accents survive. So this is not vLLM mishandling qwen3_5 -- it is vLLM mishandling SURYA'S CUSTOM 65,425-entry BYTE-LEVEL vocabulary, in which one accented letter is two tokens and either byte alone decodes to a marker. It was also a single model call, which kills the concurrent-batching hypothesis I had been carrying. OPTION 1, shipped and on by default: ocr.surya_repair_accents rebuilds the reply from the per-token bytes in the same response. Applied in the backend wrapper BEFORE surya parses the HTML, so blocks, boxes, text and confidences are all built from correct characters; repairing further down would mean fixing the same text in several places and missing one. Rewriting a model's output is serious, so it refuses unless all three hold: strict UTF-8 decode, no marker in the result, and an ASCII skeleton identical to the original's. The third is what makes it a repair and not a substitution -- it can only ever change non-ASCII characters, so a truncation, a different block or a shifted offset all leave the original alone, markers and all. Clean text is untouched, not even reassembled, and a repaired page reports `accents_repaired: {blocks, recovered, how}` because the text is no longer byte-for-byte what the server sent. Default ON because the alternative is shipping text measured to be corrupt. Set false to see exactly what the server sent. OPTION 2, prepared: a `coderai-vllm-ocr` pod image pinned to vLLM 0.20.1 -- the version surya-ocr itself spawns (surya/settings.py:96) -- so the model can be served on the stack its authors validate, the repair can be checked against a reference rather than trusted, and the upstream bug report has a known-good comparison. Selected per model with `runpod.engine: "coderai-vllm-ocr"`, built with `build_capability_image.sh vllm-ocr`. The pin set was verified to exist before the profile was written, because a profile that cannot build is worse than none: wheels.vllm.ai/0.20.1/cu129 is live, and torch 2.11.0 / torchvision 0.26.0 / torchaudio 2.11.0 (vLLM 0.20.1's own metadata pins) all have +cu129 cp313 wheels. Qwen3_5ForConditionalGeneration is in 0.20.1's registry, so it can serve the model. No torchcodec pin: 0.20.1 does not require one, and pinning a package nothing needs is its own hazard -- the file says what to add if a build ever surprises us. NOT a drop-in for `coderai-vllm`: everything else on it would be pinned back seven minor releases. tests: test_ocr_accent_repair.py, 29 tests -- the repair itself, every guard, and the image profile. 833 passed across the ocr/runpod/config/admin/front selection with the same 8 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
coderai_engine_inflight / coderai_requests_active are load-bearing: the hourly autoupgrade's idle guard reads them to decide whether a restart is safe, and with MAX_DEFERRALS=0 a non-zero count makes it SKIP for ever. So one request counted in and never out silently stops a host taking upgrades, and the only symptom is a version that stops moving. Observed on the Digesta orchestrator today: the gauge read 13 against two real requests, the 09:01 tick logged "SKIP: 7 piece(s) of work in progress" and left the image alone, and only a manual restart cleared it. The structural leak is in the streaming proxy path. Starlette awaits a response's background task only AFTER the body is sent -- `await self.stream_response(send)` then `await self.background()` -- so when the client disconnects mid-stream the send raises, the exception propagates, and `BackgroundTask(_release)` never runs. The relay generator's `finally` DOES run in that case (Starlette closes the iterator) but not when the body is returned without streaming, so both now call `_release`, which is idempotent via a flag. `exit_request` is idempotent per rid as well, and that is required rather than decorative: two release paths firing for one request would otherwise decrement an UNRELATED in-flight request, and the count would under-report real work -- worse than over-reporting, because then a restart lands during a request. Every request is now timestamped, not just the inference ones that carry `meta`. Without that a leaked non-inference request was invisible as well as unreleasable: the count said 13 and the Tasks page showed nothing, which is how this went unnoticed. And a reaper underneath all of it (`server.inflight_max_age_s`, default 3600, 0 disables), called from the two places that ASK whether anything is in flight: * the /metrics scrape, because that is the exact moment a stale count does its damage -- the scrape that would report a lie is the right place to correct it, not to hope somebody notices; * the supervisor's drain wait, which would otherwise sit out the whole grace period for a request that finished an hour ago. An hour, not a minute: a cold RunPod pod can make a single OCR page take 14 minutes, and reaping a live request would be a worse bug than the one being fixed. Training is unaffected -- it is tracked as a JOB and the request that starts it returns at once. NOT fixed here, noted for later: `_proxy_passthrough` has the same BackgroundTask-only release for the system worker. It never calls enter_request, so it cannot leak the gauge -- it leaks an httpx stream, which is a smaller and separate problem. tests/test_front_inflight_leak.py: 14 tests, including double-release, late release after a reap, thread safety, and the source invariants for the two release paths. 299 passed across the front/metrics/registry/config/admin selection; the 9 failures are pre-existing (broker_config x5, broker_streaming, studio_composed_surfaces, whisper_server x2) and touch no file changed here. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
v0.3.25 shipped the trace as a bare list of findings, and defined the empty list as "the bytes were sound, so the loss is in the serving path". That is wrong whenever the upstream returned no logprobs: there was nothing to analyse, and the emptiness would have read as a confident conclusion drawn from missing data -- exactly the class of mistake the trace exists to stop. So the trace is now a self-describing envelope: {"calls": 7, "with_logprobs": 7, "conclusive": true, "kinds": {"invalid-utf8": 29}, "why": "...", "findings": [...]} and `unreadable.blame` is one of "the model", "the serving path" or "unknown". `conclusive` is false when no calls were recorded (the hook never fired) or when none carried logprobs, and then blame is "unknown" rather than defaulting to either culprit. surya asks for logprobs by default (SURYA_INFERENCE_LOGPROBS = True) and the OCR gateway forwards the body verbatim precisely so they survive, so the normal case is conclusive. But an upstream that declines them is a configuration, not a bug, and it has to be visible instead of silently inverted into an answer. Also: a trace analysis that throws now reports {"conclusive": false, "why": "trace analysis failed: ..."} rather than no key at all, so "the analysis broke" and "the bytes were fine" cannot look the same either. tests/test_ocr_token_trace.py: 37 (was 30), including the inconclusive paths. 825 passed across the OCR/config/admin/runpod selection, same 10 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`ocr.surya_token_trace` (off by default) reports the per-token bytes at every place a surya page lost a character. It exists to answer one question that the finished text cannot: whose fault is it? invalid-utf8 the reassembled stream does not decode -- a lone UTF-8 lead byte with nothing to complete it. The MODEL emitted a broken sequence; the text faithfully renders a wrong answer. model-emitted the stream decodes and a token IS the replacement character. The model chose it. no findings the bytes were sound and the markers appeared anyway, so the loss happened AFTER them -- OURS, in the serving stack. That last case is why an empty trace is a result, not a failure. CORRECTION, and the reason this was needed. v0.3.21 claimed U+FFFD could not be a byte-level artefact, on the grounds that `a`-grave was a single token so one marker had to mean the model chose it. That was wrong. Measured: 'a-grave' utf8=[195, 160] ids=[2131, 2258] pieces=['A-tilde', 'l-bar'] decode([2131]) = U+FFFD <- a lone lead byte, exactly ONE marker decode([2131, 2104]) = 'e-grave' surya-ocr-2's vocabulary is BYTE-level, so one marker per accented letter is precisely the signature of a byte-level failure -- the thing it was claimed to rule out. The vocab lookup was misread too: `vocab['a-grave'] = 2160` does not mean the letter is token 2160; in a byte-mapped vocabulary that key is the glyph standing for byte 0xA0, which alone also decodes to one marker. The "one marker vs two" discriminator does not exist. Digesta's charset instinct was closer to the mark than the rebuttal. base.UNREADABLE and docs/ocr.md now carry the correction rather than the refuted argument. What IS established, measured on the Aruba install at the same DPI through the same transport on perfect synthetic Italian (crisp vector glyphs, 34pt, 250 DPI): surya 29 U+FFFD markers, zero accents surviving chandra 0 markers, all 12 accents intact So the document is exonerated. Also notable: the em dash survived surya intact while every two-byte accent died, and surya emitted U+1EEA (Vietnamese) in place of one -- no decoding bug can fabricate a character that was never in the input. Implementation notes: - surya DROPS this data. Its recognition predictor does pass it on (`BlockOCRResult(..., raw_logprobs=out.logprobs)`) but the model never declares the field and pydantic defaults to extra="ignore", so it is constructed and silently discarded -- their own comment calls it "phase 2 use". Verified by construction: the attribute does not exist on the result. So the value is taken one layer earlier, wrapping `_generate_one`, which still has it. The wrapper only reads and swallows every error: a diagnostic that could break a page would be worse than no diagnostic. - The scan reassembles the WHOLE byte stream and walks it as UTF-8 rather than judging tokens pairwise. Per token is meaningless here -- one character legitimately arrives as two or three tokens, each "invalid" alone -- and the first version of this did judge them pairwise and flagged every accent and every em dash, the exact false positive that would make it worthless. - Capped at 40 findings per page, two tokens of context each side, because a lead byte is only wrong in the light of what followed it. - Clean pages are byte-identical to before: the key appears only when the text actually contains a marker, and it is reported ONCE document-wide under `unreadable.tokens` rather than repeated in every page's meta. - The flag is in _pool_params, because the hook patches surya's backend at LOAD -- without that, toggling it would appear to save and take effect never. A third discriminator needing no code: serve the same weights through llama.cpp (`datalab-to/surya-ocr-2-gguf`, `surya_serve: "llamacpp"`). A different detokenizer on identical weights would convict vLLM outright. tests/test_ocr_token_trace.py: 30 tests. 818 passed across the OCR/config/admin/ runpod selection with the same 10 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A fifth OCR engine. Chandra (datalab-to/chandra-ocr-2) is from the same lab as Surya and is driven the way olmOCR is — one page image over an OpenAI chat endpoint, three serving modes (model | vllm | server) — but its `ocr_layout` prompt asks the model to answer as one <div> per layout block carrying data-bbox (normalised 0-1000) and data-label. So it is the only engine here that returns text, boxes, labelled regions, tables AND page metadata in one pass, with no separate detector. Why it was worth adding now: it is a DIFFERENT MODEL FAMILY from surya-2 (Qwen3-VL's tokenizer, against surya-2's own 65k WordLevel vocabulary) reached over the same transport, at the same pixel ceiling, from the same rasterised page. That makes it a cheap controlled experiment on the U+FFFD accent loss: accents intact under chandra and lost under surya points at the model; both losing them points back at the document's pixels, and then pdfimages -list is the next thing to read. Implementation notes: - NO new pip dependency. The two prompts are mirrored verbatim from chandra/prompts.py (down to the Python repr of the allowed-tag list, which is trained-on wording), the input scaling is chandra's own scale_to_fit reimplemented, and the reply is split with stdlib html.parser rather than pulling bs4 and markdownify into the main venv for a tag split. - Chandra's pixel cap is 3072x2048 = 6,291,456 — the SAME numbers as surya-2 — and it snaps to a 28-pixel grid for the Qwen vision tower. So chandra_dpi defaults to 250 for exactly the reason surya_dpi does. Boxes are de-normalised against the image the MODEL saw, and the size the caller rasterised is reported as meta.source_size when they differ: de-normalising against the pre-scaling size would put every box on a scaled page in the wrong place. - A repetition loop answers 200. A VLM that loses its place repeats the last thing it saw until the token budget runs out, and the result looks like a successful page. chandra_max_retries (2) re-asks with a raised temperature — a greedy decode falls into the same loop twice — and a page that still loops is returned flagged with meta.repeat_detected rather than handed on silently. - The parse never loses page. No blocks at all, content outside any block, an unclosed final div (routine at the token limit), a bbox that will not parse: each keeps the text. A bbox that cannot be read yields NO region rather than a full-page one, because a wrong box looks like a real answer. - Table cells get a space, not a newline, so a table does not flatten to "VoceImporto". License: narrower than Surya's GPL. The code is Apache-2.0 but the WEIGHTS are a modified OpenRAIL-M — research, personal use and companies under $2M funding/revenue, no use competing with Datalab's API. So it is gated on chandra_accept_license, and the pod path differs from surya's on purpose: asking a pod for surya by name IS the GPL acceptance, while chandra's is only ever inherited from a decision already recorded in this install's config. A pod cannot accept a licence on an operator's behalf. lang= is accepted and does not steer it (language_aware: false), like surya-2 and olmOCR: 90+ languages, no language input. So it keeps ONE pool whatever is asked — a language-keyed pool is a second set of loaded instances holding VRAM. Wired through: engine registry, licence gate, per-engine DPI, the vLLM stop path (its VRAM is the subprocess's, not a pool's), the capability catalogue (which now reports boxes/regions/tables from chandra_layout, because with layout off there genuinely are none), model-name aliasing, the automatic fallback chain, pod planning + env seeding, the frontproxy's local-OCR check, models.json capability advertisement, the pipeline step's engine list, admin save/load and the Settings OCR card. tests/test_ocr_chandra.py: 61 tests. 788 passed across the OCR/config/admin/ runpod selection with the same 10 pre-existing failures (broker_config x5, studio_composed_surfaces x3, whisper_server x2). test_ocr_fanout's catalogue test asserted a frozen engine set; it now asserts against the registry, which is what "lists every engine" was always supposed to mean. Also fixed while in there: the Settings page used `parseInt(v) || default` for numbers whose zero is meaningful, so a deliberate 0 saved as the default. New _num() helper only falls back for an empty or unparseable field. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/ocr has accepted `lang` since the beginning, with the comment "reserved": it was parsed and dropped. A client asking for Italian got whatever the install was configured for and no way to tell. It is honoured now, on /v1/ocr, /v1/ocr/batch and the pipeline step, with precedence request > ocr.<engine>_lang > ocr.lang > auto. The request beats the engine-specific setting deliberately: that setting is an install's default, while a caller naming a language knows something about the document in hand. Asking for `auto` explicitly is NOT the same as sending nothing, so an install configured for Italian can still be asked to work it out. The default is auto-recognition rather than any particular language, because a server that has not been told what it is reading should not assume one. For surya-2, olmOCR and docTR that is simply what they already do: they are multilingual and have NO language input to constrain -- surya-2's entire prompt is "OCR this block image to HTML." and its own module warns the wording is the training-time contract, so there is nowhere to put a language and nothing to gain. Rather than let `lang` look effective everywhere, every response now carries what was applied AND whether the engine that served could act on it ("lang": "it", "lang_applied": false), and /v1/ocr/engines reports the same per engine so a client can choose before sending. PaddleOCR is the exception that shapes the rest. It loads a language-specific recognition model and its options are sent once at load, so an instance is built for one language and cannot be re-languaged. Hence: `auto` becomes ocr.paddle_auto_lang (default `latin`, a SCRIPT family rather than detection -- PaddleOCR's own default is `ch`, a worse guess for most documents than no guess), and paddle gets one pool per language, keyed on the model it is BUILT with so `auto` and `latin` share a pool instead of loading it twice. The multilingual engines keep a single pool whatever is asked. That is not tidiness: a language-keyed pool is a second set of loaded instances holding VRAM, and a production install runs 24 surya instances. Both pool-iteration sites already use .values(), and _Pool keeps the plain engine name, so eviction and the VRAM releaser handle the new keys unchanged. ocr.lang's shipped default moves "it" -> "auto"; installs that set it explicitly (Digesta production does) are unaffected. ocr.surya_langs stays as the legacy setting it is -- it reaches only the <=0.17 recognition API and does nothing on the pod path. 21 tests. The 8 failures in test_broker_config / test_studio_composed_surfaces / test_whisper_server_local_models are pre-existing, verified at HEAD. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
ocr: olmOCR has to trust a pod's certificate itself, and U+FFFD is one token not two bytes (v0.3.22) olmOCR could not reach a rented pod at all: CERTIFICATE_VERIFY_FAILED. A pod on the direct-TCP path serves HTTPS with a certificate signed by THIS install's own CA (codai/api/pod_tls.py), at a public IP that was not known when the certificate was issued, so stock `requests` fails it twice over. Surya never hit this because surya attaches through the fan-out gateway, which learned to trust pod certificates in 0.3.16. olmOCR talks to the upstream the dispatcher chose DIRECTLY, so it needed the same trust of its own -- and the gap only surfaced when olmOCR was asked to cover for surya, which is exactly when a fallback engine must not be the thing that is broken. It now goes through codai/api/pod_http.py, which verifies https-to-a-bare-IP against our CA with the hostname check off and hands every other URL -- localhost, a named server, another coderai -- to requests untouched. Swept the rest of the tree: every other raw requests call targets a localhost worker or RunPod's own public API, where pod_http would be a pass-through anyway. test_ocr_olmocr's HTTP-status test patched sys.modules["requests"] wholesale, which now breaks pod_http's own import of it; it patches pod_http.post instead and asserts the same two statuses. Also records what the accent investigation actually established, because the natural reading of U+FFFD is wrong and expensive. It is NOT a charset bug: U+FFFD is a real entry in surya-ocr-2's vocabulary (id 65421 of 65425, the only id that decodes to it), the tokenizer is WordLevel and character-level for Latin so an accent cannot be split across tokens, and the decisive detail is the COUNT -- a byte-level mis-decode yields TWO markers per accented letter (a-grave is two UTF-8 bytes, each one undecodable), and what arrives is ONE. One marker is one token. Every other hop was checked: _html_to_text is regex plus html.unescape and survives literal and entity forms; the isolated-venv worker emits json.dumps with ensure_ascii=True, so the pipe carries pure-ASCII escapes and the locale cannot matter; the gateway forwards raw bytes. And it corrects the previous commit's claim. Raising DPI is right on its own merits -- 200 really was feeding surya 62% of the pixels it accepts -- but a production document came back with the same markers at 200 AND at 300, which is evidence against too-few-pixels being the whole story: a perception failure would be stochastic, not every accent every time. The docs now say what distinguishes the remaining candidates (the document's embedded scan resolution, which `pdfimages -list` reports and which no rasterisation DPI can improve, versus the serving stack) and how to tell them apart: OCR perfect synthetic Italian through the same path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Accented letters came back from surya as U+FFFD -- "piu`" as "pi?", "facolta`" as "facolt?", nine of them in one page, as real EF BF BD bytes. That is exactly what a byte-boundary decoding bug looks like: a` is C3 A0, and byte-level BPE splitting it across two tokens is the classic cause. It is not that. U+FFFD is a REAL ENTRY in surya-ocr-2's vocabulary -- id 65421 of 65425, the only id that decodes to it, with <unk> a separate token at 65424. The tokenizer is WordLevel and character-level for Latin text, so a` is a single token (2160) and an accent CANNOT be split across two: no partial-UTF-8 artefact is possible, and per-token decoding is byte-identical to decoding the whole string. Surya reads text only from message.content, never from the per-token logprobs, and the fan-out gateway forwards the upstream body as raw bytes. So the model predicted "there is a character here I cannot identify" and every hop delivered that faithfully. Two things follow. It is a RESOLUTION problem. Surya caps its own input at 3072x2048 = 6,291,456 pixels in scale_to_fit, which is called with DEFAULTS ONLY -- no surya setting raises it, IMAGE_DPI_HIGHRES included. Measured, for A4: 200 DPI reaches the model as 3.89 Mpixels, 250 as 6.03, and 255 saturates it; 300 and 400 are downscaled back to the same 6.23. The shared ocr.dpi = 200 was therefore handing the model 62% of the pixels it accepts, and a diacritic sits above the x-height at ~3 pixels tall. So DPI is now chosen per engine (ocr.surya_dpi, default 250) and resolved BEFORE rasterising; paddle and docTR have their own optima and keep ocr.dpi. The lookup is generic (f"{engine}_dpi"), so another engine only needs the field. Resolving the engine first also refuses an unknown one before a large PDF is rasterised rather than after. It must not be SILENT. "pi?" indexes, searches and reads wrong for ever, and for judicial documents a known unknown beats a quiet corruption. Lines, pages and the document now count their unreadable glyphs and locate each one with the confidence of the line it sits in -- the finest granularity that exists, since surya reports one mean token probability per block, not per character, and that is also the signal worth acting on: a low-confidence line with an unreadable glyph is a line to re-read at a higher DPI. The key is OMITTED when there are none, so a clean response is byte-for-byte what it always was, spots are capped with a truncated count, and the text itself is never rewritten -- stripping the marker would destroy the only record of where the loss is. 20 tests, including the arithmetic that chose 250, kept so a later edit has to face it. The 8 unrelated failures in test_broker_config / test_studio_composed_surfaces / test_whisper_server_local_models are pre-existing -- verified identical in a clean worktree at HEAD. Co-Authored-By:
-
- 11 Oct, 2026 20 commits
-
-
Stefy Lanza (nextime / spora ) authored
/v1/models answers the moment a vLLM process starts — minutes before it can serve, because the checkpoint still has to be downloaded (every pod re-downloads it without a network volume) and the KV cache built on first use. The pool called that ready, every client read it as usable, and the whole model load landed inside somebody's first request, where it is indistinguishable from a hang. That is what cost today. One OCR page took 807s end to end — ~400s image pull, ~400s weights — while the client timed out at 180s and then 300s and concluded the serving was broken. It was not: nobody had ever waited long enough, and the only reason we know is a probe run with a 1400s timeout, which came back 200 with text and conf 0.9019. So the pool now sends one token against the served model before calling the pod ready. The pod bills through the load either way; this only decides whether the wait is visible as a pod that is not ready yet, or hidden inside a request that looks stuck. `pods_ready` becomes a signal a client can act on, which is what every client already assumed it was. A failed warm-up is reported and the pod used anyway: it may still serve (an engine that takes no chat completions, a model whose warm-up shape we guessed wrong), and rejecting a pod we have already paid to boot over a diagnostic request would be worse than the hidden latency this removes. Off with warmup_on_boot=false or warmup_timeout_s=0. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A rented pod is the one machine in the system whose logs cannot be read. RunPod publishes no pod-log API — its REST spec lists 23 routes and none serve logs, which is why every attempt returns HTTP 400 — and boot.sh writes its phases to a file and then execs uvicorn, whose output goes to the container's stdout and nowhere else. /boot therefore covered the way up and nothing after it, and _dump_pod_logs only ever ran on a failure, printing into the orchestrator's log. That cost an evening. Three pods were investigated in one session and two were already destroyed by the time there was a question to ask — one reclaimed by RunPod mid-request, taking the only evidence with it — so "the serving hangs" stayed a hypothesis for an hour. It was not hanging: a page took 807s, most of it the weights download every pod repeats because global volumes are console-only, and every client had given up at 180-300s. So the pod keeps a bounded ring of its own stdout and stderr (codai/api/podlog) and serves it at /logs with a cursor, and the pool drains that on every scaler tick into <config_dir>/pod-logs/<pod_id>.log — kept AFTER the pod dies, which is the whole point. It also drains once more before tearing a pod down, because the interesting lines are the last ones and they are only readable while the pod exists. A pull, not a push: the orchestrator already reaches pods to serve requests, so the bearer token and pinned TLS apply unchanged and a pod with no outbound route still gets its log kept. Bounded at both ends, since neither disk is free, and a gap in the pod's ring is written into the file instead of passing silently. The stdout tee writes through to the real stream on every failure path — a logging aid that could break printing would be worse than none. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod reclaimed an A40 mid-document today. The pool went on reporting it healthy, so every page chose the same corpse: the in-flight count climbed, the pod was never dropped, and nothing drained. From the client that reads as "requests neither complete nor time out", and it is what wedged the test. RunpodBackend has always handled this — _unreachable() drops the pod and the request goes again. The fan-out gateway never did: it returned 502 and left the dead pod in the pool as healthy. So a lease can now report its upstream dead, which drops the pod, clears the cached candidate set and releases the slot; the page is retried ONCE against whatever replaces it, and a third attempt would just be a retry loop against a real outage. Only connect-level failures count as dead. A read timeout is a model taking its time over a dense page, and discarding a healthy pod for that would be an outage of its own. Also: the first version of that handler could not work. It used a metaclass with __instancecheck__ so `except _DEAD_UPSTREAM` would match httpx's errors — but `except` matches on the class hierarchy at C level and never consults __instancecheck__, so it caught nothing. A test caught it; the check is now an explicit isinstance() where it cannot lie. And the registry path again: 0.3.15 keyed the fallback off XDG_CONFIG_HOME, which the container launcher exports only into the processes IT starts. A diagnostic script entering the container another way printed a THIRD registry path and created a second pod TLS CA. The fallback now reads CODERAI_CONFIG_DIR, the variable the launcher actually sets, so every process agrees however it was started — and since pod_tls.tls_dir() derives from the registry path, the CA follows. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
refuses our token is not an asset (v0.3.17) Both from watching a real OCR run. THE LOST INTENT. A client asked for the fleet back while a page was still being read. The pods correctly stayed — work is never dropped to save money — but the intent was then forgotten: the pool fell back to idle_timeout_s (900s there) and billed for a fleet nobody had asked to keep. It took a second teardown call, four minutes later, to actually release it. The pool now remembers: pods kept only because they are working are retired the moment they fall idle, the same "do not wait out the timeout" treatment a closed schedule window already gets. Asking for warm pods again cancels it. Also fixes the reply and the log, which said "floor 1" when the 1 came from work in flight. That reads as though the operator had asked for the pod — the opposite of what it meant. Demand and configuration are now reported separately ("needed" and "floor"), because they are two different reasons a pod survived. THE ADOPTED POD THAT 401s. After a restart the pool adopts the pod it left running instead of renting another, and carries that pod's bearer token across through the registry — when one was recorded. With none recorded it fell back to our own token and never checked the result, so a pod launched with a different one answered 401 to every request for the rest of its paid life. Seen today, right after a restart: "reusing the pod already running" followed by 401 on every OCR page. The health probe that found it proves nothing here — a coderai pod answers /healthz to anybody. Adoption now probes an AUTHENTICATED path with the token it would use, and a pod that refuses it is terminated rather than adopted: with one registry per deployment (0.3.15) an unrecorded key is a lost key, so that pod can never serve anyone, and paying for it is the only outcome worse than killing it. An unreachable pod is NOT treated as unauthorised — that stays with the health machinery, which has a budget for it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
First real document, first real pod: it came up at https://194.68.245.79:22198 and every page came back 502 — OCR upstream pod failed: [SSL: CERTIFICATE_VERIFY_FAILED] unable to get local issuer certificate A direct-TCP pod serves the certificate coderai's OWN CA signed, on a bare IP that certificate can never name. codai.api.pod_http has always known this: it pins to pod_tls.ensure_ca() and switches hostname checking off. The fan-out gateway is async and could not reuse that requests-based module, and I gave it a bare httpx client — so the one hop that now carries every OCR page was the one hop with no pod trust. It keeps TWO clients per event loop, chosen by pod_http.is_pinned_url: the pinned context trusts only our CA, so using it for everything would reject a cluster node whose certificate a public CA signed, and sharing one client would let whichever upstream connected first decide the policy for both. Worth recording: this was never fan-out-specific. With serve_fanout off, surya attaches straight to that same https:// IP with its own httpx and fails identically — `model` mode has never worked against a TLS pod. The gateway is what makes it fixable in one place. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two things, both from today's live incident. THE SPLIT REGISTRY. _pod_registry_path() fell back to ~/.coderai whenever config_manager was not initialised in the calling process. The container's launcher puts the real config dir one level deeper — it exports XDG_CONFIG_HOME="$CODERAI_CONFIG_DIR" and uses "$CODERAI_CONFIG_DIR/coderai", which is also where platform_paths.user_config_dir() lands. So a process with the config manager wrote one registry and a process without it read another, and the pods each owned were orphans to the other. That is what made the documented mutual-kill loop recur: pod created 19:24:56, terminated as untracked 19:24:57. The fallback is now the same XDG-aware path the launcher computes, and the resolved path is logged once — a registry that splits silently is invisible until it costs a GPU. CLIENT-FORCED TEARDOWN. `pods: 0` released a client's lease but gave nothing back: the pods then waited out scale_down_after_s, or idle_timeout_s before that. The automatic path is slow on purpose — it infers from a momentary reading, and a cold pod costs minutes to replace — but a client that has just finished a batch is not inferring. `teardown: true` sheds the surplus at once, skipping both the window and the one-per-pass limit, and answers with what happened. It still cannot scale through the operator's min_pods, and a pod with requests on it drains rather than dying under them. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Observed live on the Digesta orchestrator while testing OCR: pod created 19:24:56, "terminated STALE pod … not tracked by any pool" 19:24:57. The pool then waited out its 900s boot timeout on a pod that no longer existed, every OCR request 503'd, and the next attempt rented another. This is the race the source already documents from an earlier incident — "the nvidia engine created a pod and the radeon engine terminated it four seconds later, then the reverse — a mutual kill loop that rented and destroyed pods until stopped" — and the defence against it was supposed to be the age guard, "never reap a pod younger than this, whatever the registry says". It was written as `age is not None and age < REAP_GRACE_SECONDS`, and list_pods() returned {id, name, status, cost_per_hr, image} with no timestamp of any kind. So age was None for every pod, the condition was never true, and the guard abstained on exactly the pods it existed to protect. A guard that fails open is worse than none: it reads as covered. Two changes. The guard now fails CLOSED — an undatable pod is left alone and says so — and the listing carries runtime.uptimeInSeconds so it can measure instead of abstaining. Not reaping costs one more cycle; reaping a live pod costs the work on it and bills in a loop that never serves a request. What made it recur here is worth recording: this install has TWO config dirs, so the cross-process registry that shields a new pod exists twice and each process reads the one the other does not write. That is the deeper fault and is not fixed by this commit. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two faults, one cause: `local` was both the default and the only engine mode that would quietly settle for the CPU. Surya's own settings pick the device and fall back cuda -> mps -> cpu (surya/settings.py TORCH_DEVICE_MODEL), so `local` on a machine with no CUDA ran a VLM pipeline on the processor: minutes a page, a result that looks perfectly normal, and nothing anywhere saying why. paddle and docTR already refuse in exactly that situation — surya was the exception. It now refuses too, with a 503 that names both ways out, and the worker is TOLD its device rather than left to guess. Detection fails closed: torch is asked first (it is what surya will ask), nvidia-smi second, and if neither can answer the answer is no. And `surya_serve` no longer defaults to `local`. An install that said nothing got the one mode that runs the model on this box — on a GPU-less orchestrator whose whole job is renting one, that is the worst possible default. It now defaults to `model`, which adapts to wherever the checkpoint is served and asks for surya_model_id when it has not been told. Safe to change: both production installs set the mode explicitly (`model` on Aruba, `vllm` on zeiss), so nothing was relying on the old default. Running on the CPU is still fully supported — surya_use_gpu = false — it just has to be a decision now. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pool rents at whatever the market offered when each pod was started, so one fleet can hold a $0.35 card and a $0.79 card doing identical work. Shedding the cheap one saves less than half as much, and which to prefer is the operator's call, not ours — hence a checkbox rather than a new default. With scale_down_costliest_first the dearest surplus pod goes first, tie-broken by the idlest and then the coldest cache. Cost decides WHICH pod goes, never whether one goes, and a dear pod with requests on it is drained rather than dropped — so choosing it costs time, not work. Worth weighing before switching on: an on-demand pod costs more than a spot pod, so "dearest first" prefers to keep the interruptible ones. Also fixes the default tie-break, which was inverted when scale-down was written: `-last_used` picked the pod used MOST recently, discarding the warmest prefix cache of the set. It now retires the coldest. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Only half the scaler existed. The pool adds a pod once every pod it has carries scale_up_inflight_per_pod requests, and pods went away again only through the idle reaper — which fires on a pod with NO work and no request for idle_timeout_s. That misses the expensive case. A pool that grew for a burst and is now serving a trickle keeps every pod: each is touched often enough that last_used never ages out, each is a fraction busy, and the bill stays at the peak for as long as the trickle lasts. With max_pods raised to 64 that is the difference between a few dollars an hour and a few hundred. So the pool also asks how many pods the CURRENT concurrency justifies and retires the excess. The rules that matter are not "fewer pods": * work is never dropped to save money. The pod retired is the least loaded, and if it still has requests it is marked draining — no new work, and terminated only once it has finished what it had; * one pod per pass, after the surplus has persisted for scale_down_after_s, and the window restarts after each retirement. A dip of a few seconds must not cost a pod wanted again seconds later: a cold one takes minutes; * a quiet pool is NOT this routine's business. With nothing in flight, teardown stays entirely with idle_timeout_s — a number the operator chose, weighing the bill against a cold start — so this never silently shortens it. A draining pod is also excluded from the growth decision, which reads the LEAST loaded pod: an emptying drainer would otherwise answer "no need to grow" with the one pod that cannot take the work. Two existing tests had to change. A _Pod stub stood in for PodHandle without the new field, and a source-text assertion used a 600-character window that a new field plus its comment pushed data_center out of — now asserted against the whole class, which is what it was always about. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The gateway built an httpx.AsyncClient for every request, so each page paid a fresh TCP and TLS handshake to the upstream. With surya_instances: 24 that is 24 streams reconnecting per page, against a round trip that already dominates the cost of reading one. One pooled client with keep-alive instead, so a document's pages reuse the connection. Correctness-neutral; it only ever cost time. Keyed by the event loop, because an AsyncClient binds to the loop that created it and coderai runs more than one (the OCR pool keeps its own). The key is the loop OBJECT held weakly, not id(): keying by id would leak an entry per dead loop and, since CPython reuses addresses, could hand a new loop a client bound to the closed one that used to live at that address — a fault that would only appear under the load this change exists to handle. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The fan-out gateway lived at /v1/ocr/engine/<lease>/… and the new catalogue at /v1/ocr/engines. They do not collide, but both auth gates exempt the gateway by PREFIX — so any later route under /v1/ocr/engine* would have inherited that exemption silently, and the two names are indistinguishable at a glance in a middleware condition. The gateway is now /v1/ocr/attach/<lease>/…: internal by name, and nothing a public path family will grow into. Tests assert /v1/ocr/engines still requires ordinary authentication at both gates, and that a plausible future sibling (/v1/ocr/engine-stats) does too. Free to rename because nothing consumes the lease URL yet — it is minted per attachment at run time and never written down. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Naming an engine an install had not enabled was a 400, and there was no way to find out beforehand — the client had to know, or guess and handle the refusal. GET /v1/ocr/engines answers it from this install's own config: every engine with `enabled` and, when it is not, a `reason` that names the setting to change (an unaccepted licence is reported as such rather than as merely disabled, because that is a different checkbox). Each row also carries `kind` — an engine owns its weights, a pipeline drives a model — the `aliases` that also resolve to it, and `returns`, so a caller that needs coordinates can see that olmOCR has none rather than discovering it in an empty field. The top level reports both what the operator wrote as the default and what it resolves to. Where the model runs is deliberately not in the answer. It is configuration, it can differ between two requests, and a client that could see it would start depending on it. A test asserts the payload mentions no pod, node or address. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
docTR called .cuda() when torch happened to see a card and otherwise simply stayed on the CPU, so a GPU-less install read every page many times slower with nothing in the response to say why — the result looked normal, it just arrived minutes late. Nothing else in coderai quietly relocates GPU work, and paddle refuses outright in exactly this situation; docTR was the odd one out. `doctr_use_gpu` is now a requirement: no CUDA device means a 503 that names the setting. Running docTR on the CPU is still fully supported and unchanged — it just has to be asked for, with doctr_use_gpu = false. The test drives the real load() with docTR and torch stubbed, rather than re-implementing the placement rules beside it; reintroducing the fallback makes it fail, which is the only reason to trust it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three corrections to the fan-out, all the same mistake in different places: the code was deciding things the configuration had already decided. Nodes are no longer discovered. Asking every cluster node what it holds and using whoever answered is configuration by probe: it puts work on a machine nobody chose and moves it the moment a different node replies first. A node serves a model because the model is PINNED to it — `engine` on the entry, the same hard constraint the front router already honours, comma-separated when several nodes should share the work. A pinned node that does not hold the model is reported, not silently replaced. The decision is not an OCR decision, so it does not live in codai/ocr: any model, any caller that needs an endpoint rather than a proxy hop, gets the same answer from codai/models/placement.py. For requests that pass THROUGH coderai this was already true — the router honours the pin, treats a node as an engine by name, reaches RunPod through the backend. This is that decision for a caller that must hold an address itself, which is why OCR needed it. And "how busy is too busy" is the model's own scale_up_inflight_per_pod, on the model page, instead of the parallel ocr.serve_spill_inflight knob I had added — one number per model, already there, already meaning exactly this. A model with no RunPod block has nothing to rent, so load alone never moves its pages. Finally, the caller: resolve_engine accepted the four pipeline names only, so a client naming the model it can see in /v1/models got "unknown OCR engine". Both now work — engine=surya and engine=datalab-to/surya-ocr-2 reach the same place — because which pipeline reads a page, and where its model runs, are ours to know and not the client's. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
An OCR pipeline is handed one server address and drives it itself, which is what keeps its page images, its strict json_schema and its logprobs intact. One address cannot spread load, and three things followed from that: * a pool grows on in-flight requests counted as they pass through it. An attached engine never passed through, so the pool saw no load however hard OCR ran, never grew, and the engine had one pod's URL anyway; * several cluster nodes serving the model resolved to whichever answered first, for ever, however busy it became; * a local model configured to burst to RunPod never did: that trigger lives on the proxy path an attached engine bypasses. So the engine gets a lease on an address of ours, and codai/ocr/dispatch.py chooses per request: a pool that is the model's configured home takes every page (picking the least-loaded pod, renting when they are all at the threshold); otherwise this machine and the nodes holding the model compete on load — including the load a node reports for its whole install, so a node busy with someone else's work is left alone — and the overflow past ocr.serve_spill_inflight rents instead of queueing. An explicit `backend` pin still wins over discovery: that was a decision, and load-aware picking must not quietly repeal it. The hop is allowed to exist only because it decides nothing: body bytes in, body bytes out. The losses it is there to avoid came from PARSING a request and re-emitting it through the backend abstraction. Two gates stood in the way and neither could be satisfied the usual way: the engine has no API token, and surya's /health and /v1/models probes send no headers at all (bare httpx GETs), while refusing to attach if /health is not 200. The lease is therefore read from the URL, and the engine-only internal gate exempts that path alone — tested to be no wider than that. olmOCR's model mode asks the same dispatcher, and still takes the in-process route when the answer is local, where the model manager's loading, eviction and queueing belong. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Surya asks every request for logprobs and averages exp(logprob) over the generated tokens to score each line. coderai accepted the parameter, never sent it, and hardcoded `logprobs: None` in every response builder. Surya's fallback is what makes that expensive: with no logprobs it does not report "unknown", it reports a literal 1.0 (recognition/__init__.py:248, :302, layout/__init__.py:97). Every line of every page comes back perfectly confident, the guesses included, and nothing downstream can tell which judicial document needs a human to look at it. So the round trip is wired where it can exist: the proxy backends (vLLM, RunPod) forward `logprobs`/`top_logprobs` and hand back what the server measured, buffered and streamed, accumulating the streamed pieces into one response-shaped object. The manager asks the backend's signature rather than negotiating by TypeError, which would have called it twice for a kwarg it never had. A request that did not ask is sent exactly as before, and a backend that cannot answer still returns null. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Everywhere else in coderai an engine is what RUNS a model. `ocr.default_engine` borrows the word for two different kinds of thing: paddle and docTR are runners with their own weights, while surya-2 and olmOCR are pipelines over a model that vLLM runs. Only the second kind has to be told where a server is — and that is the same question the model layer already answers for everything else. So answer it in all the places a model can actually be served. A cluster node was the one that was missing: a node is a whole coderai registered with the head as a remote ENGINE, so nothing in the backend layer knows it exists, and a VLM hosted there came back as "not served over HTTP" from an install that was serving it perfectly well. The node's own catalogue is the authority, so it is asked. An explicit backend pin still wins — discovery is the fallback, not an override — and the nodes are not probed at all when clustering is off, which would have added a timeout to the first page of every document. Also: one test here was passing for the wrong reason. `from codai.models import manager` reads the package attribute and never consults sys.modules, so a fake installed there was simply not seen. Patched as attributes of the real module, which is what made the precedence test fail honestly. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three silent drops on the way to a rented pod, all found while wiring Surya to the model layer, all of the same shape: coderai accepted a field, forwarded a reduced version of it, and the pod answered something plausible. * Images. The front flattens multipart content to the literal text "[image_url content]" unless the backend claims vision, and only the llama.cpp backend ever did. So every page sent to a VLM served on RunPod — olmOCR in "model" mode, or Surya through a coderai-vllm pod — arrived as that placeholder, and the model answered fluently about nothing. A proxy is the wrong place to judge: forwarded, a text-only server returns a plain 400 and a VLM server reads the page. * Structured output. Only {"type":"json_object"} survived, so a json_schema — the strict form current clients send, Surya's guided decoding among them — was dropped and the reply came back as free text the caller could not parse. The streamed path ignored the field entirely, json_object included. * extra_args. Whitelisted per engine on a model's runpod block since pods shipped, parsed, stored, and never put on a command line. Setting it looked like tuning and did nothing, which is worse than refusing it: the pod boots, serves, and ignores the decision. A page-reading model wants prefix caching and the Qwen-VL pixel bounds, and until now the only way in was docker_args, which replaces the whole generated line. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
OCR on the Digesta install answered 503: "vLLM isolated venv not found at /cache/vllm_venv". That host has no GPU at all — renting A40 pods is its whole job — so the venv it was told to build could never have run, while the Surya-2 checkpoint it needed was already a configured model (backend: runpod, engine: coderai-vllm) nothing in the OCR path could ask for. The model layer was never the problem; `ocr.surya_serve` was asking the wrong question. It enumerated server PROCESSES — torch in a local venv, our own vLLM, a URL typed in by hand — when the answer the operator had already written down was a MODEL. olmOCR has had that bridge since it shipped ("model" mode, through our own chat path); Surya never got one, because Surya2 drives its server directly with guided-JSON decoding and logprobs that a proxy hop would drop. So it gets the same bridge, in the only shape it can take: resolve the model through whichever backend owns it and hand the worker the address. Three values, because Surya refuses a server whose /v1/models disagrees with the checkpoint it expects, and a pod URL is useless without its token — which only Surya's vLLM client reads, so a bearer-locked endpoint is always attached to as "vllm". The address is a lease, not a location: a min_pods=0 pod is reaped when OCR goes quiet, and the page that finds it gone re-resolves once — renting a fresh one — instead of every later page failing against a dead URL. A failure against a server that still answers /health is a real error and is reported as one. Co-Authored-By:
-
- 10 Oct, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authored
When zeiss ran out of memory this morning the OOM killer picked coderai-nvidia. Not because it was at fault — a runaway test process was — but because oom_score is essentially "how much memory does this task hold", and an engine with a model loaded is always near the top of that list. The leak carried on; the thing that was serving died; the box needed a hard reset. Linux offers one lever, /proc/<pid>/oom_score_adj, and two rules that decide where it can be pulled from. Both were verified on the two production hosts rather than assumed: raising is free, lowering needs CAP_SYS_RESOURCE — and a docker container does not have it (tested: in-container `echo -500` is denied, `echo 200` succeeds). So the protected base has to come from the engine daemon, via --oom-score-adj on the run, and everything inside can only step back UP from it. rootless podman cannot protect at all: a negative value is clamped to 0. 0 is still worth having on the Digesta host, where coderai currently sits at +200 — inherited from systemd's user manager, which makes it a PREFERRED victim with a kernel score of 800 against zeiss's 670. So: the container gets -500 (CODERAI_OOM_SCORE_ADJ, 0 disables), and inside it codai/util/oom.py states the order as three tiers relative to that base — engines +100, workers +300, training +500. Relative, because the base differs by host and a tier should keep its distance whatever it turns out to be. Engines are marked in the existing _engine_preexec hook, which covers the isolated model workers too: they are spawned by the engine, not the front, so they inherit it. Training is marked from the parent after Popen — a preexec_fn would run Python between fork and exec in a threaded server, which is the documented way to deadlock. The ordering is the point. Protecting everything equally would just move the bullet to an innocent bystander; this way the kernel sheds a training run (hours, restartable) before an engine (loses its in-flight work, and the front notices and respawns it) before the front (loses everything, including the ability to respawn anything). Every helper can only ever make a process MORE killable: nudge() refuses a negative delta rather than attempting a privileged write, because a process that believes it is protected and is not is worse than one that knows it is exposed. All of it is best-effort — a process that cannot adjust its score still serves. 16 tests, including that the number reaches the kernel and moves the score, that a child can be marked from the parent, and that the engine mark sits inside a try (it runs between fork and exec, so anything that raises there fails the spawn). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
zeiss came back from this morning's hard reset with no container at all, and the hourly upgrade then skipped every tick — so the host sat on old code and, worse, stayed DOWN, with nothing in the loop that would ever have recovered it. Both idle probes are HTTP, so an absent server is indistinguishable from a wrong token: inflight() said "unknown" and the fail-closed guard refused to act. The Digesta host, whose container was up, upgraded itself fourteen seconds after the push; zeiss needed hands. container_state() answers the one question the probes cannot — is there a server at all — by name, then by any running container of the image being upgraded. Only a POSITIVE "not there" counts as idle: an engine that cannot be asked (daemon down, wrong socket) stays unknown and still fails closed, because the point is to recognise an absent server, not to assume one. Checked before the token, since a host with no server is in the same position as one with no token. Both installs run their engine's ps as the service user and name the container `coderai`, so the default matches; CONTAINER_NAME covers an install that does not. The second half: every knob reads `VAR="${VAR:-default}"`, which looks like the environment may override it, but the conf is sourced first and assigns unconditionally — so `MAX_DEFERRALS=1 coderai-autoupgrade` did nothing at all, with no line in the log to say why. I hit that landing this morning's restart and had to source the conf from a wrapper to get round it. The environment is now snapshotted before the conf and restored after, for that run only, and a test checks the snapshot list against every knob the script resolves so the next one added cannot quietly fall out. require-idle and max-deferrals are also logged now: "it skipped again" and "it was told to skip" used to look identical. The conf example claimed that without a token "the restart happens whenever the timer says so". It is the opposite — the check fails closed, so a running server is then never upgraded. Corrected, and REQUIRE_IDLE_CHECK documented next to it. tests/test_autoupgrade_recovers_a_down_server.py RUNS the script against a stubbed engine, runner, restart command and health file: the env restore is eval-based and the container check branches on an exit code, neither of which reading the source can confirm. 12 new tests, 56 across the three autoupgrade files. No version bump: nothing in the image changed, and bumping would send both production installs through an image upgrade and a restart for a host-side script. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A full `pytest tests/` run reached 46 GB RSS plus 20 GB of swap on a 54 GB machine. Swap filled, the OOM killer took coderai-nvidia — which was serving — and the box needed a hard reset, which lost the last forty minutes of kern.log with it. atop's history is what identified the process: a pytest worker, carrying the `coderai-front` process title because importing the front sets it, growing 7.8 GB → 34.8 GB → 46.6 GB across three ten-minute samples. The defect is `except asyncio.CancelledError: pass` wrapped around an await whose job is to collect a CHILD task. asyncio delivers a cancellation as a CancelledError at the next await point, so that handler catches two unrelated things: the child reporting it stopped (fine to swallow) and the caller asking us to stop (never fine). run_forever's reconnect handler sat on exactly such an await, so `task.cancel()` was absorbed and the loop carried on. With the reconnect delay mocked to zero by the test driving it, that became a busy loop allocating ~78 MB/s — AsyncMock records every call it receives — and 400,000 iterations in the five seconds I measured. being_cancelled() (Task.cancelling(), 3.11+) tells the two apart, and all four sites now re-raise ours: _stop_heartbeat_task, _cancel_inflight_tasks, the two keepalive collects in handle_message, and BrokerService.stop. The inflight ones matter in production independently of the leak — absorbing a cancel there means a shutdown waits out a whole video generation instead of stopping it, and run_forever simply reconnected forever instead of exiting. run_forever's own cancellation path clears the cancel for the duration of its cleanup and re-raises at the end, so honouring the cancel does not cost the websocket close. The test that drove it now brakes itself: the zero-delay sleep stand-in counts, and raises a BaseException (anything else is caught by the reconnect handler) well past the two reconnects it needs. With the product bug reintroduced the file fails in 1.5 s at 441 MB instead of eating the machine. tests/conftest.py adds what was missing in general: an RSS ceiling (6 GB, CODERAI_TEST_RSS_LIMIT_MB) and a per-test stall ceiling (180 s, CODERAI_TEST_STALL_SECONDS), both dumping every thread's stack to a file before exiting — pytest captures stderr per test and drops the buffer when the process does not unwind, so a watchdog that only printed would have died silently. No new dependency. "A test can consume the host" is a property of the run, not of one test. tests/test_broker_protocol.py could not finish before this — it hung at the 14th test. The whole suite now completes for the first time: 78 s, 1.57 GB peak, 1635 passed. The 25 failures are the 17 that already failed plus the 8 stale register-ack expectations in that file, which were unreachable behind the hang and are a separate question (the fixtures send `accepted: true`; the client has required `status: "ok"` for some time). Co-Authored-By:
-