- 13 Oct, 2026 9 commits
-
-
Stefy Lanza (nextime / spora ) authoredTwo things, from one sample document. RESUMING A CUT-OFF PAGE. Raising surya_max_tokens only moves the cliff: a dense enough page overflows any fixed number, and incomplete block markup parses to a fragment or to nothing. `ocr.surya_max_continuations` (default 2) resumes instead -- the text so far goes back as a prefilled assistant turn with vLLM's `continue_final_message`, so the model carries on the exact string rather than answering afresh about the same image. A server rejecting the flag (llama.cpp) gets a plain "continue" instruction; a server rejecting both leaves the page partial and honestly marked, since a page that cannot be finished is still a page. Logprobs are concatenated in step with the text, because surya_repair_accents rebuilds characters from the model's own per-token bytes and a later segment paired with earlier bytes would corrupt exactly the accented characters the repair exists to fix; if they cannot be aligned the repair is given up on rather than fed a mismatch. Temperature 0 -- the opposite of the retry path, which raises it to break a repetition. A page still unfinished after its budget carries "complete": false so a caller can refuse a partial rather than store it as read. On by default: returning a page KNOWN to be incomplete is a correctness failure, not a cost trade-off. IS THERE ANYTHING ON THE PAGE AT ALL. A four-page court PDF came back with 47 characters and was reported as an OCR failure. It was not. Every page was a ~900-byte CCITT Group 4 image -- what an EMPTY page costs, a couple of bytes per all-white row -- carrying 0.13% ink, all of it a diagonal scan edge and a diagonal "copia non ufficiale" watermark. The 47 characters WERE the watermark. The engine had read the page perfectly. "The model could not read it", "the reply was cut off" and "the page is blank" were indistinguishable in a short result, and they lead to completely different places -- a better engine, a bigger budget, or a document fixed upstream. Any page whose text is shorter than ocr.blank_page_max_chars (200) now carries the fraction of its pixels that are not background, from an image already in memory, with blank_pages listed at the top of the response. The opposite verdict is as useful as the blank one: ink above the floor says there IS something there, so the engine's failure is real and worth chasing. It applies to every engine, because the measurement is of the image rather than of the model, and it is a diagnostic only -- nothing is skipped, refused or altered because of it. Verified against the sample: all four pages 0.13-0.33% ink, correctly called blank against the 0.5% floor. Two bugs found while testing: `rounds` counted resumption ATTEMPTS rather than successes, so a failed resume would have reported the page as whole; and three assertions matched a source window that moved, one of them for a sentence that is wrapped across f-string lines and never appears contiguously. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
-
Stefy Lanza (nextime / spora ) authored
Three defects, found from a live batch that reported ~98 pages of about 71 characters, 100 with NO text at all, and 13 reading perfectly. 1. `model_attempts` could not reach anyone. `_page_from_dict` copied the worker's diagnostics onto page.meta BY NAME and knew only `unreadable_trace` and `accents_repaired` — so the field added in v0.3.45 was dropped at the subprocess pipe, and was being hunted for in a live batch that could not have contained it. The same reassembly had already dropped `accents_repaired` once, reporting null for a repair that worked. It now forwards by exclusion: every key that is not an OcrPage field becomes meta. A key-by-key copy means each new diagnostic is born broken and nobody finds out until they need it. 2. An empty page and a 71-character page are the same bug. surya-ocr-2 emits block markup — a category and a bbox per block, BEFORE the text — so a reply stopped at its token cap is incomplete markup, and incomplete markup parses to a few blocks or to none. The cap is surya's `max_tokens_default=2048`, which its vLLM backend never overrides, and the same batch averaged 2,644 generated tokens per request: above it. `ocr.surya_max_tokens` raises it (0 = surya's own), opt-in because a higher cap costs tokens on every page. What is not optional is telling the cases apart, so a page that hit the cap now carries tokens, cap and `truncated: true` with an explanation that it is a cut-off reply and NOT a page the model could not read. Counts are per THREAD, like the trace, so one lane never reports another lane's verdict. 3. The fleet could grow by at most one pod per BOOT. `_should_grow` opened on a boolean held for the whole provision, and a provision is ~12 minutes — so a burst needing four pods took forty minutes to be served by four, whatever max_pods said (measured: 64 pages at 16 per pod got two pods, ready ten minutes apart; reported as "no real scale-out"). The grower also could not see the SIZE of the demand: `min(inflight) >= threshold` answers "is the fleet full?", so the only request it could formulate was "one more", and the callers blocked in acquire() — the actual measure of unmet demand — were invisible. Both are counts now: `_booting` (booting pods ARE capacity, so one demand is never provisioned for twice) and `_waiting`. Growth runs while pods + booting < demand, bounded by max_pods and by the new `scale_up_max_parallel_boots` (default 4) — a real limit, because the pods the demand justifies are the GPUs that start billing together. The original single-flight anticipatory trigger is KEPT beside it: max_inflight_per_pod defaults to unlimited, so requests are often never blocked and a merely saturated fleet would otherwise never grow at all. A prediction gets one pod; measured demand may get several. Found while testing (3): acquire() ignored its own timeout when a sibling process was booting a pod for the same pool — it waited a flat 15s and looped back into the same decision without re-checking the deadline, for as long as the sibling's registry entry survived (up to 15 minutes). A test hung on it. 42 tests across tests/test_runpod_scale_out.py (new) and tests/test_surya_retry_budget.py. Two of my own test harnesses were wrong and are fixed, not worked around: a sys.modules ledger stub that `from codai.api import runpod_ledger` ignores once anything has imported the real module, and a static count of increments against decrements, which cannot express "one increment, four mutually exclusive exits" — replaced by runtime balance tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
baidu/Unlimited-OCR (MIT, 3B, BF16, 8GB) is an in-process OpenAI-endpoint VLM that emits layout-tagged blocks with bboxes and categories, and parses long documents in one 32k pass. Four phases: the engine and the <|det|> parser (offline-testable, no GPU); serve modes so a local GPU, an endpoint the operator already runs, a managed model, a rented pod and no-GPU-at-all are each first-class; a serve profile following vllm-ocr.txt, since the architecture is in no released vLLM wheel; and document-level OCR designed as a capability of the SUBSYSTEM, because coderai's OCR contract is per-page end to end and this model's one-shot mode is the reason to lift that. Records what upstream requires and what each requirement costs us, including the three decode preconditions that yield EMPTY OUTPUT rather than an error, and the logits processor we cannot set on someone else's endpoint. Language coverage is unenumerated upstream, so it is treated as unknown and measured, never designed in or out. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Comparing 64 pages against max_pods x inflight (64 x 16 = 1024 on the Digesta orchestrator) would never have fired, and would have been reassuring if it had: the fleet's ceiling is not what serves a burst. A pod takes ~12 minutes to become ready, so the burst is served by the pods already warm. What is worth saying at setup time is the arithmetic itself — these pages need that many pods — plus the case where it exceeds max_pods and the surplus can only queue. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Read from the pod's own drained log, ten minutes of batch OCR on 2026-10-13: Running: 15-19 reqs, Waiting: 0 reqs, GPU KV cache usage: 5-7% Avg generation throughput: ~1,900-2,100 tokens/s, every minute 524 x "POST /v1/chat/completions" 200 OK, no aborts 2,644 generated tokens PER REQUEST (second pod: 4,010) MM cache hit rate: 25-27% Three things in those numbers, and only the third is a defect. The GPU was never the bottleneck: 5-7% of the KV cache and `Waiting: 0` for ten solid minutes. So a card to itself cannot be the fix, and the co-loaded pyannote cost nothing measurable — the limit was never VRAM or batch slots. Oversubscription converts throughput into latency, not into stalling. 64 pages into max_pods x scale_up_inflight_per_pod = 32 slots queues correctly, but every page's latency rises by the queue, and a client counting finished DOCUMENTS sees nothing until a document's LAST page lands: spread 64 pages over sixteen documents and all sixteen advance while none completes. The pool now says so at build time instead of leaving it to be inferred. The defect: 2,644 generated tokens for a page whose text is ~400. surya re-runs a page whose generation repeated itself (`_should_retry`) up to three more times, each at a rising temperature and each able to reach its token cap — four full generations to arrive at the same unreadable answer — and its repeat detector cannot tell "the model broke" from "this page really does repeat", which a tax table does. The 25-27% multimodal cache hit rate is the direct evidence: a quarter of all requests re-sent an image the pod had already read. None of it was visible: a page generated four times over and a page that was merely slow look identical from outside. `max_retries` is unreachable from surya's settings — the vLLM backend calls chat_completions_batch without it, so the default 3 was the only value there has ever been. So the decision is hooked, beside the token-trace hook already in that module, and `ocr.surya_max_retries` is a CEILING (asking for 10 where surya allows 3 gets 3): 1 by default, 0 to disable, -1 to hand it back. Every retry is logged, and a page that needed more than one call carries `model_attempts` in its result — always, not only under surya_token_trace. A page read first time stays byte-identical to what it always returned. The counter is per THREAD, like the trace, or one lane would be blamed for another lane's retries. Also pins SURYA_INFERENCE_PARALLEL=1 in the served modes: a lane is our unit of concurrency, and left unset surya sizes its own thread pool from a built-in GPU table and fans out inside the worker, so instances x lanes would stop being the real number of pages in flight. 19 tests in tests/test_surya_retry_budget.py; settings carry the new knob. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`keep_warm` stopped re-renting whenever a pod went away behind the pool's back — terminated from the console, reclaimed as spot, lost with its host — and the scaler said nothing about it. Both halves had one cause: nothing in a pool ever re-checks a pod's health. A handle's `healthy` flag goes false only when something tried to USE the pod and failed, or when the pool itself retired it, and a warm-floor pod is precisely the pod with no traffic to fail. So `maintain()` counted the handle, found `healthy_n >= keep_min`, and neither topped up nor had anything to log. The floor stayed "satisfied" by a machine that had not existed for hours, and the first real request paid the full ~12-minute cold boot that keep_warm is bought to avoid. (The class docstring claimed the scaler "re-checks health" for months. It never has. Corrected rather than implemented: a timer-probe would have to decide what an unanswered probe means, and that decision is what killed seven A40s in v0.3.41.) `reconcile_pools` instead reads the account listing the stale-pod reaper already fetches every ~30s — authoritative, free, and unable to mistake a busy pod for a dead one. Only positive evidence acts: * a dead status (TERMINATED/EXITED/DEAD/FAILED) drops the pod at once; * a mere absence drops it only after two listings in a row, because the API is eventually consistent and one page omitting a live pod used to be enough to unregister it — a reappearance resets the count; * a pod that served a request in the last 60s is kept whatever the listing says, which cannot mask this bug since a warm-floor pod has no traffic; * a provisioning pod is shielded, and a FAILED listing never gets here at all, so an API outage cannot empty every pool and re-rent the fleet. `PodPool.forget()` is `discard()` minus the kill: nothing is left to terminate, but the bill is closed in the ledger exactly as a real termination closes it, since the pod was billed up to its death. A borrowed `(shared)` pod is dropped without a ledger entry — its owner holds the bill. The registry is pruned through the same two-sweep rule it used to skip. The top-up now announces itself before spending, because a warm floor silently failing is indistinguishable from a warm floor that is fine. 21 new tests in tests/test_runpod_warm_floor_rerent.py pin the caution, not just the fix: the failure mode in the other direction is forgetting a pod that is alive and billing. test_runpod_client_scale's source assertion moved to the new seam. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The worker protocol was one request, one reply, strictly in order, so pages in flight WAS the process count — and a process costs ~600 MB of imported torch. That is the whole reason `surya_instances: 64` meant 64 processes, 37 GB and a nine-minute stall with the rented GPU idle at 2% KV-cache usage. In the served modes that memory buys nothing: the recognition VLM is on the pod, the local predictor holds no weights (1.7 MB of tokenizer and config on disk — surya-2 returns its own boxes, so the detection model is never even fetched), and the process rasterises a page and then waits on a socket. So a lane is a worker OBJECT inside the process that already exists, with its own predictor and its own page, sharing the one torch import. Effective concurrency is instances x lanes; `surya_lanes` defaults to 4, served modes only, built on demand. Three things make one process safe to drive concurrently: - every reply carries the id of the request it answers. Out-of-order replies are the point — a short page must not queue behind a dense one — and attributing one page's text to another request would be worse than losing it, so an unknown id is dropped rather than guessed at; - the token trace and the accent repairs are per THREAD. surya's hook is module-level, one _generate_one per process, and its sink was a list belonging to whichever worker object installed it last. Shared between lanes it would have filed one page's logprobs, and surya_repair_accents' rewrites, under a different page; - a worker that dies fails its pages instead of leaving them parked on a reply that is never coming, and killing the engine wakes everything waiting. A single-lane engine sends no ids and reads replies in order exactly as before, which is how paddle, docTR and surya-`local` are still driven. `lanes()` is asked for politely (_engine_lanes), because an engine is anything with recognize_image and a pool must not refuse to build over a capability an engine simply does not have. Tested against the REAL worker protocol with a stubbed engine: four 0.25s pages finish in under 0.75s, on four distinct lane objects, and six pages over two lanes never exceed two. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A batch of documents gave one flawless 19-second burst — 22 documents, correct page counts, clean accents — and then no completions at all for nine minutes, while the rented A40 sat at 2% KV-cache usage with `Waiting: 0`. The GPU was never the bottleneck. Two local mistakes compounded: `surya_instances: 64` reads like GPU concurrency and is not: surya, paddle and docTR spawn ONE OS PROCESS per instance, in their own venv. 64 of them meant 37 GB of RSS, swap exhausted and load 25 on a 16-core host that also runs some thirty other containers. Nine hours after the batch ended, 66 of those workers were still resident with 19 seconds of CPU time each. Then it fed itself. The workers' /health probe (5s, against coderai's OWN gateway) began timing out under the load the workers were creating, and a timed-out probe counted as "the endpoint is gone" — so each worker was killed and respawned, which added load, which timed out more probes. - the process-per-instance engines are capped at one worker per core and never more than half of free RAM, with a log line naming the figure and the reason. CODERAI_OCR_INSTANCE_CEILING overrides it for a measured host. The in-process engines (chandra, olmOCR) are untouched: an instance there is a concurrency slot, not a process; - a health probe now means "gone" only on a definite 404 "unknown OCR engine lease", which is the case re-resolving was written for: a front that restarted. A timeout means busy and the worker stays up, because respawning cannot fix a front that is down — the replacement asks the same front for a lease. A 503 means the pool has nothing rented this second, which the lease does not care about. GPU-side concurrency remains runpod.scale_up_inflight_per_pod and max_pods; the local worker pool is a hard ceiling on how many pods surya can keep fed, which is the standing argument for running it in-process. Co-Authored-By:
-
- 12 Oct, 2026 23 commits
-
-
Stefy Lanza (nextime / spora ) authored
Five A40s were terminated between 19:14 and 20:11 today. None of them was broken: each pod's own log ended in `"POST /v1/chat/completions HTTP/1.1" 200 OK` at 4664 tokens/s, seconds before it was reaped, and every replacement cost a ~6-minute image pull before it could read a page. `httpx.RemoteProtocolError("Server disconnected without sending a response")` was in the dead-upstream set. It does not mean the server is gone — it is what httpx raises when a POOLED connection was closed by the far end between two requests. uvicorn closes an idle connection after 5s and nothing here raised that, while this gateway held pooled connections for 60s. So the first page after any quiet gap hit a socket the pod had already closed, the gateway read it as an empty address, and the pool terminated a rented GPU for it. - a closed pooled connection is retried on a fresh connection, against the SAME upstream: re-choosing would send the page to a different pod, or rent one; - a pod is asked before it is dropped — /healthz, then /health, on a new connection. Any answer under 500 counts as alive, a 401 included: that is a server, authenticating. Only silence drops it, because dropping is irreversible; - idle pooled connections are released after 4s (CODERAI_OCR_KEEPALIVE_EXPIRY_S), under the server's own keep-alive, so the race is rare rather than routine. Reuse within a burst is unaffected. A pod RunPod really has reclaimed mid-document is still dropped at once, which is what the original logic was for: left in the pool, every following page picks the corpse and the in-flight count climbs with nothing draining. The tests pin both halves, and a read timeout still neither probes nor drops. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`teach_model` called `seed_model_env(entry, "", value)` — no served name — so the pod registered the entry under its `path`. For an ordinary model that is merely untidy. For an ADAPTER it is fatal: a LoRA's path IS its base, so diogene-alpha registered as cyankiwi/Qwen3.8-27B-AWQ-INT4 and collapsed into the base. Measured on the live pod before the fix: taught https://…:22143 about 'diogene-alpha' (cyankiwi/Qwen3.8-27B-AWQ-INT4) catalogue: […, 'qwen38-awq'] diogene-alpha listed: False HTTP 404: Model 'diogene-alpha' is not available. Taught successfully, still 404 — and the version of this where the base answers under the adapter's name would have been silent. With the name carried, the same pod registered it and served it. Verified by repetition rather than by eye: each name is byte-identical across two runs at temperature 0 and the two differ from each other, so they are backed by different weights. diogene-alpha sha 77f7db98ab19 ×2 (reasons, then answers) qwen38-awq sha 4767ec4fcf00 ×2 (answers directly) One pod, one copy of the weights, the adapter selected per request by name. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Seeds were de-duplicated by `path`. A base model and a LoRA of it share one path by definition, so the adapter's entry was dropped and the pod was told about the base alone — observed live, one line after the sibling seeding had correctly found it: [runpod] pool 'pool:digesta-qwen38' is shared with: diogene-alpha [runpod] pod will be told about 1 model(s) The dedup exists because a capability pod's list can overlap the one model a per-model pool also seeds, which is a real case; it just has to key on what the pod will SERVE the entry as — path plus alias, id and adapters — rather than on the weights the entries happen to have in common. Orchestrator-side only: the pod registers what it is handed, so no new image. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`/v1/chat/completions` was capped at 60 requests per 60 s per client IP as a module constant — no config, no env, no way to move it. On a RENTED POD that cap is the orchestrator's own: a pod has exactly one client, the install paying for it by the hour, so every request shares one bucket. An olmOCR pod measured 3.45 pages/s (207/min) and was being refused above 60. It also produced three wrong measurements that were reported as hardware facts. Concurrency ladders ran every level inside one window, so refusals began exactly where the cumulative count crossed 60: surya 16+24=40, then 20 of 32 pass, 12 refused (13 measured) surya +48 = 72 over, all 48 refused (all 48 measured) chandra 8+16+24=48, then 12 pass, 20 refused (21 measured) olmOCR same ladder, same arithmetic (21 measured, twice) "Both peak at 16 and both fail outright by 32 — a cliff" was the rate limiter. Paced across windows, olmOCR runs N=32 with ZERO errors at its best throughput; it is flat at ~3.4 pages/s from N=8 to N=32 while p50 goes 2.3s -> 7.9s, which is saturation at 8, not a cliff at 32. So: the limits take env overrides (CODERAI_SERVER_RATE_LIMITS="prefix=count/window", CODERAI_SERVER_RATE_LIMIT_ENABLED=0), and a pod is launched with the limiter OFF — its admission is pod_max_parallel_requests + pod_queue_max_size, which describe the GPU rather than guessing at an abusive client that cannot exist there. docs/ocr.md keeps the old table with the refusals marked as refusals, because the numbers themselves are still the only ones measured and the reading of them was the error. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The autoupgrade is version-gated, so code pushed at a version Aruba has already taken would never be pulled. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two more links in the same chain, both of which fail SILENTLY — the pod serves the base weights under the adapter's name and nothing errors. * `seed_model_env` computed the content id behind `os.path.isfile(local)`, which is False for a directory, so the adapter never reached the pod's config. The per-request push sends the BYTES but its rewritten entry is discarded by the caller, so the seed is the only thing that tells a pod an adapter exists. * a named shared pool seeded only the entry of whichever model created it; `seed_entries` was populated for CAPABILITY pools alone. Sharing a pool exists so one pod answers for several names on one copy of the weights — a base and its adapters — which cannot work when only one name is known. Siblings now travel with `path` resolved to what the pod can FETCH, because these entries are described by `runpod.hf_repo` and carry no path of their own, and a seed without one is dropped. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A QLoRA trained for a client sat on disk for ten days while a second service rented its own A40 every morning, pushed the adapter over SSH and proxied to it. The path that should have carried it was written but had never been run end to end, and five things were in the way. * `ensure_text_loras` DROPPED a PEFT directory — "publish it to HuggingFace" — which is the shape every trainer writes. It now travels as one deterministic tar (sorted names, fixed uid/gid/mtime, so the content id is stable) and is unpacked by the blob store, because vLLM loads an adapter from a directory: adapter_config.json carries the rank and target modules, and a bare .safetensors is not loadable. * the adapter was re-read and re-hashed ON EVERY REQUEST. A Target is rebuilt per request, so `lora_entry` was set again each time; the bytes were never re-sent (the store is content-addressed) but 159 MB was read and hashed, and the pod asked, only to learn it already had it. Cached per (pod, path, size+mtime), so a retrained adapter at the same path is still sent again. * `_lora_args` passed a CONTENT ID straight through, producing `--lora-modules name=sha256:…`, which vLLM cannot open. On a pod that is the only form the adapter has. * an engine launched for a base model carried none of its adapters. vLLM registers them at launch and one vLLM serves one base, so the pod a shared pool rented for the base alone could never serve an adapter of it — the whole point of sharing the pool. Every model naming the same base now contributes its adapter under its own model id, which is the name requests ask for, so two versions can be A/B'd on one GPU with no reload. * `--max-loras` was never set. At its default of 1, requests naming different adapters cannot share a batch and serialise instead: the cost lands on latency, not on an error. Extracting a bundle is a write from network bytes, so member names are checked for escapes before anything is unpacked, and tar's data filter is used as well. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Triton and torch.inductor JIT-compile vLLM's kernels at run time and need a full build toolchain for the stubs they emit. Each piece was found on a live olmOCR pod, one layer at a time, because from outside the failure is identical every time: vLLM dies before registering and the pod answers 502. gcc the C compiler g++ the launcher Triton builds is C++, so gcc alone is not enough ninja-build inductor shells out to ninja for its extensions Only olmOCR hits this, and the reason is worth recording: it is FP8, so vLLM's quantisation kernels go down the Triton path. surya-2 is dense bf16 and served fine on images that had none of these, which is why "the image lacks a compiler" looked for a while like it could not be the problem. Applied to vllm.apt AND vllm-ocr.apt, which is copied from it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
It shipped at 4, which made it the binding limit for every install. The limiter is meant to be each engine's own `*_instances` -- the pool hands out one instance per page and an engine is not reentrant -- while a RunPod fleet can hold max_pods x scale_up_inflight_per_pod pages in flight: 64 x 16 = 1024 on the Digesta install. A ceiling of 4 capped that at a quarter of ONE pod. Measured on one A40, same page, same install: N surya req/s p50 chandra pg/s p50 1 0.93 1.1s 0.21 4.6s 8 3.79 2.0s 1.31 6.1s 16 7.11 2.2s 1.88 8.4s <- both peak 24 5.89 4.1s 2.09 10.6s 32 5.44 +13 err 3.5s 1.44 +21 err 7.5s 48 ALL FAILED Both peak at 16 and both fail outright by 32 -- a cliff, not a degradation. Default raised 4 -> 64: four pods' worth at the measured knee, far above what a single-box install can feed, so it stops being the limiter while staying finite. NOT higher, because every admitted page holds a rasterised bitmap (~18 MB for A4 at 250 DPI): 64 in flight is ~1.2 GB, 1024 would be ~18 GB of page images before any engine does work. Trading a throughput limit for an OOM is not an upgrade. docs/ocr.md carries the table and spells out the three settings that bound this, which are easy to confuse: the per-model scale-out threshold, the per-engine instance count (the real limiter, and for the SUBPROCESS engines it is what the box can host -- a surya worker is a separate interpreter at ~750 MB), and this ceiling. Also recorded there: surya's req/s is BLOCKS, not pages, since it calls once per layout block; and its worker protocol is synchronous, so a worker handles one page at a time and concurrency is the process count. In served mode that worker only scales an image and makes an HTTP call, so making it concurrent -- or skipping the subprocess entirely, as chandra does -- is what would let surya use a large fleet. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
SURYA READS ITALIAN AGAIN. Measured on the live install, the same synthetic page that scored 29 markers all morning: MARKERS : 0 (29 baseline) ACCENTS : AEEIOUaeeiou (all twelve) La societa` ha facolta` di piu` attivita`: percio` e` cosi` che l'unita` perche' gia` oltre. ...but it reported `accents_repaired: null` while visibly having fixed the page, because _page_from_dict carried `unreadable_trace` across the worker boundary and never carried `accents_repaired`. The fix worked; only its receipt was dropped. That matters more than a missing field: the text is no longer byte-for-byte what the server sent, and a caller comparing the two has no way to know why. A silent fix is nearly as bad as a silent corruption. tests: +2 (47 in test_ocr_accent_repair.py) for the receipt crossing the boundary and for staying absent when nothing was repaired. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
ROOT CAUSE, measured in surya-ocr-2's own tokenizer: id 2160 decode -> U+FFFD vocabulary piece -> 'a-grave' id 2168 decode -> U+FFFD vocabulary piece -> 'e-grave' id 2169 decode -> U+FFFD vocabulary piece -> 'e-acute' id 2172 decode -> U+FFFD vocabulary piece -> 'i-grave' id 2178 decode -> U+FFFD vocabulary piece -> 'o-grave' id 2185 decode -> U+FFFD vocabulary piece -> 'u-grave' The model emits the RIGHT token -- 2160 really is a-grave -- and the decode table cannot render it. Every decoder in the chain inherits that: surya's own, and vLLM's, which reports `decoded_token: None` and therefore `bytes: []` and writes U+FFFD into the text. So the accents were never lost in transport. Eliminated by measurement, each in turn: DPI (a crisp synthetic page lost them identically), the document (chandra reads the same pixels perfectly), the vLLM version (0.20.1 cannot load the model at all), --trust-remote-code (applied, `trust_remote_code=True` in the engine config, no change), and the byte channel (vLLM reports `bytes: []` precisely because the decoder failed). THE FIX: when the decoder answers with the replacement character for an id, use the VOCABULARY's piece for that id. The vocabulary is the authority on what a token means. Decoding is per id, because a bulk decode cannot be corrected after the fact -- once flattened to text, the id that failed is unidentifiable. Verified end to end against the live pod: decoder alone : 'nt ? pi? facot? perci? cos?' with vocab : 'nt a-grave piu-grave facota-grave percie-acute cosi-grave' This needs the ids, which vLLM only reports when its server is launched with --return-tokens-as-token-ids (the per-REQUEST field is ignored on the chat path: chat_completion/serving.py:1179 falls back to the server setting only when the request field is None, and the id form is reached through _get_decoded_token). That flag is now in Aruba's global vllm.extra_args, which is what a pod receives -- the per-model `vllm` block in models.json does NOT reach a pod, which cost two test cycles to discover. An id the vocabulary also cannot name keeps its marker, and the ASCII-skeleton guard still decides whether the repair as a whole is safe to apply. tests: +3 (45 in test_ocr_accent_repair.py) pinning the vocabulary fallback, the unknown-id case, and that decoding stays per id. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`return_token_ids` added to the vllm entry of _ENGINE_FORWARDED_FIELDS, so a rented pod's vLLM backend can be asked for raw vocabulary ids -- without which the v0.3.31 repair can never fire on a RunPod-served model, which is every OCR model on the Digesta install. Inert by default: engine_config_for_pod skips falsey values, so adding the field cannot change an install that has not set it. Set on Aruba for `datalab-to/surya-ocr-2` ONLY, through its models.json `vllm` block rather than globally. The flag changes what the `token` string inside logprobs looks like and that install also serves qwen38-awq and diogene-alpha on vLLM pods; nothing in coderai reads that field except the OCR repair, but a client might, and there is no reason to find out the hard way. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The byte channel is a dead end on 0.27.1 -- measured: `content` comes back with one U+FFFD per accented letter AND the per-token `bytes` come back EMPTY (257 of them on one page, no byte >= 0x80 anywhere), so neither output carries the character. But the token IDS are not lossy, and surya's OWN tokenizer decodes them correctly: decode([2131, 2104]) == 'e-grave', measured in its venv. So the repair now asks for ids and decodes them itself, bypassing vLLM's detokenizer entirely. Both halves, because each alone does nothing: 1. THE WORKER reads vLLM's `token_id:{id}` form, decodes with surya's tokenizer (lazily loaded and cached -- the worker must start while the venv is building, and a clean page must not pay for a tokenizer), and keeps the byte route as the fallback for any stack that reports bytes honestly. Ids are tried FIRST because on 0.27.1 the bytes lead nowhere. All-or-nothing on the id list: a partial one would decode to a partial page. The ASCII-skeleton guard still applies, so a decode of the wrong call cannot be substituted. 2. THE PROXY stops dropping the request. `return_tokens_as_token_ids` is a per-request field on vLLM's ChatCompletionRequest, but sending it to a pod changed nothing -- because codai/backends/vllm.py::_chat_payload builds an explicit whitelist and silently discarded it. That is exactly the failure its own neighbouring comment warns about ("a proxy that edits the request decides for it"), five lines further up. Now forwarded, gated on `vllm.return_token_ids`, and only alongside logprobs since it describes how logprob tokens are rendered. `vllm.return_token_ids` is off by default (it changes what the `token` field looks like, and only this one consumer reads it) and is in VLLM_FIELDS, so the single checkpoint with a byte-level vocabulary can ask for ids without changing what every other vLLM model reports. A config field rather than relaunching the server with --return-tokens-as-token-ids: same effect, no pod rebuild. This is a fix ON 0.27.1. It makes the coderai-vllm-ocr image (vLLM 0.20.1) a CONTROL that confirms the diagnosis rather than the only way out. tests: +11 in test_ocr_accent_repair.py (36 -> 47) covering the id form, the all-or-nothing rule, ids-before-bytes, the fallback when a decode fails, the skeleton guard on the id route, and the proxy forwarding with its logprobs gating. 912 passed across the ocr/front/config/admin/runpod/vllm/backend selection; the 9 failures are pre-existing (broker x6, studio, whisper x2) plus test_cluster's RPC check, which depends on llama_cpp being installed in this venv and touches nothing changed here. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
CONFIDENCE. chandra hard-coded `conf=0.0` and olmOCR reported nothing at all. chandra's code even argued for it: "the model reports none, and inventing one would be read as a measurement". The first half was right and the conclusion was wrong -- the number was obtainable all along, by the route surya already uses. Both are served over an OpenAI chat endpoint, so the request now asks for logprobs and the reply's mean exp(logprob) is the confidence. It is the same computation surya's own client does, so the three VLM figures are comparable with each other instead of being three different notions of "confidence". engine scope source paddle line recogniser score doctr line recogniser score surya block mean token probability chandra page mean token probability olmocr page mean token probability The SCOPE is reported too (`/v1/ocr/engines` and `meta.confidence_scope`) and it is not cosmetic: a per-line score can single out the line to re-read, a page-level one can only say the page went badly. surya calls per layout block so its figure is genuinely per-block; chandra and olmOCR generate a whole page in one call, so chandra puts the one page figure on every line and region (which is what `unreadable`'s per-spot conf reads) and says the scope is "page" rather than letting it pass for per-line precision. A server may decline logprobs, and then conf is 0.0 -- which must NOT read as "the model was hopeless". `meta.confidence_measured` separates the two. That is the same distinction the unreadable-glyph trace needed and twice got wrong in this subsystem: a zero meaning "not measured" and a zero meaning "measured as bad" are different facts, and conflating them produces confident nonsense. coderai's in-process chat path surfaces no logprobs, so it reports not-measured rather than a 0.0 posing as one. g++. Digesta got an olmOCR pod past the TLS problem and straight into a 502: Triton JIT-compiles vLLM's kernels at run time and the launcher it builds is C++, so it invokes g++ -- and `vllm.apt` shipped only gcc. With only gcc the image looks right, vLLM starts, and then the engine core dies before it registers. Added to `vllm.apt` AND to `vllm-ocr.apt`, which was copied from it and had inherited the same gap before ever being used. BOTH vLLM pod images therefore need a rebuild to carry it: coderai-vllm (which olmOCR uses, so this is what unblocks Digesta) and coderai-vllm-ocr (pushed an hour ago and affected identically). One at a time, as ever. tests: test_ocr_confidence.py, 20 tests -- the arithmetic (exp then mean, not the geometric mean), None-not-zero for an unreporting server, the page-vs-block scope, the rotated-retry carrying the retry's figure, and a registry check so a new engine cannot arrive with no answer to "what does conf mean here". Ten existing fakes updated for the new (text, confidence) contract. 858 passed across the ocr/front/config/admin/runpod selection with the same 8 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
CORRECTION. v0.3.26-0.3.28 reported `blame: "the serving path"` with `conclusive: true` and "the byte stream decoded cleanly", on data that was not there. Measured through the live gateway today: result.raw (what surya reads) : 3 markers 'nt ? propriet? della propriet? d' reassembled from logprobs : 0 markers 'nt propriet della propriet ...' entries with bytes=[] : 257 any byte >= 0x80 in stream : False vLLM 0.27.1 reports `bytes: []` for the byte-level tokens of surya-ocr-2's custom vocabulary, and not one byte >= 0x80 appears anywhere in the stream. The accent bytes are not corrupted in the logprobs, they are ABSENT from them. _reassemble accepted an empty byte list as valid data, so it built a string with the accents silently missing, found no invalid UTF-8 in it, and called the result clean. That is reading absence as cleanliness -- the exact failure the envelope in v0.3.26 was built to prevent for MISSING logprobs, walked into one level further down. Fixed: an empty byte list counts as missing. The trace now counts `tokens_without_bytes` and, when any are present, reports `conclusive: false` with a `why` that says the loss is inside the serving stack (it produced neither usable bytes nor usable text) while stating plainly that this CANNOT say whether the model's own output was sound. `unreadable.blame` is then "unknown" rather than an accusation the data does not support. Consequence for surya_repair_accents: it CANNOT work on vLLM 0.27.1 -- there is nothing to rebuild from, and that is not a tuning problem. It is kept because its guards refuse correctly: the ASCII-skeleton check catches the mismatch, so a page comes back with its markers rather than with the accents quietly deleted, and the repair will work on any stack that does report the bytes. So on vLLM 0.27.1 the real options are `chandra` (reads the same pixels perfectly) or the `coderai-vllm-ocr` image pinned to the vLLM surya validates against. tests: +5 in test_ocr_token_trace.py (42 total) pinning that empty bytes are refused by both the reassembly and the repair, that the blame goes to "unknown" rather than the serving path, and that a FULLY reported clean stream still produces the genuine verdict. 643 passed across the ocr/front/config/admin selection with the same 8 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
MEASURED VERDICT. The byte trace on a page of crisp synthetic Italian, served through the Aruba install on vLLM 0.27.1: unreadable: count=29 blame='the serving path' page 0: calls=1 with_logprobs=1 conclusive=True kinds=None why: the byte stream decoded cleanly, so the characters were lost AFTER the model: the fault is in the serving path, not the model surya-ocr-2's own byte stream was valid UTF-8 and every line came back at conf=0.9605. The model read the page correctly; something after it destroyed 29 accented characters. What that does NOT implicate: surya-ocr-2 and chandra-ocr-2 are the SAME architecture (Qwen3_5ForConditionalGeneration) on the same vLLM, and chandra's accents survive. So this is not vLLM mishandling qwen3_5 -- it is vLLM mishandling SURYA'S CUSTOM 65,425-entry BYTE-LEVEL vocabulary, in which one accented letter is two tokens and either byte alone decodes to a marker. It was also a single model call, which kills the concurrent-batching hypothesis I had been carrying. OPTION 1, shipped and on by default: ocr.surya_repair_accents rebuilds the reply from the per-token bytes in the same response. Applied in the backend wrapper BEFORE surya parses the HTML, so blocks, boxes, text and confidences are all built from correct characters; repairing further down would mean fixing the same text in several places and missing one. Rewriting a model's output is serious, so it refuses unless all three hold: strict UTF-8 decode, no marker in the result, and an ASCII skeleton identical to the original's. The third is what makes it a repair and not a substitution -- it can only ever change non-ASCII characters, so a truncation, a different block or a shifted offset all leave the original alone, markers and all. Clean text is untouched, not even reassembled, and a repaired page reports `accents_repaired: {blocks, recovered, how}` because the text is no longer byte-for-byte what the server sent. Default ON because the alternative is shipping text measured to be corrupt. Set false to see exactly what the server sent. OPTION 2, prepared: a `coderai-vllm-ocr` pod image pinned to vLLM 0.20.1 -- the version surya-ocr itself spawns (surya/settings.py:96) -- so the model can be served on the stack its authors validate, the repair can be checked against a reference rather than trusted, and the upstream bug report has a known-good comparison. Selected per model with `runpod.engine: "coderai-vllm-ocr"`, built with `build_capability_image.sh vllm-ocr`. The pin set was verified to exist before the profile was written, because a profile that cannot build is worse than none: wheels.vllm.ai/0.20.1/cu129 is live, and torch 2.11.0 / torchvision 0.26.0 / torchaudio 2.11.0 (vLLM 0.20.1's own metadata pins) all have +cu129 cp313 wheels. Qwen3_5ForConditionalGeneration is in 0.20.1's registry, so it can serve the model. No torchcodec pin: 0.20.1 does not require one, and pinning a package nothing needs is its own hazard -- the file says what to add if a build ever surprises us. NOT a drop-in for `coderai-vllm`: everything else on it would be pinned back seven minor releases. tests: test_ocr_accent_repair.py, 29 tests -- the repair itself, every guard, and the image profile. 833 passed across the ocr/runpod/config/admin/front selection with the same 8 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
coderai_engine_inflight / coderai_requests_active are load-bearing: the hourly autoupgrade's idle guard reads them to decide whether a restart is safe, and with MAX_DEFERRALS=0 a non-zero count makes it SKIP for ever. So one request counted in and never out silently stops a host taking upgrades, and the only symptom is a version that stops moving. Observed on the Digesta orchestrator today: the gauge read 13 against two real requests, the 09:01 tick logged "SKIP: 7 piece(s) of work in progress" and left the image alone, and only a manual restart cleared it. The structural leak is in the streaming proxy path. Starlette awaits a response's background task only AFTER the body is sent -- `await self.stream_response(send)` then `await self.background()` -- so when the client disconnects mid-stream the send raises, the exception propagates, and `BackgroundTask(_release)` never runs. The relay generator's `finally` DOES run in that case (Starlette closes the iterator) but not when the body is returned without streaming, so both now call `_release`, which is idempotent via a flag. `exit_request` is idempotent per rid as well, and that is required rather than decorative: two release paths firing for one request would otherwise decrement an UNRELATED in-flight request, and the count would under-report real work -- worse than over-reporting, because then a restart lands during a request. Every request is now timestamped, not just the inference ones that carry `meta`. Without that a leaked non-inference request was invisible as well as unreleasable: the count said 13 and the Tasks page showed nothing, which is how this went unnoticed. And a reaper underneath all of it (`server.inflight_max_age_s`, default 3600, 0 disables), called from the two places that ASK whether anything is in flight: * the /metrics scrape, because that is the exact moment a stale count does its damage -- the scrape that would report a lie is the right place to correct it, not to hope somebody notices; * the supervisor's drain wait, which would otherwise sit out the whole grace period for a request that finished an hour ago. An hour, not a minute: a cold RunPod pod can make a single OCR page take 14 minutes, and reaping a live request would be a worse bug than the one being fixed. Training is unaffected -- it is tracked as a JOB and the request that starts it returns at once. NOT fixed here, noted for later: `_proxy_passthrough` has the same BackgroundTask-only release for the system worker. It never calls enter_request, so it cannot leak the gauge -- it leaks an httpx stream, which is a smaller and separate problem. tests/test_front_inflight_leak.py: 14 tests, including double-release, late release after a reap, thread safety, and the source invariants for the two release paths. 299 passed across the front/metrics/registry/config/admin selection; the 9 failures are pre-existing (broker_config x5, broker_streaming, studio_composed_surfaces, whisper_server x2) and touch no file changed here. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
v0.3.25 shipped the trace as a bare list of findings, and defined the empty list as "the bytes were sound, so the loss is in the serving path". That is wrong whenever the upstream returned no logprobs: there was nothing to analyse, and the emptiness would have read as a confident conclusion drawn from missing data -- exactly the class of mistake the trace exists to stop. So the trace is now a self-describing envelope: {"calls": 7, "with_logprobs": 7, "conclusive": true, "kinds": {"invalid-utf8": 29}, "why": "...", "findings": [...]} and `unreadable.blame` is one of "the model", "the serving path" or "unknown". `conclusive` is false when no calls were recorded (the hook never fired) or when none carried logprobs, and then blame is "unknown" rather than defaulting to either culprit. surya asks for logprobs by default (SURYA_INFERENCE_LOGPROBS = True) and the OCR gateway forwards the body verbatim precisely so they survive, so the normal case is conclusive. But an upstream that declines them is a configuration, not a bug, and it has to be visible instead of silently inverted into an answer. Also: a trace analysis that throws now reports {"conclusive": false, "why": "trace analysis failed: ..."} rather than no key at all, so "the analysis broke" and "the bytes were fine" cannot look the same either. tests/test_ocr_token_trace.py: 37 (was 30), including the inconclusive paths. 825 passed across the OCR/config/admin/runpod selection, same 10 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`ocr.surya_token_trace` (off by default) reports the per-token bytes at every place a surya page lost a character. It exists to answer one question that the finished text cannot: whose fault is it? invalid-utf8 the reassembled stream does not decode -- a lone UTF-8 lead byte with nothing to complete it. The MODEL emitted a broken sequence; the text faithfully renders a wrong answer. model-emitted the stream decodes and a token IS the replacement character. The model chose it. no findings the bytes were sound and the markers appeared anyway, so the loss happened AFTER them -- OURS, in the serving stack. That last case is why an empty trace is a result, not a failure. CORRECTION, and the reason this was needed. v0.3.21 claimed U+FFFD could not be a byte-level artefact, on the grounds that `a`-grave was a single token so one marker had to mean the model chose it. That was wrong. Measured: 'a-grave' utf8=[195, 160] ids=[2131, 2258] pieces=['A-tilde', 'l-bar'] decode([2131]) = U+FFFD <- a lone lead byte, exactly ONE marker decode([2131, 2104]) = 'e-grave' surya-ocr-2's vocabulary is BYTE-level, so one marker per accented letter is precisely the signature of a byte-level failure -- the thing it was claimed to rule out. The vocab lookup was misread too: `vocab['a-grave'] = 2160` does not mean the letter is token 2160; in a byte-mapped vocabulary that key is the glyph standing for byte 0xA0, which alone also decodes to one marker. The "one marker vs two" discriminator does not exist. Digesta's charset instinct was closer to the mark than the rebuttal. base.UNREADABLE and docs/ocr.md now carry the correction rather than the refuted argument. What IS established, measured on the Aruba install at the same DPI through the same transport on perfect synthetic Italian (crisp vector glyphs, 34pt, 250 DPI): surya 29 U+FFFD markers, zero accents surviving chandra 0 markers, all 12 accents intact So the document is exonerated. Also notable: the em dash survived surya intact while every two-byte accent died, and surya emitted U+1EEA (Vietnamese) in place of one -- no decoding bug can fabricate a character that was never in the input. Implementation notes: - surya DROPS this data. Its recognition predictor does pass it on (`BlockOCRResult(..., raw_logprobs=out.logprobs)`) but the model never declares the field and pydantic defaults to extra="ignore", so it is constructed and silently discarded -- their own comment calls it "phase 2 use". Verified by construction: the attribute does not exist on the result. So the value is taken one layer earlier, wrapping `_generate_one`, which still has it. The wrapper only reads and swallows every error: a diagnostic that could break a page would be worse than no diagnostic. - The scan reassembles the WHOLE byte stream and walks it as UTF-8 rather than judging tokens pairwise. Per token is meaningless here -- one character legitimately arrives as two or three tokens, each "invalid" alone -- and the first version of this did judge them pairwise and flagged every accent and every em dash, the exact false positive that would make it worthless. - Capped at 40 findings per page, two tokens of context each side, because a lead byte is only wrong in the light of what followed it. - Clean pages are byte-identical to before: the key appears only when the text actually contains a marker, and it is reported ONCE document-wide under `unreadable.tokens` rather than repeated in every page's meta. - The flag is in _pool_params, because the hook patches surya's backend at LOAD -- without that, toggling it would appear to save and take effect never. A third discriminator needing no code: serve the same weights through llama.cpp (`datalab-to/surya-ocr-2-gguf`, `surya_serve: "llamacpp"`). A different detokenizer on identical weights would convict vLLM outright. tests/test_ocr_token_trace.py: 30 tests. 818 passed across the OCR/config/admin/ runpod selection with the same 10 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A fifth OCR engine. Chandra (datalab-to/chandra-ocr-2) is from the same lab as Surya and is driven the way olmOCR is — one page image over an OpenAI chat endpoint, three serving modes (model | vllm | server) — but its `ocr_layout` prompt asks the model to answer as one <div> per layout block carrying data-bbox (normalised 0-1000) and data-label. So it is the only engine here that returns text, boxes, labelled regions, tables AND page metadata in one pass, with no separate detector. Why it was worth adding now: it is a DIFFERENT MODEL FAMILY from surya-2 (Qwen3-VL's tokenizer, against surya-2's own 65k WordLevel vocabulary) reached over the same transport, at the same pixel ceiling, from the same rasterised page. That makes it a cheap controlled experiment on the U+FFFD accent loss: accents intact under chandra and lost under surya points at the model; both losing them points back at the document's pixels, and then pdfimages -list is the next thing to read. Implementation notes: - NO new pip dependency. The two prompts are mirrored verbatim from chandra/prompts.py (down to the Python repr of the allowed-tag list, which is trained-on wording), the input scaling is chandra's own scale_to_fit reimplemented, and the reply is split with stdlib html.parser rather than pulling bs4 and markdownify into the main venv for a tag split. - Chandra's pixel cap is 3072x2048 = 6,291,456 — the SAME numbers as surya-2 — and it snaps to a 28-pixel grid for the Qwen vision tower. So chandra_dpi defaults to 250 for exactly the reason surya_dpi does. Boxes are de-normalised against the image the MODEL saw, and the size the caller rasterised is reported as meta.source_size when they differ: de-normalising against the pre-scaling size would put every box on a scaled page in the wrong place. - A repetition loop answers 200. A VLM that loses its place repeats the last thing it saw until the token budget runs out, and the result looks like a successful page. chandra_max_retries (2) re-asks with a raised temperature — a greedy decode falls into the same loop twice — and a page that still loops is returned flagged with meta.repeat_detected rather than handed on silently. - The parse never loses page. No blocks at all, content outside any block, an unclosed final div (routine at the token limit), a bbox that will not parse: each keeps the text. A bbox that cannot be read yields NO region rather than a full-page one, because a wrong box looks like a real answer. - Table cells get a space, not a newline, so a table does not flatten to "VoceImporto". License: narrower than Surya's GPL. The code is Apache-2.0 but the WEIGHTS are a modified OpenRAIL-M — research, personal use and companies under $2M funding/revenue, no use competing with Datalab's API. So it is gated on chandra_accept_license, and the pod path differs from surya's on purpose: asking a pod for surya by name IS the GPL acceptance, while chandra's is only ever inherited from a decision already recorded in this install's config. A pod cannot accept a licence on an operator's behalf. lang= is accepted and does not steer it (language_aware: false), like surya-2 and olmOCR: 90+ languages, no language input. So it keeps ONE pool whatever is asked — a language-keyed pool is a second set of loaded instances holding VRAM. Wired through: engine registry, licence gate, per-engine DPI, the vLLM stop path (its VRAM is the subprocess's, not a pool's), the capability catalogue (which now reports boxes/regions/tables from chandra_layout, because with layout off there genuinely are none), model-name aliasing, the automatic fallback chain, pod planning + env seeding, the frontproxy's local-OCR check, models.json capability advertisement, the pipeline step's engine list, admin save/load and the Settings OCR card. tests/test_ocr_chandra.py: 61 tests. 788 passed across the OCR/config/admin/ runpod selection with the same 10 pre-existing failures (broker_config x5, studio_composed_surfaces x3, whisper_server x2). test_ocr_fanout's catalogue test asserted a frozen engine set; it now asserts against the registry, which is what "lists every engine" was always supposed to mean. Also fixed while in there: the Settings page used `parseInt(v) || default` for numbers whose zero is meaningful, so a deliberate 0 saved as the default. New _num() helper only falls back for an empty or unparseable field. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/ocr has accepted `lang` since the beginning, with the comment "reserved": it was parsed and dropped. A client asking for Italian got whatever the install was configured for and no way to tell. It is honoured now, on /v1/ocr, /v1/ocr/batch and the pipeline step, with precedence request > ocr.<engine>_lang > ocr.lang > auto. The request beats the engine-specific setting deliberately: that setting is an install's default, while a caller naming a language knows something about the document in hand. Asking for `auto` explicitly is NOT the same as sending nothing, so an install configured for Italian can still be asked to work it out. The default is auto-recognition rather than any particular language, because a server that has not been told what it is reading should not assume one. For surya-2, olmOCR and docTR that is simply what they already do: they are multilingual and have NO language input to constrain -- surya-2's entire prompt is "OCR this block image to HTML." and its own module warns the wording is the training-time contract, so there is nowhere to put a language and nothing to gain. Rather than let `lang` look effective everywhere, every response now carries what was applied AND whether the engine that served could act on it ("lang": "it", "lang_applied": false), and /v1/ocr/engines reports the same per engine so a client can choose before sending. PaddleOCR is the exception that shapes the rest. It loads a language-specific recognition model and its options are sent once at load, so an instance is built for one language and cannot be re-languaged. Hence: `auto` becomes ocr.paddle_auto_lang (default `latin`, a SCRIPT family rather than detection -- PaddleOCR's own default is `ch`, a worse guess for most documents than no guess), and paddle gets one pool per language, keyed on the model it is BUILT with so `auto` and `latin` share a pool instead of loading it twice. The multilingual engines keep a single pool whatever is asked. That is not tidiness: a language-keyed pool is a second set of loaded instances holding VRAM, and a production install runs 24 surya instances. Both pool-iteration sites already use .values(), and _Pool keeps the plain engine name, so eviction and the VRAM releaser handle the new keys unchanged. ocr.lang's shipped default moves "it" -> "auto"; installs that set it explicitly (Digesta production does) are unaffected. ocr.surya_langs stays as the legacy setting it is -- it reaches only the <=0.17 recognition API and does nothing on the pod path. 21 tests. The 8 failures in test_broker_config / test_studio_composed_surfaces / test_whisper_server_local_models are pre-existing, verified at HEAD. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
ocr: olmOCR has to trust a pod's certificate itself, and U+FFFD is one token not two bytes (v0.3.22) olmOCR could not reach a rented pod at all: CERTIFICATE_VERIFY_FAILED. A pod on the direct-TCP path serves HTTPS with a certificate signed by THIS install's own CA (codai/api/pod_tls.py), at a public IP that was not known when the certificate was issued, so stock `requests` fails it twice over. Surya never hit this because surya attaches through the fan-out gateway, which learned to trust pod certificates in 0.3.16. olmOCR talks to the upstream the dispatcher chose DIRECTLY, so it needed the same trust of its own -- and the gap only surfaced when olmOCR was asked to cover for surya, which is exactly when a fallback engine must not be the thing that is broken. It now goes through codai/api/pod_http.py, which verifies https-to-a-bare-IP against our CA with the hostname check off and hands every other URL -- localhost, a named server, another coderai -- to requests untouched. Swept the rest of the tree: every other raw requests call targets a localhost worker or RunPod's own public API, where pod_http would be a pass-through anyway. test_ocr_olmocr's HTTP-status test patched sys.modules["requests"] wholesale, which now breaks pod_http's own import of it; it patches pod_http.post instead and asserts the same two statuses. Also records what the accent investigation actually established, because the natural reading of U+FFFD is wrong and expensive. It is NOT a charset bug: U+FFFD is a real entry in surya-ocr-2's vocabulary (id 65421 of 65425, the only id that decodes to it), the tokenizer is WordLevel and character-level for Latin so an accent cannot be split across tokens, and the decisive detail is the COUNT -- a byte-level mis-decode yields TWO markers per accented letter (a-grave is two UTF-8 bytes, each one undecodable), and what arrives is ONE. One marker is one token. Every other hop was checked: _html_to_text is regex plus html.unescape and survives literal and entity forms; the isolated-venv worker emits json.dumps with ensure_ascii=True, so the pipe carries pure-ASCII escapes and the locale cannot matter; the gateway forwards raw bytes. And it corrects the previous commit's claim. Raising DPI is right on its own merits -- 200 really was feeding surya 62% of the pixels it accepts -- but a production document came back with the same markers at 200 AND at 300, which is evidence against too-few-pixels being the whole story: a perception failure would be stochastic, not every accent every time. The docs now say what distinguishes the remaining candidates (the document's embedded scan resolution, which `pdfimages -list` reports and which no rasterisation DPI can improve, versus the serving stack) and how to tell them apart: OCR perfect synthetic Italian through the same path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Accented letters came back from surya as U+FFFD -- "piu`" as "pi?", "facolta`" as "facolt?", nine of them in one page, as real EF BF BD bytes. That is exactly what a byte-boundary decoding bug looks like: a` is C3 A0, and byte-level BPE splitting it across two tokens is the classic cause. It is not that. U+FFFD is a REAL ENTRY in surya-ocr-2's vocabulary -- id 65421 of 65425, the only id that decodes to it, with <unk> a separate token at 65424. The tokenizer is WordLevel and character-level for Latin text, so a` is a single token (2160) and an accent CANNOT be split across two: no partial-UTF-8 artefact is possible, and per-token decoding is byte-identical to decoding the whole string. Surya reads text only from message.content, never from the per-token logprobs, and the fan-out gateway forwards the upstream body as raw bytes. So the model predicted "there is a character here I cannot identify" and every hop delivered that faithfully. Two things follow. It is a RESOLUTION problem. Surya caps its own input at 3072x2048 = 6,291,456 pixels in scale_to_fit, which is called with DEFAULTS ONLY -- no surya setting raises it, IMAGE_DPI_HIGHRES included. Measured, for A4: 200 DPI reaches the model as 3.89 Mpixels, 250 as 6.03, and 255 saturates it; 300 and 400 are downscaled back to the same 6.23. The shared ocr.dpi = 200 was therefore handing the model 62% of the pixels it accepts, and a diacritic sits above the x-height at ~3 pixels tall. So DPI is now chosen per engine (ocr.surya_dpi, default 250) and resolved BEFORE rasterising; paddle and docTR have their own optima and keep ocr.dpi. The lookup is generic (f"{engine}_dpi"), so another engine only needs the field. Resolving the engine first also refuses an unknown one before a large PDF is rasterised rather than after. It must not be SILENT. "pi?" indexes, searches and reads wrong for ever, and for judicial documents a known unknown beats a quiet corruption. Lines, pages and the document now count their unreadable glyphs and locate each one with the confidence of the line it sits in -- the finest granularity that exists, since surya reports one mean token probability per block, not per character, and that is also the signal worth acting on: a low-confidence line with an unreadable glyph is a line to re-read at a higher DPI. The key is OMITTED when there are none, so a clean response is byte-for-byte what it always was, spots are capped with a truncated count, and the text itself is never rewritten -- stripping the marker would destroy the only record of where the loss is. 20 tests, including the arithmetic that chose 250, kept so a later edit has to face it. The 8 unrelated failures in test_broker_config / test_studio_composed_surfaces / test_whisper_server_local_models are pre-existing -- verified identical in a clean worktree at HEAD. Co-Authored-By:
-
- 11 Oct, 2026 8 commits
-
-
Stefy Lanza (nextime / spora ) authored
/v1/models answers the moment a vLLM process starts — minutes before it can serve, because the checkpoint still has to be downloaded (every pod re-downloads it without a network volume) and the KV cache built on first use. The pool called that ready, every client read it as usable, and the whole model load landed inside somebody's first request, where it is indistinguishable from a hang. That is what cost today. One OCR page took 807s end to end — ~400s image pull, ~400s weights — while the client timed out at 180s and then 300s and concluded the serving was broken. It was not: nobody had ever waited long enough, and the only reason we know is a probe run with a 1400s timeout, which came back 200 with text and conf 0.9019. So the pool now sends one token against the served model before calling the pod ready. The pod bills through the load either way; this only decides whether the wait is visible as a pod that is not ready yet, or hidden inside a request that looks stuck. `pods_ready` becomes a signal a client can act on, which is what every client already assumed it was. A failed warm-up is reported and the pod used anyway: it may still serve (an engine that takes no chat completions, a model whose warm-up shape we guessed wrong), and rejecting a pod we have already paid to boot over a diagnostic request would be worse than the hidden latency this removes. Off with warmup_on_boot=false or warmup_timeout_s=0. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A rented pod is the one machine in the system whose logs cannot be read. RunPod publishes no pod-log API — its REST spec lists 23 routes and none serve logs, which is why every attempt returns HTTP 400 — and boot.sh writes its phases to a file and then execs uvicorn, whose output goes to the container's stdout and nowhere else. /boot therefore covered the way up and nothing after it, and _dump_pod_logs only ever ran on a failure, printing into the orchestrator's log. That cost an evening. Three pods were investigated in one session and two were already destroyed by the time there was a question to ask — one reclaimed by RunPod mid-request, taking the only evidence with it — so "the serving hangs" stayed a hypothesis for an hour. It was not hanging: a page took 807s, most of it the weights download every pod repeats because global volumes are console-only, and every client had given up at 180-300s. So the pod keeps a bounded ring of its own stdout and stderr (codai/api/podlog) and serves it at /logs with a cursor, and the pool drains that on every scaler tick into <config_dir>/pod-logs/<pod_id>.log — kept AFTER the pod dies, which is the whole point. It also drains once more before tearing a pod down, because the interesting lines are the last ones and they are only readable while the pod exists. A pull, not a push: the orchestrator already reaches pods to serve requests, so the bearer token and pinned TLS apply unchanged and a pod with no outbound route still gets its log kept. Bounded at both ends, since neither disk is free, and a gap in the pod's ring is written into the file instead of passing silently. The stdout tee writes through to the real stream on every failure path — a logging aid that could break printing would be worse than none. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod reclaimed an A40 mid-document today. The pool went on reporting it healthy, so every page chose the same corpse: the in-flight count climbed, the pod was never dropped, and nothing drained. From the client that reads as "requests neither complete nor time out", and it is what wedged the test. RunpodBackend has always handled this — _unreachable() drops the pod and the request goes again. The fan-out gateway never did: it returned 502 and left the dead pod in the pool as healthy. So a lease can now report its upstream dead, which drops the pod, clears the cached candidate set and releases the slot; the page is retried ONCE against whatever replaces it, and a third attempt would just be a retry loop against a real outage. Only connect-level failures count as dead. A read timeout is a model taking its time over a dense page, and discarding a healthy pod for that would be an outage of its own. Also: the first version of that handler could not work. It used a metaclass with __instancecheck__ so `except _DEAD_UPSTREAM` would match httpx's errors — but `except` matches on the class hierarchy at C level and never consults __instancecheck__, so it caught nothing. A test caught it; the check is now an explicit isinstance() where it cannot lie. And the registry path again: 0.3.15 keyed the fallback off XDG_CONFIG_HOME, which the container launcher exports only into the processes IT starts. A diagnostic script entering the container another way printed a THIRD registry path and created a second pod TLS CA. The fallback now reads CODERAI_CONFIG_DIR, the variable the launcher actually sets, so every process agrees however it was started — and since pod_tls.tls_dir() derives from the registry path, the CA follows. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
refuses our token is not an asset (v0.3.17) Both from watching a real OCR run. THE LOST INTENT. A client asked for the fleet back while a page was still being read. The pods correctly stayed — work is never dropped to save money — but the intent was then forgotten: the pool fell back to idle_timeout_s (900s there) and billed for a fleet nobody had asked to keep. It took a second teardown call, four minutes later, to actually release it. The pool now remembers: pods kept only because they are working are retired the moment they fall idle, the same "do not wait out the timeout" treatment a closed schedule window already gets. Asking for warm pods again cancels it. Also fixes the reply and the log, which said "floor 1" when the 1 came from work in flight. That reads as though the operator had asked for the pod — the opposite of what it meant. Demand and configuration are now reported separately ("needed" and "floor"), because they are two different reasons a pod survived. THE ADOPTED POD THAT 401s. After a restart the pool adopts the pod it left running instead of renting another, and carries that pod's bearer token across through the registry — when one was recorded. With none recorded it fell back to our own token and never checked the result, so a pod launched with a different one answered 401 to every request for the rest of its paid life. Seen today, right after a restart: "reusing the pod already running" followed by 401 on every OCR page. The health probe that found it proves nothing here — a coderai pod answers /healthz to anybody. Adoption now probes an AUTHENTICATED path with the token it would use, and a pod that refuses it is terminated rather than adopted: with one registry per deployment (0.3.15) an unrecorded key is a lost key, so that pod can never serve anyone, and paying for it is the only outcome worse than killing it. An unreachable pod is NOT treated as unauthorised — that stays with the health machinery, which has a budget for it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
First real document, first real pod: it came up at https://194.68.245.79:22198 and every page came back 502 — OCR upstream pod failed: [SSL: CERTIFICATE_VERIFY_FAILED] unable to get local issuer certificate A direct-TCP pod serves the certificate coderai's OWN CA signed, on a bare IP that certificate can never name. codai.api.pod_http has always known this: it pins to pod_tls.ensure_ca() and switches hostname checking off. The fan-out gateway is async and could not reuse that requests-based module, and I gave it a bare httpx client — so the one hop that now carries every OCR page was the one hop with no pod trust. It keeps TWO clients per event loop, chosen by pod_http.is_pinned_url: the pinned context trusts only our CA, so using it for everything would reject a cluster node whose certificate a public CA signed, and sharing one client would let whichever upstream connected first decide the policy for both. Worth recording: this was never fan-out-specific. With serve_fanout off, surya attaches straight to that same https:// IP with its own httpx and fails identically — `model` mode has never worked against a TLS pod. The gateway is what makes it fixable in one place. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two things, both from today's live incident. THE SPLIT REGISTRY. _pod_registry_path() fell back to ~/.coderai whenever config_manager was not initialised in the calling process. The container's launcher puts the real config dir one level deeper — it exports XDG_CONFIG_HOME="$CODERAI_CONFIG_DIR" and uses "$CODERAI_CONFIG_DIR/coderai", which is also where platform_paths.user_config_dir() lands. So a process with the config manager wrote one registry and a process without it read another, and the pods each owned were orphans to the other. That is what made the documented mutual-kill loop recur: pod created 19:24:56, terminated as untracked 19:24:57. The fallback is now the same XDG-aware path the launcher computes, and the resolved path is logged once — a registry that splits silently is invisible until it costs a GPU. CLIENT-FORCED TEARDOWN. `pods: 0` released a client's lease but gave nothing back: the pods then waited out scale_down_after_s, or idle_timeout_s before that. The automatic path is slow on purpose — it infers from a momentary reading, and a cold pod costs minutes to replace — but a client that has just finished a batch is not inferring. `teardown: true` sheds the surplus at once, skipping both the window and the one-per-pass limit, and answers with what happened. It still cannot scale through the operator's min_pods, and a pod with requests on it drains rather than dying under them. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Observed live on the Digesta orchestrator while testing OCR: pod created 19:24:56, "terminated STALE pod … not tracked by any pool" 19:24:57. The pool then waited out its 900s boot timeout on a pod that no longer existed, every OCR request 503'd, and the next attempt rented another. This is the race the source already documents from an earlier incident — "the nvidia engine created a pod and the radeon engine terminated it four seconds later, then the reverse — a mutual kill loop that rented and destroyed pods until stopped" — and the defence against it was supposed to be the age guard, "never reap a pod younger than this, whatever the registry says". It was written as `age is not None and age < REAP_GRACE_SECONDS`, and list_pods() returned {id, name, status, cost_per_hr, image} with no timestamp of any kind. So age was None for every pod, the condition was never true, and the guard abstained on exactly the pods it existed to protect. A guard that fails open is worse than none: it reads as covered. Two changes. The guard now fails CLOSED — an undatable pod is left alone and says so — and the listing carries runtime.uptimeInSeconds so it can measure instead of abstaining. Not reaping costs one more cycle; reaping a live pod costs the work on it and bills in a loop that never serves a request. What made it recur here is worth recording: this install has TWO config dirs, so the cross-process registry that shields a new pod exists twice and each process reads the one the other does not write. That is the deeper fault and is not fixed by this commit. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two faults, one cause: `local` was both the default and the only engine mode that would quietly settle for the CPU. Surya's own settings pick the device and fall back cuda -> mps -> cpu (surya/settings.py TORCH_DEVICE_MODEL), so `local` on a machine with no CUDA ran a VLM pipeline on the processor: minutes a page, a result that looks perfectly normal, and nothing anywhere saying why. paddle and docTR already refuse in exactly that situation — surya was the exception. It now refuses too, with a 503 that names both ways out, and the worker is TOLD its device rather than left to guess. Detection fails closed: torch is asked first (it is what surya will ask), nvidia-smi second, and if neither can answer the answer is no. And `surya_serve` no longer defaults to `local`. An install that said nothing got the one mode that runs the model on this box — on a GPU-less orchestrator whose whole job is renting one, that is the worst possible default. It now defaults to `model`, which adapts to wherever the checkpoint is served and asks for surya_model_id when it has not been told. Safe to change: both production installs set the mode explicitly (`model` on Aruba, `vllm` on zeiss), so nothing was relying on the old default. Running on the CPU is still fully supported — surya_use_gpu = false — it just has to be a decision now. Co-Authored-By:
-