- 15 Oct, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authoredA shared pool is one pod serving several names over one copy of the weights — a base model and its adapters. Live, every request naming the adapter got 404 Client Error: Not Found for url: https://…/v1/chat/completions and kept getting it: the pool keeps the pod warm, so nothing replaced it. The pod, asked directly, listed only the base and reported {"loras": []}. Which of the two names worked depended on which model happened to rent the pod, so it looked intermittent rather than wrong. Five places decided "same model?" by comparing `path`, and a LoRA's path IS its base's path: * the orchestrator seeded the other pool members with no served name, so they arrived with no alias and were registered under their path; * the pod's boot seeder skipped the second entry as already known; * register_model_runtime did the same, which made teaching a running pod a no-op that answered "already known"; * the engine-side seed filter in main.py did it a third time; * and vLLM's service key included the served name, so had the adapter got through it would have started a SECOND copy of a 27B on the same card. Identity is now (weights, name) in all five, and one engine serves one set of weights under every name configured for it: the base under its own, each adapter through --lora-modules, which --served-model-name must not repeat. Adoption checks the image, the GPU class and the bearer token — none of which say anything about the catalogue. So a pod that cannot serve what the pool promises is now TAUGHT: the adapter bytes are pushed, the entry is registered, and the same request is retried on the same pod. Renting a second card to serve a name the running one can hold is the answer this avoids. Once per pod per model; a pod that still refuses is a real failure. An engine is also relaunched when it does not serve the name asked of it: vLLM registers adapters at LAUNCH, and a pod is taught after it boots. Only an ANSWERED probe that omits the name forces that — restarting a loaded 27B because one probe timed out would be worse than the doubt. Finally, raise_for_status() discarded the pod's body, which is the whole message: it names the models the pod actually has. The caller saw only the status line, which reads like a routing fault. The pod's own words now travel. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
-
Stefy Lanza (nextime / spora ) authored
0.3.84 made `_is_bge_m3` see `sparse_linear.pt` on a machine that had never downloaded the repo, so the multi-vector backend was finally selected on a pod. It then loaded the heads with the same cache-only lookup, and `from_pretrained` does not fetch them: it pulls the config and the weights the config NAMES, and bge-m3's sparse/colbert heads are plain tensors sitting beside the model. So they were still absent, and the request came back HTTP 400 "This model has no sparse head." which reads as a property of the MODEL rather than of the disk. `_fetch_repo_file` is the companion to `_resolve_repo_file` for files that have to be READ rather than merely detected: cache first, then a real hf_hub_download. Offline short-circuits, a local directory is the whole answer, and a failure returns None -- the dense path, which the caller already handles -- because a head that cannot be fetched must not turn into a failed load. Both halves of this were the same mistake made twice: a cache-only helper answering a question about existence on a machine whose cache is empty by design. A rented pod is that machine every single time. Needs a coderai-embeddings rebuild: this code runs on the pod. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`_is_bge_m3` keys off `sparse_linear.pt` -- the definitive signal for a multi-vector embedder, since plain dense models do not ship one. It asked `_resolve_repo_file`, which answers from the local disk and NEVER the network. Right for "give me the path", wrong for "does this exist", and the difference is invisible until the model has never been downloaded. On a rented pod it has never been downloaded. The pod is disposable and its cache starts empty, so the check ran before anything could satisfy it: on every pod, every time, bge-m3 loaded through the DENSE sentence-transformers path. The request for sparse was then refused with a 400 telling the caller to serve the model on the coderai-embeddings image -- which is the image it was already running on. Measured end to end on a real A40: 379 seconds, then that. `_repo_has_file` keeps the local answer first and, only when the file is not on disk, asks the Hub for the repo's file LIST: metadata, no download, cached once per process. Offline installs short-circuit on HF_HUB_OFFLINE, a local directory is taken as the whole answer (asking the Hub about a filesystem path would look up a stranger's model of the same name), and any failure answers False exactly as before -- a metadata lookup must never be able to fail a load. Needs a coderai-embeddings rebuild: this code runs on the pod. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
In models.json a model's kind is the dict KEY it sits under. Only the ENTRY travels to a pod, and the pod's seeder reads section = e.get("model_type") or "text_models" A models.json entry carries no `model_type` -- so every capability pod registered every seeded model as a TEXT model, whatever it was. Found on a real A40 immediately after the gateway fix finally let a request reach a pod. It booted in 303s, served /v1/models, listed BAAI/bge-m3, and answered: Model 'BAAI/bge-m3' is a text model and cannot be used for embedding generation. Use the text endpoint instead. Which is the expensive part: nothing looks broken. The routing is right, the image is right, the catalogue is right, and the error blames the caller for calling the wrong endpoint. Both seed builders now carry the section -- `capability_seed_entries` already had it and dropped it at `out.append(entry)`, and `seed_model_env` gets it from the new `entry_model_type()` (by identity, then by id, "" on any failure). Stamped on a COPY, so nothing writes a field into models.json that this install never had, and an entry that already declares its own `model_type` is not second-guessed. Text pods were unaffected, which is why this survived: a vLLM pod serves text, and OCR engines are picked by name rather than registered at all. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The pod booted healthy, answered /v1/models, advertised bge-m3 -- and failed every embedding request with Model 'BAAI/bge-m3' failed to load: Using a `device_map`, `tp_plan`, `torch.device` context manager or setting `torch.set_default_device(device)` requires `accelerate` sentence-transformers loads under a device_map context and transformers refuses that without accelerate. It is in requirements.txt and in the text, images, video and stt-crisper profiles; it was never in this one. What makes it worth a comment in the file rather than just a line: the failure does not look like a missing dependency from outside. The image is healthy, the catalogue is right, and the error names the MODEL. On a real A40 that cost 303 seconds of boot before anyone could find out, and the next error after it ("'BAAI/bge-m3' is a text model") sends the reader somewhere else entirely. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
THE /v1/embeddings bug, root-caused on a live 0.3.80 instead of guessed at. `RemoteGatewayMiddleware` short-circuits on `any_remote_configured()` BEFORE it calls `resolve_target`. That gate knew about capability endpoints, capability pods and `service_url` -- and nothing about a per-model `backend`. So an install whose remote models are all per-model RunPod pods answered "nothing is remote", every /v1 request was handed straight to the local app, and a GPU-less orchestrator loaded the model itself. Measured on production: POST /v1/embeddings for bge-m3 spent 312 seconds downloading from HuggingFace and returned 500, while /v1/models/test/state?model=bge-m3 answered `"placement":"pod"` in the same process. Both were right. `model_placement` is asked directly by every page that reports where a model runs; the gate is asked by the code that actually routes, and the two disagreed in silence. Two earlier fixes aimed at the same report and missed it: v0.3.77 made `model_capability()` infer "embeddings" for an entry in the embedding section, and the config was given an explicit image and capability. Both were necessary and neither was reached. OCR is why this survived a release: the OCR manager calls `placement.choose` itself rather than going through this middleware, so the pod-side pipeline worked and made the gateway look fine. The gate now makes the same two tests `model_placement` makes -- a `backend` WITH its block, for runpod and for host. A leftover block whose backend was flipped off still does not count, which is how a model's pod is meant to be disabled. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three reports from the same run, all of them about giving pods back. SCALE BLOCKED THE WHOLE SERVER. The endpoint called into the pool straight from the coroutine, so while it walked dozens of (card, region) candidates -- a network round trip each -- nothing else was served. /v1/runpod/status stopped answering during exactly the operation someone would poll it about, which makes a slow rental look identical to an outage. It now runs off the event loop and the reply is bounded: after `wait_s` (default 20) it answers `pending` with a request id, and the SCALING CONTINUES -- shielded, because cancelling it would leave a half-made lease and possibly a pod. GET /v1/runpod/scale/{id} has the result for a caller that gave up. A BOOT COULD NOT BE CANCELLED. `_booting` is an integer, not pod objects -- a pod being created does not exist here yet -- so `shed_now` could not see one, count one or stop one. A client that asked for a pod, got four on the way and released them was told `kept:0, needed:1, floor:1` and paid every boot out in full. A shed now bumps a cancel epoch; an ANTICIPATORY boot (warm floor, background growth) checks it during the model load and again just before it joins the fleet, and hands its pod straight back. A boot a request is blocked on is never cancellable, whatever arrives meanwhile: the request is what the pod is for. Booting pods also count as capacity in the surplus arithmetic, and the shed reports how many it stopped. A POD COULD NOT BE RELEASED unless some pool floor could be lowered to lose it. `'pool:digesta-qwen38' has no runpod block` came from code whose own comment says the caller cannot look that key up -- so a live pool is now addressable by the key /v1/runpod/status reports. And nothing anywhere could terminate ONE pod by id: not an adopted pod, not one left by a crash, not one whose pool was reconfigured underneath it. POST /v1/runpod/release does, admin-scoped (leasing is bounded and expires; terminating a named pod is neither), attributing the spend before it drops the pod so the money does not land nowhere. And the over-provisioning: `ensure_ready` incremented `_booting` by ONE and then started `want` boots, so with min_pods 3 three cards were on the way while the 15s scaler, `_should_grow` and the parallel-boot cap all believed there was one -- each topping up against a deficit already being filled. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
From last night's run: ~97% of a 1,000-document batch came out truncated, with no error anywhere. The engine answered 200 every time. Its decode had fallen into a loop, run to the token cap emitting the same fragment, and the resumer correctly refused to continue a reply that was already repeating itself -- so the page was flagged, returned, and indexed. `fallback_engines` never fired, because it only fires on an exception. The same pages read cleanly on another engine (38,055 characters against a truncated stub), so the recovery was there the whole time and nothing reached for it. `ocr.retry_degraded` (default "auto") re-reads ONLY the degraded pages on another engine and keeps the replacement only if it is not degraded in turn. Bounded by construction: a clean document costs nothing, a ruined one costs at most a second pass over its ruined pages, and `retry_degraded_max_pages` caps even that. The page keeps its own index and carries `degraded_retry` saying who re-read it and why; the response reports `recovered_pages`. Degraded means looping or truncated -- the two the engine itself attests. NOT a short page, a blank page or an unreadable one: those have their own meaning and re-reading them spends a card to be told the same thing. Which leaves the other half of the report: olmOCR returned 4,046 characters where chandra read 30,983 on the same page, ending cleanly well short of the cap, flagging nothing. The only evidence an early stop leaves is the company it keeps, so `retry_short_pages_below` compares a page with the DOCUMENT'S OWN median rather than a character count -- a threshold in characters would need re-tuning for every language, DPI and layout. Default 0, OFF, and deliberately: this signal is inferred where the others are attested, and a genuine title or signature page looks identical. A short page is also only replaced by a LONGER one. Off on pods: only the orchestrator can reach another engine's pods, so a pod would re-read on the engine that just produced the flag. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ran both side by side on the same documents instead of reading the spec, and the spec's prose turned out to allow several readings the port took differently. Seven divergences in reference extraction, one in sectioning: * `artt. 163, 164 e 167 c.p.c.` extracted nothing -- the code had to sit immediately after the number, so an enumeration lost the whole citation. It now resolves to its first article, which is what the indexes hold. * `art. 2055-bis c.c.` extracted nothing (only a space was tolerated before the suffix); the hyphen is now kept in the key, a space still squeezed out. * `art. 2043 cod. civ.` extracted nothing -- the code spelled out was in the normalisation table but not in the pattern. * `articolo 2043 c.c.` and `art. 2043 della c.c.` extracted a reference the original does not. Looking like an improvement does not make it one: it puts keys in the field that the corpus it will be matched against lacks. * `legge 241 del 1990` extracted nothing; `l 241/1990` without the dot extracted one. Both inverted. * `art. 360` with no code came back as `art.360`, and `d.p.r. 115/02` as nothing -- the two already reported. And the off-by-one, which was none of the things it looked like. Section bodies were not stripped, so merging two adjacent same-label sections (a filing opening its conclusions twice) joined a body ending in a blank line to the next, turning a line break into a paragraph break -- packed as `\n\n` rather than riding inside one piece. Exactly one character, on exactly the documents that merge. Empty sections were also kept, spending a `parent` id and shifting every id after it. Verified by diffing chunk-for-chunk over 63 generated documents covering CRLF, repeated headers, enumerations, two-digit years and degenerate input: zero divergences in text, section, parent and norm_refs. The one deliberate difference is `**FATTO**`, which the original does not match and `it-legal` does. `it-legal-strict` is added for re-chunking a corpus indexed with the original, where a boundary that moves makes the new index incomparable with the old. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two divergences from the spec, both reported from a real corpus: * `art. 360` with no code came back as `art.360`. Every procedural code has an art. 360, so an unqualified number is not a citation -- it files three different articles under one exact-match key, which is the one thing this field exists to prevent. The code suffix is now required. * `d.p.r. 115/02` extracted as nothing, because the year was pinned to four digits. Two to four now, kept verbatim: expanding `02` to `2002` would split one act across two keys depending on the source's style. The spec's prose allowed both readings; it now says which. Co-Authored-By:
-
- 14 Oct, 2026 13 commits
-
-
Stefy Lanza (nextime / spora ) authored
Switching a model's `runpod.engine` from `coderai-vllm` to `coderai` -- the correct setting for a capability pod -- silently removed its capability. `_CODERAI_SERVER_ENGINES["coderai-vllm"]` had been supplying the classification; plain `coderai` supplies none, and `model_capability()` returned "". Both halves of the failure follow from that: * the front could not classify the request, so the model was loaded LOCALLY on a GPU-less orchestrator; * default_capability_image("") is "", so a blank image field raises "set `image` on the runpod block" at provisioning time. It presented as "tries to load on the wrong machine", which is a long way from "this entry has no capability". The cause: model_capability() receives a BARE DICT, and _model_entry_for() iterates md.values() -- it matches inside `embedding_models` and discards the section key. Nothing had ever told it that an entry in `embedding_models` is an embeddings model. Now it asks: by object identity first (the dict the loader holds), falling back to the id so a copied or rebuilt entry still resolves, and returning False on any failure because a capability GUESS is never worth an exception in a placement path. An explicit `capability` field still wins, which is what that field is for. Only `embedding_models` is inferred, deliberately. The other sections work today, and remapping them all to fix one would change placement for every model kind -- a blast radius this does not need. A test asserts the inference does not reach into text_models, image_models, video_models or tts_models. Found by a user report ("broken -- front loads bge-m3 locally on the GPU-less host; not in /v1/models") after I made exactly that config change. I had already noted this gap when the capability came back empty during the switch, said it was worth raising rather than working around, and then worked around it in the config anyway -- which is how it reached production. 11 tests. Suite at the 25 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
RETRIEVAL CHUNKING. A new endpoint that splits a document into the pieces an index stores, with the document's structure decided per request by a named or inline template. NOT embedding windowing: `embed_long_input` makes ONE input fit a model window and pools it into ONE vector; this produces MANY chunks, each indexable, each with its section, parent and references. It never pools and never drops text. Pure CPU -- token counts are characters / chars_per_token, so no tokenizer is loaded and no model, GPU or pod is involved. Every image serves it, light-core ones included, and a test asserts no torch/transformers import creeps in. The packing loop is reproduced verbatim, including the parts that look improvable: paragraphs joined with "\n\n" (hence the `+ 2`), and an overlap that is a RAW CHARACTER TAIL which may begin mid-word. Changing any of it moves every existing chunk boundary, and then a re-chunked index cannot be compared with the one it replaces. `conclusioni` is its own label and is never folded into `dispositivo`: in a ruling the dispositivo is what the judge decided, in a filing the conclusioni are what a party ASKED FOR, and merging them would let a demand surface as a holding. TWO DEVIATIONS FROM THE SPEC, both recorded in the source where someone comparing against the original will look: * The ceiling is target + overlap + 2 (3682), not the 3680 the prose states -- which agrees with the spec's own "observed max is 3682" two lines later. Asserting 3680 would fail a correct port. * The verbatim patterns cannot match `**FATTO**`, yet the fidelity list asks for that case. The line-anchored alternatives are widened at BOTH ends (`^[\s*#>_]*fatto[\s*#>_]*$`), which is how the patterns that already coped spell it. Purely additive: every line that matched still matches, so no existing match moves -- but a document whose headers were emphasised used to fall back to flat chunking and now gets sectioned, so ITS chunks change and it needs re-chunking. FORWARDABLE, with the hazard that comes with it. /v1/chunk rides the EMBEDDINGS capability: chunks are on their way to be embedded, that image is full-core so it carries the chunker, and a capability of its own would be one nothing advertises -- or would rent a GPU to run a regex split. It was first classified as deliberately local on the grounds that it is cheap; that was the wrong lens, since the orchestrator is the box that was OOM-killed today, so WHERE the CPU is matters more than how much there is. But a template registered HERE does not exist on the remote: forwarded by name it returns 404, or worse matches a DIFFERENT template with the same name and returns different boundaries silently. So a named non-builtin template is resolved and INLINED before the request leaves. The hook never raises -- a body it cannot parse is forwarded untouched -- because a rewrite that can throw would take down every forwarded request, not just a chunk one. THE OCR PIPELINE NOW COVERS ALL THREE VLM ENGINES, not just surya. Leaving chandra and olmOCR on the orchestrator was an asymmetry rather than a design: surya needed a 1.6 GB client venv baked in, while those two speak only HTTP in served mode (PIL + requests, already in the main venv), so for them the pod side costs nothing beyond the rasteriser. Two bugs fixed on the way, both of which would have produced plausible WRONG text rather than an error: the dispatch hardcoded `engine: "surya"` in the form it posted, and the pod env was surya-shaped throughout -- a chandra pod was told to run surya against chandra's model id. Per-engine tuning and per-engine licence acceptance now travel. AND THE LOG THAT CAUSED A MISDIAGNOSIS. One pod request walks a candidate list until a (card, region) pair has capacity, and each attempt printed "provisioning pod for ...". 312 attempts for 10 pods therefore read as runaway provisioning and was reported as such. It now says "trying candidate N/M for ONE pod", so the shape of the list is visible instead of inferred. What these tests CANNOT do is prove byte-identity with the implementation this ports -- that needs fixtures from a live corpus. They prove the documented behaviour on each path; identity is what decides whether a re-chunk is safe and it is still outstanding. Suite at the 25 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`_read_on_pods` returned None in silence when no pod could serve, and the caller then read the document locally. So an install configured for the pod-side pipeline could fall back on EVERY request and look exactly like a success: the OCR works, the text is right, and the operator concludes the new path is working when it has never run once. A test that cannot distinguish the two paths proves nothing about either. Both paths now announce themselves in one line -- the fallback naming the CAUSE ("placement answered 'local' — nothing rented, or the pool could not serve") rather than only the effect, and the success saying the pages were rasterised and read on the pods rather than here. Found while answering "can I launch the new OCR to test?", which is exactly when it mattered: the answer was yes, and the test would have been uninterpretable. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two gaps in 0.3.72/73, both of which left chunking looking enabled while doing nothing useful. THE THRESHOLD WAS A NUMBER SOMEONE TYPED. max_len = int(_emb_cfg.get('n_ctx') or _raw.get('n_ctx') or 8192) Set n_ctx above what the model accepts and the windows are built too big; the tokenizer then truncates each window at its real limit anyway and chunking silently stops protecting anything -- the same invisible loss this code exists to remove, reached through a config typo instead of a missing feature. The window is now min(configured, tokenizer.model_max_length), with one trap handled: HuggingFace sets model_max_length to a sentinel near 1e30 when the checkpoint declares none, and believing that means never chunking at all, so a value that large is read as "unknown" and the caller's number stands. Same clamp on the sentence-transformers path, taking the smaller of max_seq_length and the tokenizer's own limit. THE MULTI-VECTOR PATH COLLECTED THE COUNTS AND DROPPED THEM. It builds its own response, so the dense path's report did not cover it -- chunking was invisible on exactly the path that needs it most, which is the bug this feature was written to fix. It now reports "chunked": {"windows": [...], "pool": ..., "token_limit": N} and logs the limit with the merges applied. `token_limit` is deliberate: it is what makes a WRONG n_ctx visible rather than merely ineffective. 46 tests. Suite at the 25 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
bge-m3's sparse and colbert come from linear heads (sparse_linear.pt / colbert_linear.pt) applied to the last hidden state -- a transformers forward pass, not an embedding server's output. Two consequences that were both silent: A DENSE-ONLY BACKEND NOW SAYS SO. `embedding_types` was simply ignored outside the native path: a caller asking for sparse got a normal-looking response with only `embedding` in it, its hybrid retrieval then ran on half the signal, and the result was merely a worse score -- which reads as a model problem, not a configuration one. Now a 400 that names the cause, where it DOES work (coderai-embeddings), and why the vLLM image cannot: that image is LIGHT core with no torch in its main venv BY DESIGN, so _load_bge_m3 could not even import. Dense-only requests are untouched. MULTI-VECTOR LONG INPUTS ARE WINDOWED. 8192 tokens is ~6000 words, so this bites on whole documents rather than passages -- and when it bit, the tokenizer's `truncation=True` kept the opening and said nothing. Each type is merged the way the encoder already merges WITHIN a window, which is what makes these choices defensible rather than invented: dense mean, RE-NORMALIZED. It is an L2-normalized CLS vector and a mean of unit vectors is not one; cosine similarity on a non-unit vector is quietly wrong. sparse MAX per token id -- the encoder max-pools per token id already (scatter_reduce amax), and a term's weight should be its strongest occurrence, not an average diluted by the pages that never mention it. A test guards against someone later tidying max into mean. colbert CONCATENATED: max-sim scores each query token against ALL document token vectors, so every content token of every window belongs. `embed_long_input=truncate` keeps the old behaviour explicitly, and `first` keeps only the opening window, for an operator who would rather have that than a merge. Also removes three customer references I had put in shipped profile comments, two of which went out in 0.3.69. A capability must be scoped and justified by what it does, never by one install's needs, and a customer name does not belong in a released image. The measurements keep their force without the attribution: "hybrid retrieval wants dense and sparse from one pass", "a production fleet", "a GPU-less orchestrator". 39 tests. Suite at the 25 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Only the GGUF path ever chunked. Every other backend passed `truncation=True` and returned a confident UNIT vector of roughly the first page of a long document, with nothing in the response to say so: truncation=True, max_length=1800 # GME-Qwen2-VL, a hardcoded ceiling truncation=True, max_length=max_length # bge-m3 native model.encode(texts, ...) # sentence-transformers, its own cap Not an error -- a wrong answer that looks right, which is the failure mode this codebase keeps paying for. A caller had no way to tell an embedding of a document from an embedding of its first page. Now a long input is windowed to the model's real limit, every window of every input goes to the backend in ONE batch (a 40-page document must not become 40 round trips), and the chunk vectors are mean-pooled AND RE-NORMALIZED: mean pooling unit vectors does not give a unit vector, and cosine similarity on a non-unit vector is quietly wrong, so the renormalization is part of the contract rather than a tidy-up. Tunable per-model -> env -> default, the convention the admission gate already uses: embed_long_input chunk | truncate | error (default chunk) embed_chunk_pool mean | first (default mean) `truncate` stays reachable and is now DELIBERATE: for a short-text encoder a 40-page input is usually a caller bug, and an operator may prefer the old behaviour to a vector pooled from forty windows. `error` says so out loud. AND IT IS REPORTED: resp.chunked = {"windows": [...], "pool": ...} whenever any input needed more than one. Replacing a silent truncation with a silent pooling would have been the same bug with a better answer. The GGUF path's hard-won guard now covers every backend: each window is RE-ENCODED and shrunk until it verifiably fits, because a detokenize->retokenize round trip is not token-count-stable (specials and BOS are re-added) and a window cut at exactly the limit can re-inflate past it -- which is how that path first tripped llama.cpp's GGML_ASSERT. Wired: sentence-transformers, and GME-Qwen2-VL (windowed BEFORE the chat prompt is built around the text, since the processor's own truncation would cut the template's closing tokens as readily as the body). GGUF already chunked. NOT wired, on purpose: the CLIP family (clip, geoclip, vpr) takes 77 tokens BY DESIGN -- those encoders embed captions, not documents, and pooling forty windows of a contract into a CLIP vector would be meaningless. A test asserts this so "chunk everywhere" is not later read as "chunk where it makes no sense". NOT wired, needs a decision: bge-m3's native multi-vector path returns dense PLUS sparse and colbert. Dense mean-pools fine, but sparse would need max-merging per token id across windows and colbert would concatenate per-token vectors. Those are retrieval semantics, not a mechanical change, and improvising them would degrade search in a way that looks like a model problem. 26 tests. Suite at the 25 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The timer's own comment promised "a restart only happens on a real upgrade, and only when the fleet is idle (or after MAX_DEFERRALS hours of never being idle)". The second half could not happen: guard_idle || exit 0 # a busy host exits HERE ...upgrade... N=$(cat "$DEFERRALS" ...) # the counter only ever incremented HERE So on a busy host the script left before the image was even pulled, the counter the force depends on never advanced, and MAX_DEFERRALS was unreachable. Measured on the Digesta orchestrator 2026-10-14: 24 skips in one day, `deferrals` still 0, two versions behind, and 0.3.68 landed only because the host happened to be idle at 17:01. Now a busy skip COUNTS, and at MAX_DEFERRALS the script books a forced upgrade for the next midnight. The 00:00 tick of the existing hourly timer fires it, so there is no second unit to install and nothing stranded by a reboot. Midnight because it is the cheapest hour to interrupt a business-hours workload and a time an operator can predict without reading the script. Four things that each had a way to go wrong: * booked ONCE, not re-booked on every busy tick -- otherwise the deadline walks forward forever and never arrives; * a corrupt deadline file is discarded, not obeyed: garbage must not read as "the time has passed" and restart the service, nor sit there blocking a fresh booking; * a forced run does not defer at the restart step either -- deferring there would put it back where the booking was made; * MAX_DEFERRALS=0 keeps its old meaning (wait for an idle moment, however long, never force), so an install that did not opt in is unchanged. The booking is announced in the config banner of every later run and cancelled by deleting one file, because a forced restart that nobody saw coming is worse than a late upgrade. 16 tests, and the ones that matter are behavioural rather than source-matching: they extract the real arm_midnight_force and force_due from the script, run them under bash against a temp state dir, and assert the timestamp really is local midnight and that force_due flips at the right moment. A test that restates the logic only proves it agrees with itself. Suite at the 25 pre-existing failures. NOT installed on Aruba: that copy is byte-identical to this one and runs hourly, so overwriting it mid-run would be editing a running script. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An OCR VLM needs a model server AND the surya pipeline that drives it. Which machine runs the second one is now chosen by the pod IMAGE, and by nothing else: coderai-vllm the pod serves the model; the ORCHESTRATOR rasterises every page and sends a ~1.06 MB base64 PNG per A4 page at 250 DPI -- and again for every resumption round. Unchanged. coderai-vllm-ocr the pod carries surya's client venv and pypdfium2, so it is sent a ONE-PAGE PDF (~474 KB, 1.7 ms to split here against ~105 ms to rasterise) and reads it on the card with its own vLLM over loopback. A separate `pod_ocr_pipeline` flag was built first and removed. Two switches that can disagree is the invisible mismatch that cost three rounds of blaming a pod image for orchestrator bugs; choosing the image IS choosing the mode, it already lives in models.json and on the model page, and an image that cannot run a pipeline can no longer be asked to. PAGE-LEVEL FAN-OUT IS UNCHANGED. One page is one request in both modes -- `placement.choose` picks an upstream per page -- so every pod in the pool shares the work and none owns a document. Sending a whole PDF to one pod would turn an 8-way parallel read into a serial one, which is why the split is per page. THE TUNING TRAVELS, not just the wiring. surya_dpi, the page budget, the continuation count, the repeat-detector thresholds, langs, the page window, the resident cap. Every one was set in response to a measured failure, and a pod reading pages at surya's own defaults would be indistinguishable from the "pre-fix pod image" that was wrongly blamed three times. A test cross-checks every CODERAI_OCR_* the plan sends against what config.py actually reads: a variable sent and never read is a setting that silently does not apply, which is the exact shape of two bugs found earlier today. FALLBACK, and what it refuses to hide: * no pod rented -> the WHOLE document is read locally, not half each way (a document read by two pipelines is the hardest kind of result to explain); * a pod answers 500 -> raised, not quietly re-read here; a broken fleet must stay visible; * the image lacks the pipeline -> the orchestrator does what it always did. The image, built and verified (14306 MB, /healthz OK): venvs/surya 1.6 GB torch 2.14.1+cpu cuda=None + ultralytics 8.4.177 venvs/vllm 11 GB main venv: pypdfium2, and no torch at all (light core intact) A CUDA torch in the surya venv would add ~4.2 GB AND put surya's small local line detector on the card serving the model; the venv check asserts torch.version.cuda is None so a resolve cannot quietly undo it. 35 tests. Two flaws of my own that they caught: four passed alone and failed in the suite because they patched sys.modules while `from codai.models import placement` resolves the name as an attribute of the already-imported package -- now patched on the real module, verified in both orders; and _plan_env used a hand-rolled stub that tested the stub, now built from a real RunpodModelConfig. Suite at the 25 pre-existing failures. NOT YET EXERCISED AGAINST A REAL POD: the orchestrator path has only run against fakes, and the image is local -- not pushed or signed. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
THE WARM FLOOR. A declared floor of 8 took ~40 minutes to exist. Measured on the Digesta orchestrator, one pod per scaler pass at ~5 minutes each: 15:15:27 warm floor 8, 2 pod(s) healthy — renting one 15:21:12 warm floor 8, 3 pod(s) healthy — renting one 15:32:41 warm floor 8, 4 pod(s) healthy — renting one Two causes. First, `maintain()` rented exactly one pod per pass, gated on `not self._booting`. A floor is a DECLARATION, not a prediction -- min_pods, keep_warm or a client lease already said how many should be warm -- so the deficit is filled at once, bounded by the model's existing `scale_up_max_parallel_boots` (no new knob; it already means "how many pods may be booting at once"). Second, `_provision_one` is "Blocking (minutes)" and this called it INLINE from the scaler loop, so one warm boot stalled that loop -- including the registry heartbeat that lives in it and runs every 60s. A stalled heartbeat is exactly what makes a sibling treat our booting pod as abandoned, which 0.3.68 made load-bearing. Warm boots now run on their own threads. Adoption loops too (a restart orphans the whole fleet; eleven pods at one per tick took six minutes), but BOUNDED two ways: `_adopt_shared_pod()` returning True is a report, not a guarantee -- it can succeed without the pool gaining a healthy pod -- and the first `while ... and adopt()` form span forever the first time that happened, taking the scaler thread and the heartbeat with it. At most `deficit` attempts, and stop when an adoption does not raise the count. A pass that adopted rents nothing: renting beside a fresh adoption is how one job pays for two cards. THE IMAGE. coderai-vllm-ocr existed to pin vLLM 0.20.1, the version surya spawns, because a page of crisp Italian came back from 0.27.1 with one U+FFFD per accented letter. That was the wrong conclusion from a real measurement: surya-ocr-2's tokenizer is BYTE-level, and the fleet has been reading accented Italian on 0.27.1 ever since. Two images serving the same weights on two vLLMs is an invisible difference, and it cost three rounds of blaming the pod image. So the pins are now IDENTICAL and the check READS the fleet's pin from disk rather than restating a version -- bumping profiles/vllm cannot silently leave this image behind. What distinguishes the image instead is the OCR PIPELINE on the pod: the surya client venv beside the model, plus pypdfium2 in the main venv, so a page is rasterised and read on the card rather than crossing the network as a ~1 MB base64 PNG (and four more times over for a page resumed four times). CPU TORCH, and it is the whole image-size story. Measured in a resolved surya venv: nvidia 3196 MB + torch 1175 MB + triton 896 MB = ~5.3 GB of CUDA stack, for a venv whose only local torch workload is surya's small line detector. On the pod the GPU is already committed to serving the model, so giving the detector the card would take VRAM and SMs from the thing that matters; on a GPU-less orchestrator it can never execute at all (cuda available: False today while its surya venv carries cu130). ~280 MB instead of ~5.3 GB, and the venv check asserts torch.version.cuda is None so a resolve cannot quietly add 5 GB back. THE DETECTOR rides that same CPU torch, which is why it costs ~30 MB rather than a third torch. detect.py tries ultralytics in-process first (every existing install is unchanged) and falls back to codai/ocr/workers/detect_worker.py in whichever venv has it -- surya's first. A detector that cannot be reached looks like a MISSING detector (the ordinary 503), never a broken install. No weights are baked: detect_model_path defaults to "" and a released image must contain no models, configs, keys or secrets. Tests: 17 warm-floor (+4 for the spin), 14 detector. Four older tests encoded the decisions this reverses -- two asserted the 0.20.1 pin (replaced by one requiring the two pin files to be identical), one asserted the warm floor "must still be single-flight", one counted a source string globally and could not tell intent-held-spelling-changed from a regression (now per path). Suite at the 25 pre-existing failures. NOT BUILT and NOT USABLE YET: nothing has resolved surya-ocr + pillow<11 + torch+cpu + ultralytics together, and the orchestrator cannot drive a pod-side pipeline until the dual mode lands. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Observed live: six pods billing on the RunPod account, none in coderai's interface, and "another engine is booting a pod for 'datalab-to/surya-ocr-2' -- waiting for it" every fifteen seconds for seven minutes. Nothing was booting. A deploy had restarted coderai mid-boot, leaving that pod's URL-less registry entry with no owner. Three fixes, three different mistakes: 1. sibling_is_provisioning() judged the entry's liveness against REAP_GRACE_SECONDS (1200s). That constant answers how long the REAPER waits before killing an unowned pod -- a different question. The registry has a 60s heartbeat, so a dead owner is detectable in seconds. Now PROVISION_STALE_SECONDS (180s = three missed beats). 2. The sibling check ran BEFORE _adopt_shared_pod(), so the recovery path for orphaned pods sat behind the marker that orphaning creates. Five running, registered, billing pods were never adopted. Adoption now runs first: a pod already serving this pool beats waiting for a boot and beats renting. 3. A pod registered at CREATION recorded no $/hr -- the rate was passed only at the ready-registration 80 lines later -- so a boot interrupted before ready could only be reaped with "the registry records no rate for it". The price is known at creation. Also, hourly_usd is now carried over on re-registration like image and gpu already were: an adoption that did not know the rate turned a billing pod free. 14 tests, including the exact live entry (url-less, 910s stale) as a case. test_runpod_client_scale's adopt-before-rent assertion was pinned to the old source spelling; updated to assert the intent, which is now stronger. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.3.66 serialised every pdfium call behind a lock, because pdfium permits one caller at a time and calling it concurrently corrupts its state into "PDFium: Data format error" on valid documents. Correct, and it made rasterisation single-threaded for the whole install -- a ceiling of about one core. pdfium IS safe one-document-per-process, so `ocr.raster_workers` (0 = auto at half the cores capped at 4, 1 = in-process under the lock) hands documents to worker processes. Measured on 12 concurrent 47-page PDFs at 150 DPI: 21.9s -> 10.3s, 2.13x. Not 4x -- the parent is single-threaded receiving the pixels and rebuilding the images, and ~4.5 GB crosses the pipes. The ceiling is the parent. Three things this had to get right, each of which cost a bug first: - A worker is borrowed per CHUNK, not per document. The caller OCRs each window between pulls, which takes far longer than rasterising it, so holding a worker across that would idle it for most of its tenure and cap the install at `raster_workers` concurrent documents -- worse than the lock it replaces. - A pool that cannot start must cost nothing. The first version waited 300s for a worker that could never arrive (missing script), stalling every document for five minutes: the optimisation becoming an outage, which is the exact failure its test was written for. Creation failure is now remembered rather than retried per document, and a busy pool falls back in-process after 30s. - Workers are executed, never imported. `import codai.ocr` costs 377 MB and 1.5s via codai.models.manager -> litellm; paying that per worker, in the subsystem whose other half exists to bound memory, would be self-defeating. The worker imports pypdfium2 and PIL only, ~25 MB. This is why it is not a ProcessPoolExecutor: spawn pickles a function by qualified name and re-imports the package to find it. Also fixes a cost 0.3.65 introduced and nothing but a stopwatch would surface: pricing a document called `pdf[i].get_size()` per page, which LOADS each page object -- 593 ms for a 47-page document, paid UNDER THE LOCK, so every document in the install queued behind every other being priced. `pdf.get_page_size(i)` reads the same numbers from the page tree by index. page_estimate: ~600 ms -> 9.6 ms. Every failure path degrades to a correct in-process render: no pool, a dead worker mid-document, a missing script, an unusable module, an abandoned document. Correctness never depends on the optimisation working. 87 tests across the budget and pool suites. Full suite at the 25 pre-existing failures. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.3.65 moved PDF rendering into asyncio.to_thread so a long document would stop stalling the event loop. That was correct in intent and broke OCR within the hour: until then `load_pages` ran ON the loop, which serialised every rasterisation BY ACCIDENT. Removing the accident let documents into pdfium concurrently, and pdfium permits exactly one caller at a time. Concurrent use corrupts its state and reports failed to rasterise PDF (pypdfium2): PDFium: Data format error on documents that are perfectly valid -- so the failure accuses the input, and the operator goes looking at their files. Three valid Diogene PDFs that poppler parsed cleanly were rejected this way. Measured: 1 document OK, 8 concurrent = 8 failures, 24 = 24 failures. After the fix, 48 concurrent 47-page PDFs all return 47 pages. Every pdfium call -- document open, page size probe, page render -- now runs under one lock, held per CALL rather than per document so documents still interleave page by page instead of queueing behind each other's full length. Rendering still happens off the event loop, so this is strictly better than 0.3.64, which serialised AND blocked the loop. Cost, stated plainly: rasterisation is single-threaded, a ceiling of about one core. That is what 0.3.64 effectively was, so it is not a regression against it. Rasterising in a process pool (pdfium is safe one-document-per-process) is the scalability fix and is deliberately not bolted onto this hotfix. Two process lessons, in the tests so they outlive the memory of this: - this path is only ever exercised concurrently, so testing it sequentially tests the one case that cannot fail. I shipped it tested sequentially. - the error STRING predated the change, and I used that to argue the behaviour did too. "This message is not new" is not evidence of "this is not new". Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`ocr.max_concurrency` bounds how many pages are BEING READ. Nothing bounded how many were RESIDENT, and those differed by a factor of the document length: ocr_document rasterised every page up front and held the list until the response was written, so pages queued behind the semaphore were in memory too. The real figure was (documents in flight x their page count), which no setting expressed. A production install was OOM-killed twice inside forty minutes -- 90.7 GB with max_concurrency at 1024, then 92.7 GB with it at 128. Eight times smaller, half a gigabyte worse, which is the proof the semaphore was never holding the memory. Two bounds, on the two things that grow: - `ocr.page_window` (0 = follow max_concurrency, capped at 32) reads a document a window of pages at a time -- rendered, read, measured, detected on, dropped, then the next window. Residency is O(window) not O(document). Measured on a 40-page A4 PDF at 250 DPI: 40 pages held = +974 MB RSS, 32 = +768 MB, 8 = +215 MB. - `ocr.max_resident_mb` (0 = derive it) is a ceiling on the bytes in-flight documents may hold as pages, charged before a page exists: page_estimate() prices a PDF from its page tree without rendering it. Derived from the smaller of the cgroup limit and host RAM -- the container limit first, since a 2 GB container sizing itself from a 128 GB host has no ceiling at all. The estimate carries a measured overhead factor (PAGE_OVERHEAD = 1.6): the RGB pixel arithmetic under-predicts real residency by 40-55%, and a budget that under-charges admits half again as much as it meant to. A document larger than the whole budget still runs, clamped and alone -- a gate that can refuse forever is a deadlock with a polite error message. Reservations are returned on cancellation too, or a disconnected client shrinks the budget permanently. Also: rendering now runs off the event loop (a long PDF used to stall every other request in the process), the engine is sticky across windows so a document is not read by a different engine in each one, and pages already read keep their results when a later window falls back. The comment in config.py that reasoned "64 in flight is ~1.2 GB" is corrected in place -- it assumed resident == admitted. 59 tests. Not bounded, and not measured: /v1/ocr/batch still buffers every uploaded file; source bytes are ~1% of rasterised pages, so that was not the term behind the kills. Co-Authored-By:
-
- 13 Oct, 2026 17 commits
-
-
Stefy Lanza (nextime / spora ) authored
Raised from the Digesta side: the one high-leverage way to make surya fall through less often is to make its repeat-detector and retry tunable. Half right, and the other half turned out to be the bigger lever. Checking the installed surya rather than the description: `_should_retry` stops at `max_retries` and RETURNS the looping text — surya does not return empty. The empty page comes later, because looping markup parses to no blocks. So the symptom is real and the mechanism is not what it looked like. **The lever is that surya never said it looped.** chandra has reported `repeat_detected` since it shipped, with the text returned anyway because a looping transcription usually has a correct beginning. surya reported nothing, so a short or empty page was indistinguishable from an unreadable scan — and those need opposite handling: one is a cascade candidate, the other means the document is bad. surya now flags it at the moment the verdict is made (the retry hook's last call for the page), with `repeat_chars`, the length of the reply that was still looping: "looped and produced four thousand characters" has a usable beginning, "looped and produced nothing" does not. **And the detector is now retunable** — `ocr.surya_repeat_max_repeats`, `surya_repeat_window`, `surya_repeat_scaling`, 0 on each leaving surya's own (4, 500, 3.0). Hooked like `_should_retry` already was, since the vLLM backend passes neither. A false positive is expensive: the page is re-run several times at rising temperature and then parsed from a reply nobody wanted, usually yielding less text than the first one had. Operator settings, deliberately not per-request — they describe the corpus (tables, dotted leaders and form fields repeat short strings legitimately), not the page. An explicit argument from surya's own code still wins, so its `cut_from_end` call keeps working. 11 tests in tests/test_surya_loop_reported.py, including that a loop surya then rescues is not recorded as a verdict, and that the thresholds must be in place BEFORE the hook is installed (it patches once per process). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two gaps I left open last night, both about the ledger telling the truth. **A pod that went away without a pool noticing was never billed to anyone.** A pool closes its own pods' bills — `_terminate` for one it retires, `forget` for one RunPod says is gone — and neither runs for a pod only the REGISTRY knew about: one orphaned by a restart, or terminated from the console or straight through the RunPod API. Observed: pod lgdgfs749chkib, killed by hand, never appeared in the ledger while two pods that went through coderai did. `close_bill_from_registry()` now computes the hours from the registry's own `created` — the only record of when a pod started once no handle exists — and records them against the pool it belonged to. Both reaper paths use it, so the "terminated as stale" and "vanished from the account" cases cannot drift apart, and a pod a pool still holds is deliberately skipped: `forget` ledgers that one, and counting it here as well would double-count every pod RunPod reclaims. **`record_spend` dropped the `reaped` flag it was being handed.** It keeps a deliberate allow-list, which is right — the file is persisted and pruned for every pod — but the list has to carry what distinguishes one kind of charge from another. A pod the reaper killed looked in the file exactly like one a pool retired on purpose. `reaped` and `reason` are kept now. Writing the test for the first fix found a third thing, and it was mine: v0.3.56 made a stale registry entry stop counting as "known" so the reaper could kill the pod it named — but the reconcile was only ever asked about known ids, so that same change stopped stale entries being RECONCILED. An entry for a pod that no longer exists was then never pruned and its hours never attributed. The reaper now reconciles against the whole registry; `known` keeps its one job, which is deciding what is shielded from reaping. 9 tests in tests/test_ledger_out_of_band.py, including the double-count guard and the pruning regression. test_runpod_client_scale's source anchor moves with the change and says why. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Resumption was surya's alone. chandra and olmOCR stopped at their configured cap and returned whatever they had — and what they had is the worse half of the problem: chandra emits HTML, so a cut-off reply is unclosed markup that parses to a fragment of the page or to nothing, and olmOCR's reply opens with YAML front matter, so one cut off before the body closes parses to metadata and NO TEXT AT ALL. In the finished page both are indistinguishable from a model that could not read it, which is the confusion this whole area exists to remove. The rules now live once, in codai/ocr/resume.py, and they are the ones surya learned the hard way rather than a fresh guess: * both vLLM flags or neither — `continue_final_message` AND `add_generation_prompt: false`. With the second left at its default the chat template appends a fresh assistant turn and the model answers again about the same image, which reads exactly like a model repeating itself and did for a whole day; * never resume a reply that is already looping: it did not run out of room, it ran out of page, and resuming can only extend the loop; * the context window is the ceiling, not the budget. Resumption re-sends everything so far as INPUT, so the room shrinks by what was just produced, and a segment that no longer fits means the cap is too BIG, not too small; * progress has to be real: a segment identical to the previous one, or one that comes back looping, ends the loop; * `finish_reason` is authoritative both ways — "stop" with a full budget is a complete page and must not be resumed, which a token count alone cannot tell. Both engines' ask paths now return a `resume.Reply` carrying the finish reason and the usage counts (they were discarding both), take a `prefix` that goes back as the prefilled assistant turn, and report `model_attempts` in the same shape surya produces — including the budget in force, so a caller can tell a request that landed from one that was ignored. Being in-process, they have no thread boundary to cross: `page_options()` reaches them directly. New settings, in config.json, the admin API and the settings page: chandra_max_continuations / chandra_context_tokens and olmocr_max_continuations / olmocr_context_tokens. The UI parses them as numbers rather than falling back on `||`, because 0 is a real setting here (resumption off) and not "use the default". chandra's `detect_repeat_token` moves to being shared rather than copied: it is a port of chandra's upstream util and answers "has this reply fallen into a repetition loop", which is not chandra-specific. 42 tests: tests/test_ocr_resume_shared.py (28, the rules) and tests/test_ocr_resume_engines.py (14, each engine's own ask path). The existing chandra/olmocr/confidence/pod-TLS harnesses were updated to the new ask contract — it changed deliberately, and the stubs say why. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Adoption matched on the pool and a health probe, and the registry recorded nothing about what the pod was RUNNING. Two silent mistakes followed from that: * **the image.** The same model on a pinned tag and on `:latest` is different code. A pool configured for one could inherit a pod running the other, and nothing in the system could tell — which is precisely the confusion that cost a whole evening today: an OCR result was attributed to code that was not what the pod was running. * **the card.** A pod rented as a FALLBACK is on a card the configuration accepts only when its preferred one is unavailable. Inheriting one would serve a lease that asked for an A40 *and nothing else* (v0.3.54) from an RTX A6000 — the exact comparison that feature exists to protect. Live tonight the pool had both: an adopted A40 and a rented A6000 fallback for one model. So `register_pod` records the image and the card at creation (and keeps them across a re-registration, since an adopting owner does not know them), and `find_shared_pod` takes `image` and `gpus` and refuses anything that does not match, naming the pod and the reason. The one caller passes the image from the plan it would launch and the cards from `narrow_gpu_choice` — so a client lease's narrowing constrains adoption exactly as it constrains renting. An entry recording neither field is refused rather than trusted: refusing costs one extra pod once and the old one is reaped, while trusting costs a silent answer from the wrong code. A caller that names neither constraint keeps the old behaviour, so this adds constraints without adding requirements. Also here: an adopted pod is now billed from when it was CREATED, not from when it was taken over. `_terminate` computes the spend from `started_at`, which adoption set to "now" — so a pod created at 22:27 and adopted at 23:10 was ledgered for 16 minutes of an hour's life, and the rest was charged to nobody. The registry already records `created`, and only cost and uptime reporting read `started_at`. 11 tests in tests/test_adoption_matches_image_and_card.py, one more in tests/test_pod_token_adoption_race.py. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two reports from the Digesta side, both of which my own code caused. **"resumption is disabled server-side (ocr.surya_max_continuations = 0)".** It is not, and never was. Both config files leave the key unset, the dataclass default is 8, and the running process resolves 8 for either candidate config dir — checked inside the container through the app's own loader. The number came from this, in ocr_worker: elif capped and not resumed: info["why"] = (f"the reply stopped at the {cap}-token cap and " f"resumption is off (ocr.surya_max_continuations = 0) …") a hardcoded string in a branch that is reached four different ways: budget zero, a generation already repeating itself, no room left in the context window, or a reply that could not be resumed at all. It asserted a configuration value it never read, and it was believed — twice — and acted on. A diagnostic that names the wrong cause is worse than one that says nothing. So the reason is now recorded where it is known (`_no_resume`) and reported as itself. The budget actually in force goes into `model_attempts` as `max_continuations`, because printing the cap but not the budget is why a caller could not tell a request that landed from one that was ignored. The old message survives for the one case where it was true, and says it is about this page rather than the server. **Growth rented a card beside a pod it already owned.** Adoption was attempted in three places — `ensure_ready`, `acquire` with nothing healthy, and the warm floor — and the growth path was not one of them. Live at 23:10: a run adopted one surviving pod, grew by one, and rented an RTX A6000 (after walking thirty candidates) while a second healthy pod of its own sat idle in the registry, billing for nothing until the reaper's grace would have expired. `_grow_in_background` now adopts first, and a source-anchored test pins all five provisioning sites so a sixth cannot quietly forget. `seed_region` stays excluded by design: it exists to boot a pod in a NAMED region, and adopting one elsewhere would not fill that region's volume. tests/test_pool_growth_adopts.py (4), four more in tests/test_surya_thread_boundary.py, one in tests/test_runpod_scale_out.py — whose pool fake now stubs adoption, since left real it reads this machine's own registry. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two separate reasons the Digesta re-test still saw wrong caps (12288 and 1360, never the 3072 it sent) and "surya-ocr-2 does not report a context window". Neither is the pod image, which was blamed for both. 1. surya's `chat_completions_batch` runs every item through a ThreadPoolExecutor: with ThreadPoolExecutor(max_workers=max_workers) as executor: return list(executor.map(_process, batch)) so the hook around `_generate_one` executes on an EXECUTOR thread, while `_trace_opts` installed the page's options on the lane thread that submitted it — and that state is thread-local, deliberately, so two lanes cannot attribute one page's logprobs to another. Across that boundary every per-request budget read as absent: `max_tokens` was discarded (v0.3.57 fixed which value wins, not whether ours arrives), and so were `max_continuations` and `context_tokens`. The per-page trace went the same way — attempts, token counts, logprobs and accent repairs were recorded on the executor thread and never read, so `model_attempts` could not report what it exists to report. The item is the one object that crosses, so it is the key. The lane binds its page state to the batch's items before surya dispatches them (hooking `chat_completions_batch`, which still runs on the lane thread), and the executor thread adopts it by `id(item)`. The state is shared by reference, so writes land where the lane reads them. Bindings are released in a finally — id() is reused, and a stale entry would hand a new item another page's state. When a surya has no such entry point, that is said rather than silently leaving the budgets unreachable. 2. An attached engine reads `/v1/models` from the GATEWAY — it talks to `…/v1/ocr/attach/<token>/v1` — and the gateway answered with id, object and owned_by, nothing else. So 0.3.53's publishing of `max_model_len` was invisible on the one path whose arithmetic needs it, the worker correctly reported that nobody names a window, and that read as the pod failing to advertise one. The pod was never asked. The gateway now publishes the window it will serve: a value an upstream reported, else the model's configured ctx. And the pod now serves the MODEL's ctx instead of this install's. A runpod block asking for 18432 got a pod serving 32768, because the pod is handed `engine_config_for_pod`, which reads config.json — the local engine's settings. Publishing one number while serving another is worse than publishing none: resumption stops early, or overruns and takes a 400. The per-model value now wins and the difference is logged. Note for operators: a pod will now serve the per-model `ctx` it was always configured with. For surya on the Digesta orchestrator that is 18432, not the 32768 it had been getting by accident — set ctx higher if the larger window was wanted. A live probe of the upstream's own /v1/models was written and dropped: it built a client per page and re-probed for ever against an upstream that reports nothing. `learn_window()` remains as the hook for a value we do learn. 15 tests: tests/test_surya_thread_boundary.py (7) and tests/test_ocr_window_published.py (8). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reported from the Digesta side as "the pinned pod image boots, but coderai calls its vLLM endpoint with a token the pod rejects — every page 401, every page empty". The 401 was real and the pages were really lost. It was not the image, and not a mismatch between what coderai holds and what the pod wants — the pod answers 200 to its recorded token, before and after. It was the ORDER of two reads. `RunpodBackend.load_model` snapshotted `pool.api_key` into `self._headers`, and `_acquire_endpoint()` — which is what triggers provisioning, and therefore what runs `_adopt_shared_pod` and REPLACES `pool.api_key` — runs after it. A pod for this pool was already running (ours, from before a restart), so the pool adopted its token rather than renting a second A40: 22:27:15 [runpod] adopting pod sncrmkqdnd7c57's bearer token 22:27:15 [runpod] reusing the pod already running … instead of renting another 22:27:15 Inference error: Error code: 401 - Invalid API key. Every request that crossed that moment carried the token from before it. All 64 pages in flight failed; the same lease answered 200 minutes later, which is why it looked like a property of the run rather than of one instant in it. `_auth()` now reads the pool at send time, and the three sites that used the snapshot go through it (chat completions, streaming, openai_endpoint). Serverless keeps the snapshot: it has no pool and no adoption event. The placement candidate also reads the key AFTER acquiring rather than before, since acquiring is the thing that can change it. Behind that, the OCR gateway now retries a 401/403 once against a freshly chosen upstream. A pod only answers to the token it was launched with, so a stale credential is not recoverable downstream — and throwing a page away for a credential that was already correct by the time the error arrived is the expensive half of this bug. A second refusal is returned as itself, not retried forever. Also fixes a regression in 0.3.56: `register_pod` re-stamped `created` on every write, but an adoption re-registers an existing pod — so a pod adopted hourly looked newly born hourly, and the reaper's ledger attribution (which 0.3.56 computes from `created`) would have under-counted its hours. An existing `created` is now preserved. 9 tests in tests/test_pod_token_adoption_race.py, including the ordering pinned against the source so no send site can quietly go back to the snapshot. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Reported from the Digesta side as "the pod ignores max_tokens and uses a fixed 12288 cap, loops, and silently drops dense pages", with a pre-fix pod image blamed and the work blocked on a versioned one. The pod is not involved: 12288 appears nowhere in coderai. It is `settings.SURYA_MAX_TOKENS_FULL_PAGE` in the surya library on the ORCHESTRATOR, and surya/recognition builds every full-page item with `max_tokens=` that value. surya's own client then resolves the budget as max_tokens = item.max_tokens or max_tokens_default so `item.max_tokens` was always truthy and the `max_tokens_default` the trace hook rewrote was never read — by anything. `ocr.surya_max_tokens` and a request's own `max_tokens` were both discarded on the vLLM path, which is the path in use. Every page ran at 12288, and on a 32,768-token window that leaves room for about two segments once the ~4,000-token image prefill is counted, so a dense page reached the ceiling mid-answer. That is the "loops and drops pages" half of the same report, and it is why the benchmark numbers taken at "max_tokens=8000/4000/2000" were all the same run. The hook now applies the cap to the item as well, which is the value that decides, and says so once per worker rather than once per page: `[surya] page budget 3072 tokens (overriding surya's own 12288)`. An item that refuses the assignment says that too, instead of leaving the cap quietly unreachable. With no cap configured surya's own budget is untouched — raising it costs tokens on every page. This was already on the list as "surya's own per-page item.max_tokens overrides the request cap — flagged, undecided". The decision: an explicit cap is an instruction and a per-page estimate is a guess, so the instruction wins. What was NOT true in the report, with evidence: the pod image carries coderai 0.3.53 and `git log 42356742..HEAD -- codai/ocr codai/backends codai/api/text.py codai/api/ocr.py codai/models/manager.py` is empty, so it contains every OCR fix; the pod self-reported `coderai=0.3.53 git=42356742` at /boot. A versioned pod tag is still worth having — a cached `:latest` cannot be told apart from a current one by looking — but it was not the blocker. 9 tests in tests/test_surya_item_cap.py. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A qwen38 pool pod (A40, $0.59/hr) was found running at 21:30, outside its 08:00-20:00 schedule window, hours after it should have been down. Its config was correct and the pool said so every pass — "outside its warm-pod window — warm floor 0" — while the pod billed on. Two mechanisms each did their job and left a gap between them. The 21:17 unattended upgrade restarted coderai; the pod survived it. With a warm floor of 0 nothing provisioned, and adoption (`find_shared_pod`) only runs when a pool actually needs a pod — so no pool tracked it, and no pool applied `idle_timeout_s` to it. Meanwhile the reaper skips everything in `_known_pod_ids()`, and the dead process's registry entry was still in there, because `registered_pod_ids()` returned every key in the file regardless of who wrote it or whether that process still existed. The pod was simultaneously unmanaged and shielded. Nothing in the registry could tell the two apart: `at` was written once at registration and never updated. `at` is now a heartbeat. `touch_registered_pods()` refreshes this process's own entries from the scaler loop (every fourth 15 s pass), `live_registered_pod_ids()` counts ours unconditionally and a sibling's only while its heartbeat is fresh, and `_known_pod_ids()` uses that. An entry whose owner has been silent for REAP_GRACE_SECONDS stops shielding its pod, and the reaper treats it like any other leftover. The grace is 1200 s against a 60 s heartbeat, so a sibling between passes is never at risk. The trade this accepts: a process still running but whose maintenance loop has died stops heartbeating, and a sibling will reap its pods after the grace. Twenty minutes of silence from the loop that is supposed to manage a pod is itself the failure. Also: a pod the reaper kills is now recorded in the ledger, computed from a new `created` field (distinct from the heartbeat, which moves). Those hours were invisible before — only `_terminate` recorded spend, and that only runs for a pod a pool was tracking, which is exactly not this case. And the reaper unregisters what it terminates, so it cannot try twice. The pod itself was terminated by hand and its stale entry removed; the account is back to zero pods. 11 tests in tests/test_runpod_registry_heartbeat.py, including the whole bug in one assertion: an orphan id is in `registered_pod_ids()` and not in `_known_pod_ids()`. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod ties a network volume to a single data centre, and a pod elsewhere cannot attach it. So `network_volume_id` made `_data_centers()` answer with the volume's region and throw the `data_centers` fallback list away — which is correct for one volume and still forced a choice nobody should make: rent wherever a card is free and re-download the weights onto every cold pod, or keep the storage and wait for capacity in one region. On a contended card that is a lease that fills versus one that waits. `network_volumes` is a map {REGION: volume_id} on the account block and on a model's. The region is chosen first — the scaler already paired every candidate card with every allowed region — and the volume now follows the region that candidate was ranked for. A region with no volume in the map is dropped from the list with a line saying which, because a pod rented there would silently have no storage and boot minutes slower; if nothing is left, provisioning refuses and names both sets rather than placing the data somewhere the operator never listed. A model's map replaces the account's whole rather than merging: listing regions is a statement about where that model's data may live. `network_volume_id` keeps working unchanged and a one-entry map behaves identically to it, which is how this ships without touching any install. The global volume was supposed to make all of this unnecessary and cannot: the type is not creatable through the public API (POST /v2/network-volumes takes only STANDARD|HIGH_PERFORMANCE and demands a dataCenter), and object storage — no locking, no atomic rename, eventual consistency — is wrong for anything but read-mostly weights. `global_volume_id` stays supported; it stops being the plan. Seeding a new region needs no new pod shape: POST /admin/api/runpod/seed-volumes boots one ordinary pod per region with that region's volume attached, which IS the download because the caches already point there, then terminates it. Sequential, admin-only, under the usual spend caps. The venv needs nothing — venv_on_volume already builds it under a lock with .ready written last, so the first pod in a region builds it and every pod after finds it, which is what makes the dependency-free slim image usable in every region. Also here: _data_centers() upper-cases region ids, since the map's key matching the candidate's region is the whole mechanism; pools_schedule_status carries the map; both admin surfaces gained a "Per-region volumes" field. Everything is offline-tested only — 25 new tests in tests/test_runpod_regional_volumes.py. The first multi-region install is the real proof, and the first thing to check there is that RunPod's dataCenterId and the map's key are spelled the same. test_runpod_placement's region test counted three copies of `sel.get("_dc") or self._data_center()`; the region is now resolved once into a local because the volume lookup needs it too, so it checks that one resolution and its uses instead of duplicates of it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Per model, like every other pod setting, and bounded so that it can only ever NARROW what the operator configured. A pool with a preference and allow_other_gpus: true takes another card when the preferred one is not free -- usually right, occasionally exactly wrong. Observed 2026-10-13: gpu_types [NVIDIA A40] fell back to an L40S at $1.09/hr against the A40's $0.59, correct per the configuration and useless to a caller measuring throughput, since a number taken on a different card is not comparable with the one it is being compared to. The only way to get the A40 was to edit the model's configuration -- changing it for everybody, permanently, for one request. POST /v1/runpod/scale now takes `gpu_types` (list or comma string, in preference order) and `allow_other_gpus`. The rule is one-directional: - a card the operator did not list is DROPPED and named in `notes`; - narrowing to only cards the operator forbids is a 400 that names what IS allowed -- never a silent fall back to the default, which would be the opposite of what the caller asked for; - allow_other_gpus may be turned OFF by a request, never ON; - max_hourly_usd, min_vram_gb, cloud_types and max_pods are the operator's alone and are not reachable from a request at all. Held on the LEASE, not the pool: an "A40 only" cannot outlive the 30 minutes it asked for, releasing the lease releases its constraints, and provisioning reads the narrowing per pod so a lapsed lease stops constraining the very next one. There is deliberately no path from a request to a model's configuration. Reported back in the answer and on /v1/runpod/status as `client_gpu_narrow`, because a constraint silently dropped is worse than one refused -- the caller would read its numbers as having come from the card it asked for -- and a pod on an unexpected card should have a visible cause, not only a bill. Tests: 32 new, including that the cost ceiling and the VRAM floor still refuse a pinned card, that the operator's limits are not even mentioned in the narrowing code, and that None (not False) is the default for both fields -- False would silently turn every caller's fallback off. Also fixed a source-window test that anchored on `_rank_gpus`'s full argument list and broke the moment a parameter was added; it now anchors on the assignment, since the arguments are not what it is testing. Full suite: 25 failures, all pre-existing and identical to baseline. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/models now carries `max_model_len`, under vLLM's own name so a client written against a vLLM server reads it unchanged. It is published because a client cannot otherwise plan: anything that sends a growing conversation -- a continued generation, a resumed OCR page, a long tool loop -- has to know when the window runs out, and the only alternative is finding out from a 400 after the work is paid for. coderai's own OCR page resumption was the first casualty, and the way it failed is the point. The 0.3.50 clamp probed /v1/models for this field; coderai did not publish it, and the surya worker talks to coderai rather than to raw vLLM. So the clamp was INERT from the day it shipped, and said nothing: the log printed "? of the context window left" and every resumption ran with no ceiling. Two further details made it invisible -- `max_model_len` is not in the OpenAI model schema, so the client parks it as an EXTRA field and reading attributes alone found nothing; and the probe only logged on an exception, so a server that answered WITHOUT the field was indistinguishable from one that worked. - The number comes from the LOADED backend first, config second. The two disagree and only the served one is true: `ctx: 18432` is configured on one install while `--max-model-len 32768` is what the pods actually serve, so a client trusting config would plan around a window that does not exist. - VllmBackend.context_length() asks its own upstream /v1/models, cached: a served model cannot change shape under us. - Resolution handles BOTH manager shapes. list_models is a method of the MULTI manager, which has no .backend of its own -- the per-model manager does -- so the first version of this could never have fired either. Found by writing the test, not by reading the code. - Several spellings read (max_model_len/n_ctx/ctx/context_length, plus the vllm/runpod/llamacpp blocks), because several backends own this number. - None means UNKNOWN, never unlimited, and the worker now says out loud when a server named no window, naming ocr.surya_context_tokens as the answer. Also, so that "which build is this pod running" stops being an inference: the pod boot log prints its coderai version and the short git sha of the app layer (new GIT_SHA build arg). A mutable `:latest` is a fine way to publish the current release, but it leaves two pods started minutes apart looking identical in an account listing -- and on 2026-10-13 that cost two pods being restarted that already had the fix, because nothing in their logs named their version. Tests: 27 new. They pin what the resolver PREFERS (live over config), what it tolerates (a stray string, a list-wrapped number, a non-dict backend block), that another model's loaded backend is not consulted, that the listing actually calls it, and that a server naming no window produces a message rather than a silence. Full suite: 25 failures, all pre-existing and identical to baseline. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Found while writing the Digesta dev the answer to "which field names does 0.3.51 actually honour" -- a question that could not be answered honestly because of these two. /v1/ocr/batch never accepted max_tokens or max_continuations. A batch is how a caller sends real work, so the budgets were accepted only on the path nobody benchmarks: every claim since 0.3.49 that "/v1/ocr and /v1/ocr/batch take these per request" was half wrong, docs included. FastAPI drops an unknown form field in silence, which is the same trap this feature existed to close -- the caller sets max_tokens on every batch and is never told it did nothing. The dev's measurements at "max_tokens=8000/4000/2000" were therefore all taken at the CONFIGURED value, whatever that was at the time. And /v1/ocr's route decorator was attached to the _page_opts HELPER rather than to create_ocr, because 0.3.49 inserted the helper between the decorator and its function. FastAPI registered the helper as the handler, so the endpoint existed, answered, and was not the OCR endpoint. Nothing failed at import; no test looked at the route table. 16 tests, 12 of which fail against the pre-fix code: the route resolves to create_ocr, _page_opts is not registered as a handler, both paths take both budgets, each defaults to None (0 is meaningful for both -- no cap override, resumption off), negatives clamp, and the batch path is asserted to FORWARD them, since a signature that accepts a field proves nothing about using it. Checked against the route table and signatures, so they need no engine or GPU. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The Digesta dev's 400 on page resumption, and a bug behind it that was worse than the 400. The vLLM backend built its outgoing request from an ALLOW-LIST of fields it recognised. Resumption asks for two beyond an ordinary request -- continue_final_message, so the model carries on the exact string, and add_generation_prompt: false, so no new turn is opened after the prefilled text -- and neither was ever forwarded, by this or any previous version. What reached vLLM was a conversation ending in an assistant turn with add_generation_prompt left at its default TRUE. surya-ocr-2's chat template then appends a fresh <|im_start|>assistant after the text so far, so the model was asked to START A NEW ANSWER about the same image. It did. And the failure hid behind the guard built to catch it: the new answer repeated the beginning of the page, the no-progress check saw text it already had, logged "came back repeating itself" and discarded the segment. 253 of 262 resumptions on the live install -- read for a day as a model that degenerates on dense pages, when it was a model being asked the wrong question, answering correctly, and having the answer thrown away. It also explains the dev's table exactly: at max_tokens=8000 resumption is barely exercised and OCR holds 93%, while at 4000 and 2000 nearly every page resumes, each resumption is a full generation that produces a restart and is then discarded, and the batch times out. Fixed by forwarding BY EXCLUSION, which is the only form that stays correct: coderai names the fields it CONSUMES (_CHAT_CONSUMED_FIELDS) and everything else goes to the server, which is the thing entitled to decide what it supports. An allow-list means every field a server gains after the list was written is born broken -- the same reasoning as forwarding page diagnostics by exclusion. It reaches only a backend whose signature accepts it, so a backend that cannot carry an unknown field is unaffected rather than broken, and model/messages/stream are never overridable this way. Also: a 4xx explains itself, if nobody discards it. raise_for_status() reports "400 Client Error: Bad Request for url: ..." -- the symptom and nothing else -- while vLLM puts the reason in the BODY. A whole diagnosis cycle was spent guessing at a 400 that had already been explained and thrown away. The error now carries the server's own words. Made tolerant of a response object that reports no status_code, rather than turning an odd shape into an AttributeError inside the request path. Two findings for the operator, not code: the pods serve max-model-len 32768 while the model entry asks for ctx 18432, so per-model vllm settings still are not reaching the pod; and this fix must be in the POD image, because the proxy that dropped the fields runs there. Tests: 16 new (flags reach the payload, an unknown field is forwarded, no passthrough changes nothing, structure not overridable, exclusion at the API layer, signature-probed delivery, the error carries the server's words). Full suite: 25 failures, all pre-existing and identical to the stashed baseline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two defects that a throughput collapse on a live install made visible. With surya_max_tokens raised to 12288, 262 resumption segments spent 1.6M generated tokens in 80 minutes, 122 of them running the full cap dry, while 253 of those 262 ended in "came back repeating itself". Resumption re-sends the text so far as INPUT, so the model's context window -- not the continuation budget -- is the real ceiling. The budget did not know that: at a 12288-token cap, the third segment of a page asked for 12288 more on top of ~28,500 already in a 32,768-token window and was rejected outright. The page was then reported as one that could not be resumed AT ALL, discarding text already paid for. - surya_context_tokens, or 0 to read the server's own max_model_len from /v1/models. Configuration wins: an operator may serve a shorter window than the checkpoint advertises. - Each segment asks for the smaller of the cap and the room left, measured from what the server said the prompt cost (usage.prompt_tokens), not guessed; the guess covers the first resumption only. - Out of room stops, KEEPS what it has, and says so -- naming the fix that helps, which is a SMALLER max_tokens, not a bigger budget. - A refusal for length is not retried: the fallback resumption adds a user turn, so it is longer than the request just refused. - Not knowing the window changes nothing and guesses nothing. The second defect: a reply that trips surya's repeat detector hit the cap because it is LOOPING, not because the page is long. Resuming it can only extend the loop, and surya re-runs the whole page anyway -- so one page on that path cost four full generations. It is no longer resumed, and it is not marked truncated either: that is surya's retry to make, and "truncated" would send the caller looking for a bigger budget it cannot use. A fixed token cap is also an accidental repetition circuit-breaker, which is why raising it slowed the LOOPING pages by the ratio of the caps rather than every page. Documented in docs/ocr.md with the window arithmetic (2048 -> ~13 segments, 12288 -> 2). Tests: 8 new (window clamp, stop-one-segment-early, length refusal not retried, unknown window, configuration over server, a cap below the 256-token floor is never mistaken for a full window, a looping reply is not resumed, a genuinely long page still is). 62 pass in the surya/blank-page suites; the floor-versus-cap bug was caught by the existing 44. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/ocr took engine, dpi, lang, structured, schema and detect. FastAPI drops unknown form fields SILENTLY, so a caller sending max_tokens=8000 on every request was discarded without a word and every page ran at the configured value. Both budgets are now real request parameters. One global cap cannot serve a public deployment: a four-page contract and a fifty-page judgment need different budgets and only the caller knows which it is sending. The values travel WITH the page rather than with the engine load, because a pool is built once and then serves every caller, and they are applied thread-locally in the worker, because several pages are in flight in one process and the hooks they feed are module-level — a per-request value in a module cell would be overwritten by the next page to arrive. Carried on a ContextVar rather than widening `recognize_image`, which is the one thing every engine has, duck-typed ones included. Omitting a parameter means "use the configured value", not zero: 0 is meaningful for both (no cap override, and resumption off). surya_max_continuations 2 -> 8. It is PER PAGE, not per document: a fifty-page document is fifty independent pages each with its own budget, so what the number covers is the densest SINGLE page. The real ceiling is not the number but the model's context — each resumption re-sends the text so far as INPUT, so with surya-ocr-2's 32,768-token window and ~4,000 of it on the page image, about thirteen 2,048-token segments exhaust it whatever is configured, and raising surya_max_tokens lowers that count in proportion. A generous budget is only safe if a page that stops making progress stops costing, so resumption now ends when a segment comes back repeating itself or returns the text already in hand. The second case is also how a server that IGNORES continue_final_message is caught: a model that starts the page over returns what we already have, and without the check a budget of 8 would buy eight full generations of the same page. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two things, from one sample document. RESUMING A CUT-OFF PAGE. Raising surya_max_tokens only moves the cliff: a dense enough page overflows any fixed number, and incomplete block markup parses to a fragment or to nothing. `ocr.surya_max_continuations` (default 2) resumes instead -- the text so far goes back as a prefilled assistant turn with vLLM's `continue_final_message`, so the model carries on the exact string rather than answering afresh about the same image. A server rejecting the flag (llama.cpp) gets a plain "continue" instruction; a server rejecting both leaves the page partial and honestly marked, since a page that cannot be finished is still a page. Logprobs are concatenated in step with the text, because surya_repair_accents rebuilds characters from the model's own per-token bytes and a later segment paired with earlier bytes would corrupt exactly the accented characters the repair exists to fix; if they cannot be aligned the repair is given up on rather than fed a mismatch. Temperature 0 -- the opposite of the retry path, which raises it to break a repetition. A page still unfinished after its budget carries "complete": false so a caller can refuse a partial rather than store it as read. On by default: returning a page KNOWN to be incomplete is a correctness failure, not a cost trade-off. IS THERE ANYTHING ON THE PAGE AT ALL. A four-page court PDF came back with 47 characters and was reported as an OCR failure. It was not. Every page was a ~900-byte CCITT Group 4 image -- what an EMPTY page costs, a couple of bytes per all-white row -- carrying 0.13% ink, all of it a diagonal scan edge and a diagonal "copia non ufficiale" watermark. The 47 characters WERE the watermark. The engine had read the page perfectly. "The model could not read it", "the reply was cut off" and "the page is blank" were indistinguishable in a short result, and they lead to completely different places -- a better engine, a bigger budget, or a document fixed upstream. Any page whose text is shorter than ocr.blank_page_max_chars (200) now carries the fraction of its pixels that are not background, from an image already in memory, with blank_pages listed at the top of the response. The opposite verdict is as useful as the blank one: ink above the floor says there IS something there, so the engine's failure is real and worth chasing. It applies to every engine, because the measurement is of the image rather than of the model, and it is a diagnostic only -- nothing is skipped, refused or altered because of it. Verified against the sample: all four pages 0.13-0.33% ink, correctly called blank against the 0.5% floor. Two bugs found while testing: `rounds` counted resumption ATTEMPTS rather than successes, so a failed resume would have reported the page as whole; and three assertions matched a source window that moved, one of them for a sentence that is wrapped across f-string lines and never appears contiguously. Co-Authored-By:
-