- 13 Oct, 2026 25 commits
-
-
Stefy Lanza (nextime / spora ) authoredRaised from the Digesta side: the one high-leverage way to make surya fall through less often is to make its repeat-detector and retry tunable. Half right, and the other half turned out to be the bigger lever. Checking the installed surya rather than the description: `_should_retry` stops at `max_retries` and RETURNS the looping text — surya does not return empty. The empty page comes later, because looping markup parses to no blocks. So the symptom is real and the mechanism is not what it looked like. **The lever is that surya never said it looped.** chandra has reported `repeat_detected` since it shipped, with the text returned anyway because a looping transcription usually has a correct beginning. surya reported nothing, so a short or empty page was indistinguishable from an unreadable scan — and those need opposite handling: one is a cascade candidate, the other means the document is bad. surya now flags it at the moment the verdict is made (the retry hook's last call for the page), with `repeat_chars`, the length of the reply that was still looping: "looped and produced four thousand characters" has a usable beginning, "looped and produced nothing" does not. **And the detector is now retunable** — `ocr.surya_repeat_max_repeats`, `surya_repeat_window`, `surya_repeat_scaling`, 0 on each leaving surya's own (4, 500, 3.0). Hooked like `_should_retry` already was, since the vLLM backend passes neither. A false positive is expensive: the page is re-run several times at rising temperature and then parsed from a reply nobody wanted, usually yielding less text than the first one had. Operator settings, deliberately not per-request — they describe the corpus (tables, dotted leaders and form fields repeat short strings legitimately), not the page. An explicit argument from surya's own code still wins, so its `cut_from_end` call keeps working. 11 tests in tests/test_surya_loop_reported.py, including that a loop surya then rescues is not recorded as a verdict, and that the thresholds must be in place BEFORE the hook is installed (it patches once per process). Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
-
Stefy Lanza (nextime / spora ) authored
Two gaps I left open last night, both about the ledger telling the truth. **A pod that went away without a pool noticing was never billed to anyone.** A pool closes its own pods' bills — `_terminate` for one it retires, `forget` for one RunPod says is gone — and neither runs for a pod only the REGISTRY knew about: one orphaned by a restart, or terminated from the console or straight through the RunPod API. Observed: pod lgdgfs749chkib, killed by hand, never appeared in the ledger while two pods that went through coderai did. `close_bill_from_registry()` now computes the hours from the registry's own `created` — the only record of when a pod started once no handle exists — and records them against the pool it belonged to. Both reaper paths use it, so the "terminated as stale" and "vanished from the account" cases cannot drift apart, and a pod a pool still holds is deliberately skipped: `forget` ledgers that one, and counting it here as well would double-count every pod RunPod reclaims. **`record_spend` dropped the `reaped` flag it was being handed.** It keeps a deliberate allow-list, which is right — the file is persisted and pruned for every pod — but the list has to carry what distinguishes one kind of charge from another. A pod the reaper killed looked in the file exactly like one a pool retired on purpose. `reaped` and `reason` are kept now. Writing the test for the first fix found a third thing, and it was mine: v0.3.56 made a stale registry entry stop counting as "known" so the reaper could kill the pod it named — but the reconcile was only ever asked about known ids, so that same change stopped stale entries being RECONCILED. An entry for a pod that no longer exists was then never pruned and its hours never attributed. The reaper now reconciles against the whole registry; `known` keeps its one job, which is deciding what is shielded from reaping. 9 tests in tests/test_ledger_out_of_band.py, including the double-count guard and the pruning regression. test_runpod_client_scale's source anchor moves with the change and says why. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Resumption was surya's alone. chandra and olmOCR stopped at their configured cap and returned whatever they had — and what they had is the worse half of the problem: chandra emits HTML, so a cut-off reply is unclosed markup that parses to a fragment of the page or to nothing, and olmOCR's reply opens with YAML front matter, so one cut off before the body closes parses to metadata and NO TEXT AT ALL. In the finished page both are indistinguishable from a model that could not read it, which is the confusion this whole area exists to remove. The rules now live once, in codai/ocr/resume.py, and they are the ones surya learned the hard way rather than a fresh guess: * both vLLM flags or neither — `continue_final_message` AND `add_generation_prompt: false`. With the second left at its default the chat template appends a fresh assistant turn and the model answers again about the same image, which reads exactly like a model repeating itself and did for a whole day; * never resume a reply that is already looping: it did not run out of room, it ran out of page, and resuming can only extend the loop; * the context window is the ceiling, not the budget. Resumption re-sends everything so far as INPUT, so the room shrinks by what was just produced, and a segment that no longer fits means the cap is too BIG, not too small; * progress has to be real: a segment identical to the previous one, or one that comes back looping, ends the loop; * `finish_reason` is authoritative both ways — "stop" with a full budget is a complete page and must not be resumed, which a token count alone cannot tell. Both engines' ask paths now return a `resume.Reply` carrying the finish reason and the usage counts (they were discarding both), take a `prefix` that goes back as the prefilled assistant turn, and report `model_attempts` in the same shape surya produces — including the budget in force, so a caller can tell a request that landed from one that was ignored. Being in-process, they have no thread boundary to cross: `page_options()` reaches them directly. New settings, in config.json, the admin API and the settings page: chandra_max_continuations / chandra_context_tokens and olmocr_max_continuations / olmocr_context_tokens. The UI parses them as numbers rather than falling back on `||`, because 0 is a real setting here (resumption off) and not "use the default". chandra's `detect_repeat_token` moves to being shared rather than copied: it is a port of chandra's upstream util and answers "has this reply fallen into a repetition loop", which is not chandra-specific. 42 tests: tests/test_ocr_resume_shared.py (28, the rules) and tests/test_ocr_resume_engines.py (14, each engine's own ask path). The existing chandra/olmocr/confidence/pod-TLS harnesses were updated to the new ask contract — it changed deliberately, and the stubs say why. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Adoption matched on the pool and a health probe, and the registry recorded nothing about what the pod was RUNNING. Two silent mistakes followed from that: * **the image.** The same model on a pinned tag and on `:latest` is different code. A pool configured for one could inherit a pod running the other, and nothing in the system could tell — which is precisely the confusion that cost a whole evening today: an OCR result was attributed to code that was not what the pod was running. * **the card.** A pod rented as a FALLBACK is on a card the configuration accepts only when its preferred one is unavailable. Inheriting one would serve a lease that asked for an A40 *and nothing else* (v0.3.54) from an RTX A6000 — the exact comparison that feature exists to protect. Live tonight the pool had both: an adopted A40 and a rented A6000 fallback for one model. So `register_pod` records the image and the card at creation (and keeps them across a re-registration, since an adopting owner does not know them), and `find_shared_pod` takes `image` and `gpus` and refuses anything that does not match, naming the pod and the reason. The one caller passes the image from the plan it would launch and the cards from `narrow_gpu_choice` — so a client lease's narrowing constrains adoption exactly as it constrains renting. An entry recording neither field is refused rather than trusted: refusing costs one extra pod once and the old one is reaped, while trusting costs a silent answer from the wrong code. A caller that names neither constraint keeps the old behaviour, so this adds constraints without adding requirements. Also here: an adopted pod is now billed from when it was CREATED, not from when it was taken over. `_terminate` computes the spend from `started_at`, which adoption set to "now" — so a pod created at 22:27 and adopted at 23:10 was ledgered for 16 minutes of an hour's life, and the rest was charged to nobody. The registry already records `created`, and only cost and uptime reporting read `started_at`. 11 tests in tests/test_adoption_matches_image_and_card.py, one more in tests/test_pod_token_adoption_race.py. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two reports from the Digesta side, both of which my own code caused. **"resumption is disabled server-side (ocr.surya_max_continuations = 0)".** It is not, and never was. Both config files leave the key unset, the dataclass default is 8, and the running process resolves 8 for either candidate config dir — checked inside the container through the app's own loader. The number came from this, in ocr_worker: elif capped and not resumed: info["why"] = (f"the reply stopped at the {cap}-token cap and " f"resumption is off (ocr.surya_max_continuations = 0) …") a hardcoded string in a branch that is reached four different ways: budget zero, a generation already repeating itself, no room left in the context window, or a reply that could not be resumed at all. It asserted a configuration value it never read, and it was believed — twice — and acted on. A diagnostic that names the wrong cause is worse than one that says nothing. So the reason is now recorded where it is known (`_no_resume`) and reported as itself. The budget actually in force goes into `model_attempts` as `max_continuations`, because printing the cap but not the budget is why a caller could not tell a request that landed from one that was ignored. The old message survives for the one case where it was true, and says it is about this page rather than the server. **Growth rented a card beside a pod it already owned.** Adoption was attempted in three places — `ensure_ready`, `acquire` with nothing healthy, and the warm floor — and the growth path was not one of them. Live at 23:10: a run adopted one surviving pod, grew by one, and rented an RTX A6000 (after walking thirty candidates) while a second healthy pod of its own sat idle in the registry, billing for nothing until the reaper's grace would have expired. `_grow_in_background` now adopts first, and a source-anchored test pins all five provisioning sites so a sixth cannot quietly forget. `seed_region` stays excluded by design: it exists to boot a pod in a NAMED region, and adopting one elsewhere would not fill that region's volume. tests/test_pool_growth_adopts.py (4), four more in tests/test_surya_thread_boundary.py, one in tests/test_runpod_scale_out.py — whose pool fake now stubs adoption, since left real it reads this machine's own registry. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two separate reasons the Digesta re-test still saw wrong caps (12288 and 1360, never the 3072 it sent) and "surya-ocr-2 does not report a context window". Neither is the pod image, which was blamed for both. 1. surya's `chat_completions_batch` runs every item through a ThreadPoolExecutor: with ThreadPoolExecutor(max_workers=max_workers) as executor: return list(executor.map(_process, batch)) so the hook around `_generate_one` executes on an EXECUTOR thread, while `_trace_opts` installed the page's options on the lane thread that submitted it — and that state is thread-local, deliberately, so two lanes cannot attribute one page's logprobs to another. Across that boundary every per-request budget read as absent: `max_tokens` was discarded (v0.3.57 fixed which value wins, not whether ours arrives), and so were `max_continuations` and `context_tokens`. The per-page trace went the same way — attempts, token counts, logprobs and accent repairs were recorded on the executor thread and never read, so `model_attempts` could not report what it exists to report. The item is the one object that crosses, so it is the key. The lane binds its page state to the batch's items before surya dispatches them (hooking `chat_completions_batch`, which still runs on the lane thread), and the executor thread adopts it by `id(item)`. The state is shared by reference, so writes land where the lane reads them. Bindings are released in a finally — id() is reused, and a stale entry would hand a new item another page's state. When a surya has no such entry point, that is said rather than silently leaving the budgets unreachable. 2. An attached engine reads `/v1/models` from the GATEWAY — it talks to `…/v1/ocr/attach/<token>/v1` — and the gateway answered with id, object and owned_by, nothing else. So 0.3.53's publishing of `max_model_len` was invisible on the one path whose arithmetic needs it, the worker correctly reported that nobody names a window, and that read as the pod failing to advertise one. The pod was never asked. The gateway now publishes the window it will serve: a value an upstream reported, else the model's configured ctx. And the pod now serves the MODEL's ctx instead of this install's. A runpod block asking for 18432 got a pod serving 32768, because the pod is handed `engine_config_for_pod`, which reads config.json — the local engine's settings. Publishing one number while serving another is worse than publishing none: resumption stops early, or overruns and takes a 400. The per-model value now wins and the difference is logged. Note for operators: a pod will now serve the per-model `ctx` it was always configured with. For surya on the Digesta orchestrator that is 18432, not the 32768 it had been getting by accident — set ctx higher if the larger window was wanted. A live probe of the upstream's own /v1/models was written and dropped: it built a client per page and re-probed for ever against an upstream that reports nothing. `learn_window()` remains as the hook for a value we do learn. 15 tests: tests/test_surya_thread_boundary.py (7) and tests/test_ocr_window_published.py (8). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reported from the Digesta side as "the pinned pod image boots, but coderai calls its vLLM endpoint with a token the pod rejects — every page 401, every page empty". The 401 was real and the pages were really lost. It was not the image, and not a mismatch between what coderai holds and what the pod wants — the pod answers 200 to its recorded token, before and after. It was the ORDER of two reads. `RunpodBackend.load_model` snapshotted `pool.api_key` into `self._headers`, and `_acquire_endpoint()` — which is what triggers provisioning, and therefore what runs `_adopt_shared_pod` and REPLACES `pool.api_key` — runs after it. A pod for this pool was already running (ours, from before a restart), so the pool adopted its token rather than renting a second A40: 22:27:15 [runpod] adopting pod sncrmkqdnd7c57's bearer token 22:27:15 [runpod] reusing the pod already running … instead of renting another 22:27:15 Inference error: Error code: 401 - Invalid API key. Every request that crossed that moment carried the token from before it. All 64 pages in flight failed; the same lease answered 200 minutes later, which is why it looked like a property of the run rather than of one instant in it. `_auth()` now reads the pool at send time, and the three sites that used the snapshot go through it (chat completions, streaming, openai_endpoint). Serverless keeps the snapshot: it has no pool and no adoption event. The placement candidate also reads the key AFTER acquiring rather than before, since acquiring is the thing that can change it. Behind that, the OCR gateway now retries a 401/403 once against a freshly chosen upstream. A pod only answers to the token it was launched with, so a stale credential is not recoverable downstream — and throwing a page away for a credential that was already correct by the time the error arrived is the expensive half of this bug. A second refusal is returned as itself, not retried forever. Also fixes a regression in 0.3.56: `register_pod` re-stamped `created` on every write, but an adoption re-registers an existing pod — so a pod adopted hourly looked newly born hourly, and the reaper's ledger attribution (which 0.3.56 computes from `created`) would have under-counted its hours. An existing `created` is now preserved. 9 tests in tests/test_pod_token_adoption_race.py, including the ordering pinned against the source so no send site can quietly go back to the snapshot. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Reported from the Digesta side as "the pod ignores max_tokens and uses a fixed 12288 cap, loops, and silently drops dense pages", with a pre-fix pod image blamed and the work blocked on a versioned one. The pod is not involved: 12288 appears nowhere in coderai. It is `settings.SURYA_MAX_TOKENS_FULL_PAGE` in the surya library on the ORCHESTRATOR, and surya/recognition builds every full-page item with `max_tokens=` that value. surya's own client then resolves the budget as max_tokens = item.max_tokens or max_tokens_default so `item.max_tokens` was always truthy and the `max_tokens_default` the trace hook rewrote was never read — by anything. `ocr.surya_max_tokens` and a request's own `max_tokens` were both discarded on the vLLM path, which is the path in use. Every page ran at 12288, and on a 32,768-token window that leaves room for about two segments once the ~4,000-token image prefill is counted, so a dense page reached the ceiling mid-answer. That is the "loops and drops pages" half of the same report, and it is why the benchmark numbers taken at "max_tokens=8000/4000/2000" were all the same run. The hook now applies the cap to the item as well, which is the value that decides, and says so once per worker rather than once per page: `[surya] page budget 3072 tokens (overriding surya's own 12288)`. An item that refuses the assignment says that too, instead of leaving the cap quietly unreachable. With no cap configured surya's own budget is untouched — raising it costs tokens on every page. This was already on the list as "surya's own per-page item.max_tokens overrides the request cap — flagged, undecided". The decision: an explicit cap is an instruction and a per-page estimate is a guess, so the instruction wins. What was NOT true in the report, with evidence: the pod image carries coderai 0.3.53 and `git log 42356742..HEAD -- codai/ocr codai/backends codai/api/text.py codai/api/ocr.py codai/models/manager.py` is empty, so it contains every OCR fix; the pod self-reported `coderai=0.3.53 git=42356742` at /boot. A versioned pod tag is still worth having — a cached `:latest` cannot be told apart from a current one by looking — but it was not the blocker. 9 tests in tests/test_surya_item_cap.py. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A qwen38 pool pod (A40, $0.59/hr) was found running at 21:30, outside its 08:00-20:00 schedule window, hours after it should have been down. Its config was correct and the pool said so every pass — "outside its warm-pod window — warm floor 0" — while the pod billed on. Two mechanisms each did their job and left a gap between them. The 21:17 unattended upgrade restarted coderai; the pod survived it. With a warm floor of 0 nothing provisioned, and adoption (`find_shared_pod`) only runs when a pool actually needs a pod — so no pool tracked it, and no pool applied `idle_timeout_s` to it. Meanwhile the reaper skips everything in `_known_pod_ids()`, and the dead process's registry entry was still in there, because `registered_pod_ids()` returned every key in the file regardless of who wrote it or whether that process still existed. The pod was simultaneously unmanaged and shielded. Nothing in the registry could tell the two apart: `at` was written once at registration and never updated. `at` is now a heartbeat. `touch_registered_pods()` refreshes this process's own entries from the scaler loop (every fourth 15 s pass), `live_registered_pod_ids()` counts ours unconditionally and a sibling's only while its heartbeat is fresh, and `_known_pod_ids()` uses that. An entry whose owner has been silent for REAP_GRACE_SECONDS stops shielding its pod, and the reaper treats it like any other leftover. The grace is 1200 s against a 60 s heartbeat, so a sibling between passes is never at risk. The trade this accepts: a process still running but whose maintenance loop has died stops heartbeating, and a sibling will reap its pods after the grace. Twenty minutes of silence from the loop that is supposed to manage a pod is itself the failure. Also: a pod the reaper kills is now recorded in the ledger, computed from a new `created` field (distinct from the heartbeat, which moves). Those hours were invisible before — only `_terminate` recorded spend, and that only runs for a pod a pool was tracking, which is exactly not this case. And the reaper unregisters what it terminates, so it cannot try twice. The pod itself was terminated by hand and its stale entry removed; the account is back to zero pods. 11 tests in tests/test_runpod_registry_heartbeat.py, including the whole bug in one assertion: an orphan id is in `registered_pod_ids()` and not in `_known_pod_ids()`. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod ties a network volume to a single data centre, and a pod elsewhere cannot attach it. So `network_volume_id` made `_data_centers()` answer with the volume's region and throw the `data_centers` fallback list away — which is correct for one volume and still forced a choice nobody should make: rent wherever a card is free and re-download the weights onto every cold pod, or keep the storage and wait for capacity in one region. On a contended card that is a lease that fills versus one that waits. `network_volumes` is a map {REGION: volume_id} on the account block and on a model's. The region is chosen first — the scaler already paired every candidate card with every allowed region — and the volume now follows the region that candidate was ranked for. A region with no volume in the map is dropped from the list with a line saying which, because a pod rented there would silently have no storage and boot minutes slower; if nothing is left, provisioning refuses and names both sets rather than placing the data somewhere the operator never listed. A model's map replaces the account's whole rather than merging: listing regions is a statement about where that model's data may live. `network_volume_id` keeps working unchanged and a one-entry map behaves identically to it, which is how this ships without touching any install. The global volume was supposed to make all of this unnecessary and cannot: the type is not creatable through the public API (POST /v2/network-volumes takes only STANDARD|HIGH_PERFORMANCE and demands a dataCenter), and object storage — no locking, no atomic rename, eventual consistency — is wrong for anything but read-mostly weights. `global_volume_id` stays supported; it stops being the plan. Seeding a new region needs no new pod shape: POST /admin/api/runpod/seed-volumes boots one ordinary pod per region with that region's volume attached, which IS the download because the caches already point there, then terminates it. Sequential, admin-only, under the usual spend caps. The venv needs nothing — venv_on_volume already builds it under a lock with .ready written last, so the first pod in a region builds it and every pod after finds it, which is what makes the dependency-free slim image usable in every region. Also here: _data_centers() upper-cases region ids, since the map's key matching the candidate's region is the whole mechanism; pools_schedule_status carries the map; both admin surfaces gained a "Per-region volumes" field. Everything is offline-tested only — 25 new tests in tests/test_runpod_regional_volumes.py. The first multi-region install is the real proof, and the first thing to check there is that RunPod's dataCenterId and the map's key are spelled the same. test_runpod_placement's region test counted three copies of `sel.get("_dc") or self._data_center()`; the region is now resolved once into a local because the volume lookup needs it too, so it checks that one resolution and its uses instead of duplicates of it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Per model, like every other pod setting, and bounded so that it can only ever NARROW what the operator configured. A pool with a preference and allow_other_gpus: true takes another card when the preferred one is not free -- usually right, occasionally exactly wrong. Observed 2026-10-13: gpu_types [NVIDIA A40] fell back to an L40S at $1.09/hr against the A40's $0.59, correct per the configuration and useless to a caller measuring throughput, since a number taken on a different card is not comparable with the one it is being compared to. The only way to get the A40 was to edit the model's configuration -- changing it for everybody, permanently, for one request. POST /v1/runpod/scale now takes `gpu_types` (list or comma string, in preference order) and `allow_other_gpus`. The rule is one-directional: - a card the operator did not list is DROPPED and named in `notes`; - narrowing to only cards the operator forbids is a 400 that names what IS allowed -- never a silent fall back to the default, which would be the opposite of what the caller asked for; - allow_other_gpus may be turned OFF by a request, never ON; - max_hourly_usd, min_vram_gb, cloud_types and max_pods are the operator's alone and are not reachable from a request at all. Held on the LEASE, not the pool: an "A40 only" cannot outlive the 30 minutes it asked for, releasing the lease releases its constraints, and provisioning reads the narrowing per pod so a lapsed lease stops constraining the very next one. There is deliberately no path from a request to a model's configuration. Reported back in the answer and on /v1/runpod/status as `client_gpu_narrow`, because a constraint silently dropped is worse than one refused -- the caller would read its numbers as having come from the card it asked for -- and a pod on an unexpected card should have a visible cause, not only a bill. Tests: 32 new, including that the cost ceiling and the VRAM floor still refuse a pinned card, that the operator's limits are not even mentioned in the narrowing code, and that None (not False) is the default for both fields -- False would silently turn every caller's fallback off. Also fixed a source-window test that anchored on `_rank_gpus`'s full argument list and broke the moment a parameter was added; it now anchors on the assignment, since the arguments are not what it is testing. Full suite: 25 failures, all pre-existing and identical to baseline. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/models now carries `max_model_len`, under vLLM's own name so a client written against a vLLM server reads it unchanged. It is published because a client cannot otherwise plan: anything that sends a growing conversation -- a continued generation, a resumed OCR page, a long tool loop -- has to know when the window runs out, and the only alternative is finding out from a 400 after the work is paid for. coderai's own OCR page resumption was the first casualty, and the way it failed is the point. The 0.3.50 clamp probed /v1/models for this field; coderai did not publish it, and the surya worker talks to coderai rather than to raw vLLM. So the clamp was INERT from the day it shipped, and said nothing: the log printed "? of the context window left" and every resumption ran with no ceiling. Two further details made it invisible -- `max_model_len` is not in the OpenAI model schema, so the client parks it as an EXTRA field and reading attributes alone found nothing; and the probe only logged on an exception, so a server that answered WITHOUT the field was indistinguishable from one that worked. - The number comes from the LOADED backend first, config second. The two disagree and only the served one is true: `ctx: 18432` is configured on one install while `--max-model-len 32768` is what the pods actually serve, so a client trusting config would plan around a window that does not exist. - VllmBackend.context_length() asks its own upstream /v1/models, cached: a served model cannot change shape under us. - Resolution handles BOTH manager shapes. list_models is a method of the MULTI manager, which has no .backend of its own -- the per-model manager does -- so the first version of this could never have fired either. Found by writing the test, not by reading the code. - Several spellings read (max_model_len/n_ctx/ctx/context_length, plus the vllm/runpod/llamacpp blocks), because several backends own this number. - None means UNKNOWN, never unlimited, and the worker now says out loud when a server named no window, naming ocr.surya_context_tokens as the answer. Also, so that "which build is this pod running" stops being an inference: the pod boot log prints its coderai version and the short git sha of the app layer (new GIT_SHA build arg). A mutable `:latest` is a fine way to publish the current release, but it leaves two pods started minutes apart looking identical in an account listing -- and on 2026-10-13 that cost two pods being restarted that already had the fix, because nothing in their logs named their version. Tests: 27 new. They pin what the resolver PREFERS (live over config), what it tolerates (a stray string, a list-wrapped number, a non-dict backend block), that another model's loaded backend is not consulted, that the listing actually calls it, and that a server naming no window produces a message rather than a silence. Full suite: 25 failures, all pre-existing and identical to baseline. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Found while writing the Digesta dev the answer to "which field names does 0.3.51 actually honour" -- a question that could not be answered honestly because of these two. /v1/ocr/batch never accepted max_tokens or max_continuations. A batch is how a caller sends real work, so the budgets were accepted only on the path nobody benchmarks: every claim since 0.3.49 that "/v1/ocr and /v1/ocr/batch take these per request" was half wrong, docs included. FastAPI drops an unknown form field in silence, which is the same trap this feature existed to close -- the caller sets max_tokens on every batch and is never told it did nothing. The dev's measurements at "max_tokens=8000/4000/2000" were therefore all taken at the CONFIGURED value, whatever that was at the time. And /v1/ocr's route decorator was attached to the _page_opts HELPER rather than to create_ocr, because 0.3.49 inserted the helper between the decorator and its function. FastAPI registered the helper as the handler, so the endpoint existed, answered, and was not the OCR endpoint. Nothing failed at import; no test looked at the route table. 16 tests, 12 of which fail against the pre-fix code: the route resolves to create_ocr, _page_opts is not registered as a handler, both paths take both budgets, each defaults to None (0 is meaningful for both -- no cap override, resumption off), negatives clamp, and the batch path is asserted to FORWARD them, since a signature that accepts a field proves nothing about using it. Checked against the route table and signatures, so they need no engine or GPU. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The Digesta dev's 400 on page resumption, and a bug behind it that was worse than the 400. The vLLM backend built its outgoing request from an ALLOW-LIST of fields it recognised. Resumption asks for two beyond an ordinary request -- continue_final_message, so the model carries on the exact string, and add_generation_prompt: false, so no new turn is opened after the prefilled text -- and neither was ever forwarded, by this or any previous version. What reached vLLM was a conversation ending in an assistant turn with add_generation_prompt left at its default TRUE. surya-ocr-2's chat template then appends a fresh <|im_start|>assistant after the text so far, so the model was asked to START A NEW ANSWER about the same image. It did. And the failure hid behind the guard built to catch it: the new answer repeated the beginning of the page, the no-progress check saw text it already had, logged "came back repeating itself" and discarded the segment. 253 of 262 resumptions on the live install -- read for a day as a model that degenerates on dense pages, when it was a model being asked the wrong question, answering correctly, and having the answer thrown away. It also explains the dev's table exactly: at max_tokens=8000 resumption is barely exercised and OCR holds 93%, while at 4000 and 2000 nearly every page resumes, each resumption is a full generation that produces a restart and is then discarded, and the batch times out. Fixed by forwarding BY EXCLUSION, which is the only form that stays correct: coderai names the fields it CONSUMES (_CHAT_CONSUMED_FIELDS) and everything else goes to the server, which is the thing entitled to decide what it supports. An allow-list means every field a server gains after the list was written is born broken -- the same reasoning as forwarding page diagnostics by exclusion. It reaches only a backend whose signature accepts it, so a backend that cannot carry an unknown field is unaffected rather than broken, and model/messages/stream are never overridable this way. Also: a 4xx explains itself, if nobody discards it. raise_for_status() reports "400 Client Error: Bad Request for url: ..." -- the symptom and nothing else -- while vLLM puts the reason in the BODY. A whole diagnosis cycle was spent guessing at a 400 that had already been explained and thrown away. The error now carries the server's own words. Made tolerant of a response object that reports no status_code, rather than turning an odd shape into an AttributeError inside the request path. Two findings for the operator, not code: the pods serve max-model-len 32768 while the model entry asks for ctx 18432, so per-model vllm settings still are not reaching the pod; and this fix must be in the POD image, because the proxy that dropped the fields runs there. Tests: 16 new (flags reach the payload, an unknown field is forwarded, no passthrough changes nothing, structure not overridable, exclusion at the API layer, signature-probed delivery, the error carries the server's words). Full suite: 25 failures, all pre-existing and identical to the stashed baseline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two defects that a throughput collapse on a live install made visible. With surya_max_tokens raised to 12288, 262 resumption segments spent 1.6M generated tokens in 80 minutes, 122 of them running the full cap dry, while 253 of those 262 ended in "came back repeating itself". Resumption re-sends the text so far as INPUT, so the model's context window -- not the continuation budget -- is the real ceiling. The budget did not know that: at a 12288-token cap, the third segment of a page asked for 12288 more on top of ~28,500 already in a 32,768-token window and was rejected outright. The page was then reported as one that could not be resumed AT ALL, discarding text already paid for. - surya_context_tokens, or 0 to read the server's own max_model_len from /v1/models. Configuration wins: an operator may serve a shorter window than the checkpoint advertises. - Each segment asks for the smaller of the cap and the room left, measured from what the server said the prompt cost (usage.prompt_tokens), not guessed; the guess covers the first resumption only. - Out of room stops, KEEPS what it has, and says so -- naming the fix that helps, which is a SMALLER max_tokens, not a bigger budget. - A refusal for length is not retried: the fallback resumption adds a user turn, so it is longer than the request just refused. - Not knowing the window changes nothing and guesses nothing. The second defect: a reply that trips surya's repeat detector hit the cap because it is LOOPING, not because the page is long. Resuming it can only extend the loop, and surya re-runs the whole page anyway -- so one page on that path cost four full generations. It is no longer resumed, and it is not marked truncated either: that is surya's retry to make, and "truncated" would send the caller looking for a bigger budget it cannot use. A fixed token cap is also an accidental repetition circuit-breaker, which is why raising it slowed the LOOPING pages by the ratio of the caps rather than every page. Documented in docs/ocr.md with the window arithmetic (2048 -> ~13 segments, 12288 -> 2). Tests: 8 new (window clamp, stop-one-segment-early, length refusal not retried, unknown window, configuration over server, a cap below the 256-token floor is never mistaken for a full window, a looping reply is not resumed, a genuinely long page still is). 62 pass in the surya/blank-page suites; the floor-versus-cap bug was caught by the existing 44. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/ocr took engine, dpi, lang, structured, schema and detect. FastAPI drops unknown form fields SILENTLY, so a caller sending max_tokens=8000 on every request was discarded without a word and every page ran at the configured value. Both budgets are now real request parameters. One global cap cannot serve a public deployment: a four-page contract and a fifty-page judgment need different budgets and only the caller knows which it is sending. The values travel WITH the page rather than with the engine load, because a pool is built once and then serves every caller, and they are applied thread-locally in the worker, because several pages are in flight in one process and the hooks they feed are module-level — a per-request value in a module cell would be overwritten by the next page to arrive. Carried on a ContextVar rather than widening `recognize_image`, which is the one thing every engine has, duck-typed ones included. Omitting a parameter means "use the configured value", not zero: 0 is meaningful for both (no cap override, and resumption off). surya_max_continuations 2 -> 8. It is PER PAGE, not per document: a fifty-page document is fifty independent pages each with its own budget, so what the number covers is the densest SINGLE page. The real ceiling is not the number but the model's context — each resumption re-sends the text so far as INPUT, so with surya-ocr-2's 32,768-token window and ~4,000 of it on the page image, about thirteen 2,048-token segments exhaust it whatever is configured, and raising surya_max_tokens lowers that count in proportion. A generous budget is only safe if a page that stops making progress stops costing, so resumption now ends when a segment comes back repeating itself or returns the text already in hand. The second case is also how a server that IGNORES continue_final_message is caught: a model that starts the page over returns what we already have, and without the check a budget of 8 would buy eight full generations of the same page. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two things, from one sample document. RESUMING A CUT-OFF PAGE. Raising surya_max_tokens only moves the cliff: a dense enough page overflows any fixed number, and incomplete block markup parses to a fragment or to nothing. `ocr.surya_max_continuations` (default 2) resumes instead -- the text so far goes back as a prefilled assistant turn with vLLM's `continue_final_message`, so the model carries on the exact string rather than answering afresh about the same image. A server rejecting the flag (llama.cpp) gets a plain "continue" instruction; a server rejecting both leaves the page partial and honestly marked, since a page that cannot be finished is still a page. Logprobs are concatenated in step with the text, because surya_repair_accents rebuilds characters from the model's own per-token bytes and a later segment paired with earlier bytes would corrupt exactly the accented characters the repair exists to fix; if they cannot be aligned the repair is given up on rather than fed a mismatch. Temperature 0 -- the opposite of the retry path, which raises it to break a repetition. A page still unfinished after its budget carries "complete": false so a caller can refuse a partial rather than store it as read. On by default: returning a page KNOWN to be incomplete is a correctness failure, not a cost trade-off. IS THERE ANYTHING ON THE PAGE AT ALL. A four-page court PDF came back with 47 characters and was reported as an OCR failure. It was not. Every page was a ~900-byte CCITT Group 4 image -- what an EMPTY page costs, a couple of bytes per all-white row -- carrying 0.13% ink, all of it a diagonal scan edge and a diagonal "copia non ufficiale" watermark. The 47 characters WERE the watermark. The engine had read the page perfectly. "The model could not read it", "the reply was cut off" and "the page is blank" were indistinguishable in a short result, and they lead to completely different places -- a better engine, a bigger budget, or a document fixed upstream. Any page whose text is shorter than ocr.blank_page_max_chars (200) now carries the fraction of its pixels that are not background, from an image already in memory, with blank_pages listed at the top of the response. The opposite verdict is as useful as the blank one: ink above the floor says there IS something there, so the engine's failure is real and worth chasing. It applies to every engine, because the measurement is of the image rather than of the model, and it is a diagnostic only -- nothing is skipped, refused or altered because of it. Verified against the sample: all four pages 0.13-0.33% ink, correctly called blank against the 0.5% floor. Two bugs found while testing: `rounds` counted resumption ATTEMPTS rather than successes, so a failed resume would have reported the page as whole; and three assertions matched a source window that moved, one of them for a sentence that is wrapped across f-string lines and never appears contiguously. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three defects, found from a live batch that reported ~98 pages of about 71 characters, 100 with NO text at all, and 13 reading perfectly. 1. `model_attempts` could not reach anyone. `_page_from_dict` copied the worker's diagnostics onto page.meta BY NAME and knew only `unreadable_trace` and `accents_repaired` — so the field added in v0.3.45 was dropped at the subprocess pipe, and was being hunted for in a live batch that could not have contained it. The same reassembly had already dropped `accents_repaired` once, reporting null for a repair that worked. It now forwards by exclusion: every key that is not an OcrPage field becomes meta. A key-by-key copy means each new diagnostic is born broken and nobody finds out until they need it. 2. An empty page and a 71-character page are the same bug. surya-ocr-2 emits block markup — a category and a bbox per block, BEFORE the text — so a reply stopped at its token cap is incomplete markup, and incomplete markup parses to a few blocks or to none. The cap is surya's `max_tokens_default=2048`, which its vLLM backend never overrides, and the same batch averaged 2,644 generated tokens per request: above it. `ocr.surya_max_tokens` raises it (0 = surya's own), opt-in because a higher cap costs tokens on every page. What is not optional is telling the cases apart, so a page that hit the cap now carries tokens, cap and `truncated: true` with an explanation that it is a cut-off reply and NOT a page the model could not read. Counts are per THREAD, like the trace, so one lane never reports another lane's verdict. 3. The fleet could grow by at most one pod per BOOT. `_should_grow` opened on a boolean held for the whole provision, and a provision is ~12 minutes — so a burst needing four pods took forty minutes to be served by four, whatever max_pods said (measured: 64 pages at 16 per pod got two pods, ready ten minutes apart; reported as "no real scale-out"). The grower also could not see the SIZE of the demand: `min(inflight) >= threshold` answers "is the fleet full?", so the only request it could formulate was "one more", and the callers blocked in acquire() — the actual measure of unmet demand — were invisible. Both are counts now: `_booting` (booting pods ARE capacity, so one demand is never provisioned for twice) and `_waiting`. Growth runs while pods + booting < demand, bounded by max_pods and by the new `scale_up_max_parallel_boots` (default 4) — a real limit, because the pods the demand justifies are the GPUs that start billing together. The original single-flight anticipatory trigger is KEPT beside it: max_inflight_per_pod defaults to unlimited, so requests are often never blocked and a merely saturated fleet would otherwise never grow at all. A prediction gets one pod; measured demand may get several. Found while testing (3): acquire() ignored its own timeout when a sibling process was booting a pod for the same pool — it waited a flat 15s and looped back into the same decision without re-checking the deadline, for as long as the sibling's registry entry survived (up to 15 minutes). A test hung on it. 42 tests across tests/test_runpod_scale_out.py (new) and tests/test_surya_retry_budget.py. Two of my own test harnesses were wrong and are fixed, not worked around: a sys.modules ledger stub that `from codai.api import runpod_ledger` ignores once anything has imported the real module, and a static count of increments against decrements, which cannot express "one increment, four mutually exclusive exits" — replaced by runtime balance tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
baidu/Unlimited-OCR (MIT, 3B, BF16, 8GB) is an in-process OpenAI-endpoint VLM that emits layout-tagged blocks with bboxes and categories, and parses long documents in one 32k pass. Four phases: the engine and the <|det|> parser (offline-testable, no GPU); serve modes so a local GPU, an endpoint the operator already runs, a managed model, a rented pod and no-GPU-at-all are each first-class; a serve profile following vllm-ocr.txt, since the architecture is in no released vLLM wheel; and document-level OCR designed as a capability of the SUBSYSTEM, because coderai's OCR contract is per-page end to end and this model's one-shot mode is the reason to lift that. Records what upstream requires and what each requirement costs us, including the three decode preconditions that yield EMPTY OUTPUT rather than an error, and the logits processor we cannot set on someone else's endpoint. Language coverage is unenumerated upstream, so it is treated as unknown and measured, never designed in or out. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Comparing 64 pages against max_pods x inflight (64 x 16 = 1024 on the Digesta orchestrator) would never have fired, and would have been reassuring if it had: the fleet's ceiling is not what serves a burst. A pod takes ~12 minutes to become ready, so the burst is served by the pods already warm. What is worth saying at setup time is the arithmetic itself — these pages need that many pods — plus the case where it exceeds max_pods and the surplus can only queue. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Read from the pod's own drained log, ten minutes of batch OCR on 2026-10-13: Running: 15-19 reqs, Waiting: 0 reqs, GPU KV cache usage: 5-7% Avg generation throughput: ~1,900-2,100 tokens/s, every minute 524 x "POST /v1/chat/completions" 200 OK, no aborts 2,644 generated tokens PER REQUEST (second pod: 4,010) MM cache hit rate: 25-27% Three things in those numbers, and only the third is a defect. The GPU was never the bottleneck: 5-7% of the KV cache and `Waiting: 0` for ten solid minutes. So a card to itself cannot be the fix, and the co-loaded pyannote cost nothing measurable — the limit was never VRAM or batch slots. Oversubscription converts throughput into latency, not into stalling. 64 pages into max_pods x scale_up_inflight_per_pod = 32 slots queues correctly, but every page's latency rises by the queue, and a client counting finished DOCUMENTS sees nothing until a document's LAST page lands: spread 64 pages over sixteen documents and all sixteen advance while none completes. The pool now says so at build time instead of leaving it to be inferred. The defect: 2,644 generated tokens for a page whose text is ~400. surya re-runs a page whose generation repeated itself (`_should_retry`) up to three more times, each at a rising temperature and each able to reach its token cap — four full generations to arrive at the same unreadable answer — and its repeat detector cannot tell "the model broke" from "this page really does repeat", which a tax table does. The 25-27% multimodal cache hit rate is the direct evidence: a quarter of all requests re-sent an image the pod had already read. None of it was visible: a page generated four times over and a page that was merely slow look identical from outside. `max_retries` is unreachable from surya's settings — the vLLM backend calls chat_completions_batch without it, so the default 3 was the only value there has ever been. So the decision is hooked, beside the token-trace hook already in that module, and `ocr.surya_max_retries` is a CEILING (asking for 10 where surya allows 3 gets 3): 1 by default, 0 to disable, -1 to hand it back. Every retry is logged, and a page that needed more than one call carries `model_attempts` in its result — always, not only under surya_token_trace. A page read first time stays byte-identical to what it always returned. The counter is per THREAD, like the trace, or one lane would be blamed for another lane's retries. Also pins SURYA_INFERENCE_PARALLEL=1 in the served modes: a lane is our unit of concurrency, and left unset surya sizes its own thread pool from a built-in GPU table and fans out inside the worker, so instances x lanes would stop being the real number of pages in flight. 19 tests in tests/test_surya_retry_budget.py; settings carry the new knob. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`keep_warm` stopped re-renting whenever a pod went away behind the pool's back — terminated from the console, reclaimed as spot, lost with its host — and the scaler said nothing about it. Both halves had one cause: nothing in a pool ever re-checks a pod's health. A handle's `healthy` flag goes false only when something tried to USE the pod and failed, or when the pool itself retired it, and a warm-floor pod is precisely the pod with no traffic to fail. So `maintain()` counted the handle, found `healthy_n >= keep_min`, and neither topped up nor had anything to log. The floor stayed "satisfied" by a machine that had not existed for hours, and the first real request paid the full ~12-minute cold boot that keep_warm is bought to avoid. (The class docstring claimed the scaler "re-checks health" for months. It never has. Corrected rather than implemented: a timer-probe would have to decide what an unanswered probe means, and that decision is what killed seven A40s in v0.3.41.) `reconcile_pools` instead reads the account listing the stale-pod reaper already fetches every ~30s — authoritative, free, and unable to mistake a busy pod for a dead one. Only positive evidence acts: * a dead status (TERMINATED/EXITED/DEAD/FAILED) drops the pod at once; * a mere absence drops it only after two listings in a row, because the API is eventually consistent and one page omitting a live pod used to be enough to unregister it — a reappearance resets the count; * a pod that served a request in the last 60s is kept whatever the listing says, which cannot mask this bug since a warm-floor pod has no traffic; * a provisioning pod is shielded, and a FAILED listing never gets here at all, so an API outage cannot empty every pool and re-rent the fleet. `PodPool.forget()` is `discard()` minus the kill: nothing is left to terminate, but the bill is closed in the ledger exactly as a real termination closes it, since the pod was billed up to its death. A borrowed `(shared)` pod is dropped without a ledger entry — its owner holds the bill. The registry is pruned through the same two-sweep rule it used to skip. The top-up now announces itself before spending, because a warm floor silently failing is indistinguishable from a warm floor that is fine. 21 new tests in tests/test_runpod_warm_floor_rerent.py pin the caution, not just the fix: the failure mode in the other direction is forgetting a pod that is alive and billing. test_runpod_client_scale's source assertion moved to the new seam. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The worker protocol was one request, one reply, strictly in order, so pages in flight WAS the process count — and a process costs ~600 MB of imported torch. That is the whole reason `surya_instances: 64` meant 64 processes, 37 GB and a nine-minute stall with the rented GPU idle at 2% KV-cache usage. In the served modes that memory buys nothing: the recognition VLM is on the pod, the local predictor holds no weights (1.7 MB of tokenizer and config on disk — surya-2 returns its own boxes, so the detection model is never even fetched), and the process rasterises a page and then waits on a socket. So a lane is a worker OBJECT inside the process that already exists, with its own predictor and its own page, sharing the one torch import. Effective concurrency is instances x lanes; `surya_lanes` defaults to 4, served modes only, built on demand. Three things make one process safe to drive concurrently: - every reply carries the id of the request it answers. Out-of-order replies are the point — a short page must not queue behind a dense one — and attributing one page's text to another request would be worse than losing it, so an unknown id is dropped rather than guessed at; - the token trace and the accent repairs are per THREAD. surya's hook is module-level, one _generate_one per process, and its sink was a list belonging to whichever worker object installed it last. Shared between lanes it would have filed one page's logprobs, and surya_repair_accents' rewrites, under a different page; - a worker that dies fails its pages instead of leaving them parked on a reply that is never coming, and killing the engine wakes everything waiting. A single-lane engine sends no ids and reads replies in order exactly as before, which is how paddle, docTR and surya-`local` are still driven. `lanes()` is asked for politely (_engine_lanes), because an engine is anything with recognize_image and a pool must not refuse to build over a capability an engine simply does not have. Tested against the REAL worker protocol with a stubbed engine: four 0.25s pages finish in under 0.75s, on four distinct lane objects, and six pages over two lanes never exceed two. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A batch of documents gave one flawless 19-second burst — 22 documents, correct page counts, clean accents — and then no completions at all for nine minutes, while the rented A40 sat at 2% KV-cache usage with `Waiting: 0`. The GPU was never the bottleneck. Two local mistakes compounded: `surya_instances: 64` reads like GPU concurrency and is not: surya, paddle and docTR spawn ONE OS PROCESS per instance, in their own venv. 64 of them meant 37 GB of RSS, swap exhausted and load 25 on a 16-core host that also runs some thirty other containers. Nine hours after the batch ended, 66 of those workers were still resident with 19 seconds of CPU time each. Then it fed itself. The workers' /health probe (5s, against coderai's OWN gateway) began timing out under the load the workers were creating, and a timed-out probe counted as "the endpoint is gone" — so each worker was killed and respawned, which added load, which timed out more probes. - the process-per-instance engines are capped at one worker per core and never more than half of free RAM, with a log line naming the figure and the reason. CODERAI_OCR_INSTANCE_CEILING overrides it for a measured host. The in-process engines (chandra, olmOCR) are untouched: an instance there is a concurrency slot, not a process; - a health probe now means "gone" only on a definite 404 "unknown OCR engine lease", which is the case re-resolving was written for: a front that restarted. A timeout means busy and the worker stays up, because respawning cannot fix a front that is down — the replacement asks the same front for a lease. A 503 means the pool has nothing rented this second, which the lease does not care about. GPU-side concurrency remains runpod.scale_up_inflight_per_pod and max_pods; the local worker pool is a hard ceiling on how many pods surya can keep fed, which is the standing argument for running it in-process. Co-Authored-By:
-
- 12 Oct, 2026 15 commits
-
-
Stefy Lanza (nextime / spora ) authored
Five A40s were terminated between 19:14 and 20:11 today. None of them was broken: each pod's own log ended in `"POST /v1/chat/completions HTTP/1.1" 200 OK` at 4664 tokens/s, seconds before it was reaped, and every replacement cost a ~6-minute image pull before it could read a page. `httpx.RemoteProtocolError("Server disconnected without sending a response")` was in the dead-upstream set. It does not mean the server is gone — it is what httpx raises when a POOLED connection was closed by the far end between two requests. uvicorn closes an idle connection after 5s and nothing here raised that, while this gateway held pooled connections for 60s. So the first page after any quiet gap hit a socket the pod had already closed, the gateway read it as an empty address, and the pool terminated a rented GPU for it. - a closed pooled connection is retried on a fresh connection, against the SAME upstream: re-choosing would send the page to a different pod, or rent one; - a pod is asked before it is dropped — /healthz, then /health, on a new connection. Any answer under 500 counts as alive, a 401 included: that is a server, authenticating. Only silence drops it, because dropping is irreversible; - idle pooled connections are released after 4s (CODERAI_OCR_KEEPALIVE_EXPIRY_S), under the server's own keep-alive, so the race is rare rather than routine. Reuse within a burst is unaffected. A pod RunPod really has reclaimed mid-document is still dropped at once, which is what the original logic was for: left in the pool, every following page picks the corpse and the in-flight count climbs with nothing draining. The tests pin both halves, and a read timeout still neither probes nor drops. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`teach_model` called `seed_model_env(entry, "", value)` — no served name — so the pod registered the entry under its `path`. For an ordinary model that is merely untidy. For an ADAPTER it is fatal: a LoRA's path IS its base, so diogene-alpha registered as cyankiwi/Qwen3.8-27B-AWQ-INT4 and collapsed into the base. Measured on the live pod before the fix: taught https://…:22143 about 'diogene-alpha' (cyankiwi/Qwen3.8-27B-AWQ-INT4) catalogue: […, 'qwen38-awq'] diogene-alpha listed: False HTTP 404: Model 'diogene-alpha' is not available. Taught successfully, still 404 — and the version of this where the base answers under the adapter's name would have been silent. With the name carried, the same pod registered it and served it. Verified by repetition rather than by eye: each name is byte-identical across two runs at temperature 0 and the two differ from each other, so they are backed by different weights. diogene-alpha sha 77f7db98ab19 ×2 (reasons, then answers) qwen38-awq sha 4767ec4fcf00 ×2 (answers directly) One pod, one copy of the weights, the adapter selected per request by name. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Seeds were de-duplicated by `path`. A base model and a LoRA of it share one path by definition, so the adapter's entry was dropped and the pod was told about the base alone — observed live, one line after the sibling seeding had correctly found it: [runpod] pool 'pool:digesta-qwen38' is shared with: diogene-alpha [runpod] pod will be told about 1 model(s) The dedup exists because a capability pod's list can overlap the one model a per-model pool also seeds, which is a real case; it just has to key on what the pod will SERVE the entry as — path plus alias, id and adapters — rather than on the weights the entries happen to have in common. Orchestrator-side only: the pod registers what it is handed, so no new image. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`/v1/chat/completions` was capped at 60 requests per 60 s per client IP as a module constant — no config, no env, no way to move it. On a RENTED POD that cap is the orchestrator's own: a pod has exactly one client, the install paying for it by the hour, so every request shares one bucket. An olmOCR pod measured 3.45 pages/s (207/min) and was being refused above 60. It also produced three wrong measurements that were reported as hardware facts. Concurrency ladders ran every level inside one window, so refusals began exactly where the cumulative count crossed 60: surya 16+24=40, then 20 of 32 pass, 12 refused (13 measured) surya +48 = 72 over, all 48 refused (all 48 measured) chandra 8+16+24=48, then 12 pass, 20 refused (21 measured) olmOCR same ladder, same arithmetic (21 measured, twice) "Both peak at 16 and both fail outright by 32 — a cliff" was the rate limiter. Paced across windows, olmOCR runs N=32 with ZERO errors at its best throughput; it is flat at ~3.4 pages/s from N=8 to N=32 while p50 goes 2.3s -> 7.9s, which is saturation at 8, not a cliff at 32. So: the limits take env overrides (CODERAI_SERVER_RATE_LIMITS="prefix=count/window", CODERAI_SERVER_RATE_LIMIT_ENABLED=0), and a pod is launched with the limiter OFF — its admission is pod_max_parallel_requests + pod_queue_max_size, which describe the GPU rather than guessing at an abusive client that cannot exist there. docs/ocr.md keeps the old table with the refusals marked as refusals, because the numbers themselves are still the only ones measured and the reading of them was the error. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The autoupgrade is version-gated, so code pushed at a version Aruba has already taken would never be pulled. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two more links in the same chain, both of which fail SILENTLY — the pod serves the base weights under the adapter's name and nothing errors. * `seed_model_env` computed the content id behind `os.path.isfile(local)`, which is False for a directory, so the adapter never reached the pod's config. The per-request push sends the BYTES but its rewritten entry is discarded by the caller, so the seed is the only thing that tells a pod an adapter exists. * a named shared pool seeded only the entry of whichever model created it; `seed_entries` was populated for CAPABILITY pools alone. Sharing a pool exists so one pod answers for several names on one copy of the weights — a base and its adapters — which cannot work when only one name is known. Siblings now travel with `path` resolved to what the pod can FETCH, because these entries are described by `runpod.hf_repo` and carry no path of their own, and a seed without one is dropped. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A QLoRA trained for a client sat on disk for ten days while a second service rented its own A40 every morning, pushed the adapter over SSH and proxied to it. The path that should have carried it was written but had never been run end to end, and five things were in the way. * `ensure_text_loras` DROPPED a PEFT directory — "publish it to HuggingFace" — which is the shape every trainer writes. It now travels as one deterministic tar (sorted names, fixed uid/gid/mtime, so the content id is stable) and is unpacked by the blob store, because vLLM loads an adapter from a directory: adapter_config.json carries the rank and target modules, and a bare .safetensors is not loadable. * the adapter was re-read and re-hashed ON EVERY REQUEST. A Target is rebuilt per request, so `lora_entry` was set again each time; the bytes were never re-sent (the store is content-addressed) but 159 MB was read and hashed, and the pod asked, only to learn it already had it. Cached per (pod, path, size+mtime), so a retrained adapter at the same path is still sent again. * `_lora_args` passed a CONTENT ID straight through, producing `--lora-modules name=sha256:…`, which vLLM cannot open. On a pod that is the only form the adapter has. * an engine launched for a base model carried none of its adapters. vLLM registers them at launch and one vLLM serves one base, so the pod a shared pool rented for the base alone could never serve an adapter of it — the whole point of sharing the pool. Every model naming the same base now contributes its adapter under its own model id, which is the name requests ask for, so two versions can be A/B'd on one GPU with no reload. * `--max-loras` was never set. At its default of 1, requests naming different adapters cannot share a batch and serialise instead: the cost lands on latency, not on an error. Extracting a bundle is a write from network bytes, so member names are checked for escapes before anything is unpacked, and tar's data filter is used as well. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Triton and torch.inductor JIT-compile vLLM's kernels at run time and need a full build toolchain for the stubs they emit. Each piece was found on a live olmOCR pod, one layer at a time, because from outside the failure is identical every time: vLLM dies before registering and the pod answers 502. gcc the C compiler g++ the launcher Triton builds is C++, so gcc alone is not enough ninja-build inductor shells out to ninja for its extensions Only olmOCR hits this, and the reason is worth recording: it is FP8, so vLLM's quantisation kernels go down the Triton path. surya-2 is dense bf16 and served fine on images that had none of these, which is why "the image lacks a compiler" looked for a while like it could not be the problem. Applied to vllm.apt AND vllm-ocr.apt, which is copied from it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
It shipped at 4, which made it the binding limit for every install. The limiter is meant to be each engine's own `*_instances` -- the pool hands out one instance per page and an engine is not reentrant -- while a RunPod fleet can hold max_pods x scale_up_inflight_per_pod pages in flight: 64 x 16 = 1024 on the Digesta install. A ceiling of 4 capped that at a quarter of ONE pod. Measured on one A40, same page, same install: N surya req/s p50 chandra pg/s p50 1 0.93 1.1s 0.21 4.6s 8 3.79 2.0s 1.31 6.1s 16 7.11 2.2s 1.88 8.4s <- both peak 24 5.89 4.1s 2.09 10.6s 32 5.44 +13 err 3.5s 1.44 +21 err 7.5s 48 ALL FAILED Both peak at 16 and both fail outright by 32 -- a cliff, not a degradation. Default raised 4 -> 64: four pods' worth at the measured knee, far above what a single-box install can feed, so it stops being the limiter while staying finite. NOT higher, because every admitted page holds a rasterised bitmap (~18 MB for A4 at 250 DPI): 64 in flight is ~1.2 GB, 1024 would be ~18 GB of page images before any engine does work. Trading a throughput limit for an OOM is not an upgrade. docs/ocr.md carries the table and spells out the three settings that bound this, which are easy to confuse: the per-model scale-out threshold, the per-engine instance count (the real limiter, and for the SUBPROCESS engines it is what the box can host -- a surya worker is a separate interpreter at ~750 MB), and this ceiling. Also recorded there: surya's req/s is BLOCKS, not pages, since it calls once per layout block; and its worker protocol is synchronous, so a worker handles one page at a time and concurrency is the process count. In served mode that worker only scales an image and makes an HTTP call, so making it concurrent -- or skipping the subprocess entirely, as chandra does -- is what would let surya use a large fleet. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
SURYA READS ITALIAN AGAIN. Measured on the live install, the same synthetic page that scored 29 markers all morning: MARKERS : 0 (29 baseline) ACCENTS : AEEIOUaeeiou (all twelve) La societa` ha facolta` di piu` attivita`: percio` e` cosi` che l'unita` perche' gia` oltre. ...but it reported `accents_repaired: null` while visibly having fixed the page, because _page_from_dict carried `unreadable_trace` across the worker boundary and never carried `accents_repaired`. The fix worked; only its receipt was dropped. That matters more than a missing field: the text is no longer byte-for-byte what the server sent, and a caller comparing the two has no way to know why. A silent fix is nearly as bad as a silent corruption. tests: +2 (47 in test_ocr_accent_repair.py) for the receipt crossing the boundary and for staying absent when nothing was repaired. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
ROOT CAUSE, measured in surya-ocr-2's own tokenizer: id 2160 decode -> U+FFFD vocabulary piece -> 'a-grave' id 2168 decode -> U+FFFD vocabulary piece -> 'e-grave' id 2169 decode -> U+FFFD vocabulary piece -> 'e-acute' id 2172 decode -> U+FFFD vocabulary piece -> 'i-grave' id 2178 decode -> U+FFFD vocabulary piece -> 'o-grave' id 2185 decode -> U+FFFD vocabulary piece -> 'u-grave' The model emits the RIGHT token -- 2160 really is a-grave -- and the decode table cannot render it. Every decoder in the chain inherits that: surya's own, and vLLM's, which reports `decoded_token: None` and therefore `bytes: []` and writes U+FFFD into the text. So the accents were never lost in transport. Eliminated by measurement, each in turn: DPI (a crisp synthetic page lost them identically), the document (chandra reads the same pixels perfectly), the vLLM version (0.20.1 cannot load the model at all), --trust-remote-code (applied, `trust_remote_code=True` in the engine config, no change), and the byte channel (vLLM reports `bytes: []` precisely because the decoder failed). THE FIX: when the decoder answers with the replacement character for an id, use the VOCABULARY's piece for that id. The vocabulary is the authority on what a token means. Decoding is per id, because a bulk decode cannot be corrected after the fact -- once flattened to text, the id that failed is unidentifiable. Verified end to end against the live pod: decoder alone : 'nt ? pi? facot? perci? cos?' with vocab : 'nt a-grave piu-grave facota-grave percie-acute cosi-grave' This needs the ids, which vLLM only reports when its server is launched with --return-tokens-as-token-ids (the per-REQUEST field is ignored on the chat path: chat_completion/serving.py:1179 falls back to the server setting only when the request field is None, and the id form is reached through _get_decoded_token). That flag is now in Aruba's global vllm.extra_args, which is what a pod receives -- the per-model `vllm` block in models.json does NOT reach a pod, which cost two test cycles to discover. An id the vocabulary also cannot name keeps its marker, and the ASCII-skeleton guard still decides whether the repair as a whole is safe to apply. tests: +3 (45 in test_ocr_accent_repair.py) pinning the vocabulary fallback, the unknown-id case, and that decoding stays per id. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`return_token_ids` added to the vllm entry of _ENGINE_FORWARDED_FIELDS, so a rented pod's vLLM backend can be asked for raw vocabulary ids -- without which the v0.3.31 repair can never fire on a RunPod-served model, which is every OCR model on the Digesta install. Inert by default: engine_config_for_pod skips falsey values, so adding the field cannot change an install that has not set it. Set on Aruba for `datalab-to/surya-ocr-2` ONLY, through its models.json `vllm` block rather than globally. The flag changes what the `token` string inside logprobs looks like and that install also serves qwen38-awq and diogene-alpha on vLLM pods; nothing in coderai reads that field except the OCR repair, but a client might, and there is no reason to find out the hard way. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The byte channel is a dead end on 0.27.1 -- measured: `content` comes back with one U+FFFD per accented letter AND the per-token `bytes` come back EMPTY (257 of them on one page, no byte >= 0x80 anywhere), so neither output carries the character. But the token IDS are not lossy, and surya's OWN tokenizer decodes them correctly: decode([2131, 2104]) == 'e-grave', measured in its venv. So the repair now asks for ids and decodes them itself, bypassing vLLM's detokenizer entirely. Both halves, because each alone does nothing: 1. THE WORKER reads vLLM's `token_id:{id}` form, decodes with surya's tokenizer (lazily loaded and cached -- the worker must start while the venv is building, and a clean page must not pay for a tokenizer), and keeps the byte route as the fallback for any stack that reports bytes honestly. Ids are tried FIRST because on 0.27.1 the bytes lead nowhere. All-or-nothing on the id list: a partial one would decode to a partial page. The ASCII-skeleton guard still applies, so a decode of the wrong call cannot be substituted. 2. THE PROXY stops dropping the request. `return_tokens_as_token_ids` is a per-request field on vLLM's ChatCompletionRequest, but sending it to a pod changed nothing -- because codai/backends/vllm.py::_chat_payload builds an explicit whitelist and silently discarded it. That is exactly the failure its own neighbouring comment warns about ("a proxy that edits the request decides for it"), five lines further up. Now forwarded, gated on `vllm.return_token_ids`, and only alongside logprobs since it describes how logprob tokens are rendered. `vllm.return_token_ids` is off by default (it changes what the `token` field looks like, and only this one consumer reads it) and is in VLLM_FIELDS, so the single checkpoint with a byte-level vocabulary can ask for ids without changing what every other vLLM model reports. A config field rather than relaunching the server with --return-tokens-as-token-ids: same effect, no pod rebuild. This is a fix ON 0.27.1. It makes the coderai-vllm-ocr image (vLLM 0.20.1) a CONTROL that confirms the diagnosis rather than the only way out. tests: +11 in test_ocr_accent_repair.py (36 -> 47) covering the id form, the all-or-nothing rule, ids-before-bytes, the fallback when a decode fails, the skeleton guard on the id route, and the proxy forwarding with its logprobs gating. 912 passed across the ocr/front/config/admin/runpod/vllm/backend selection; the 9 failures are pre-existing (broker x6, studio, whisper x2) plus test_cluster's RPC check, which depends on llama_cpp being installed in this venv and touches nothing changed here. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
CONFIDENCE. chandra hard-coded `conf=0.0` and olmOCR reported nothing at all. chandra's code even argued for it: "the model reports none, and inventing one would be read as a measurement". The first half was right and the conclusion was wrong -- the number was obtainable all along, by the route surya already uses. Both are served over an OpenAI chat endpoint, so the request now asks for logprobs and the reply's mean exp(logprob) is the confidence. It is the same computation surya's own client does, so the three VLM figures are comparable with each other instead of being three different notions of "confidence". engine scope source paddle line recogniser score doctr line recogniser score surya block mean token probability chandra page mean token probability olmocr page mean token probability The SCOPE is reported too (`/v1/ocr/engines` and `meta.confidence_scope`) and it is not cosmetic: a per-line score can single out the line to re-read, a page-level one can only say the page went badly. surya calls per layout block so its figure is genuinely per-block; chandra and olmOCR generate a whole page in one call, so chandra puts the one page figure on every line and region (which is what `unreadable`'s per-spot conf reads) and says the scope is "page" rather than letting it pass for per-line precision. A server may decline logprobs, and then conf is 0.0 -- which must NOT read as "the model was hopeless". `meta.confidence_measured` separates the two. That is the same distinction the unreadable-glyph trace needed and twice got wrong in this subsystem: a zero meaning "not measured" and a zero meaning "measured as bad" are different facts, and conflating them produces confident nonsense. coderai's in-process chat path surfaces no logprobs, so it reports not-measured rather than a 0.0 posing as one. g++. Digesta got an olmOCR pod past the TLS problem and straight into a 502: Triton JIT-compiles vLLM's kernels at run time and the launcher it builds is C++, so it invokes g++ -- and `vllm.apt` shipped only gcc. With only gcc the image looks right, vLLM starts, and then the engine core dies before it registers. Added to `vllm.apt` AND to `vllm-ocr.apt`, which was copied from it and had inherited the same gap before ever being used. BOTH vLLM pod images therefore need a rebuild to carry it: coderai-vllm (which olmOCR uses, so this is what unblocks Digesta) and coderai-vllm-ocr (pushed an hour ago and affected identically). One at a time, as ever. tests: test_ocr_confidence.py, 20 tests -- the arithmetic (exp then mean, not the geometric mean), None-not-zero for an unreporting server, the page-vs-block scope, the rotated-retry carrying the retry's figure, and a registry check so a new engine cannot arrive with no answer to "what does conf mean here". Ten existing fakes updated for the new (text, confidence) contract. 858 passed across the ocr/front/config/admin/runpod selection with the same 8 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
CORRECTION. v0.3.26-0.3.28 reported `blame: "the serving path"` with `conclusive: true` and "the byte stream decoded cleanly", on data that was not there. Measured through the live gateway today: result.raw (what surya reads) : 3 markers 'nt ? propriet? della propriet? d' reassembled from logprobs : 0 markers 'nt propriet della propriet ...' entries with bytes=[] : 257 any byte >= 0x80 in stream : False vLLM 0.27.1 reports `bytes: []` for the byte-level tokens of surya-ocr-2's custom vocabulary, and not one byte >= 0x80 appears anywhere in the stream. The accent bytes are not corrupted in the logprobs, they are ABSENT from them. _reassemble accepted an empty byte list as valid data, so it built a string with the accents silently missing, found no invalid UTF-8 in it, and called the result clean. That is reading absence as cleanliness -- the exact failure the envelope in v0.3.26 was built to prevent for MISSING logprobs, walked into one level further down. Fixed: an empty byte list counts as missing. The trace now counts `tokens_without_bytes` and, when any are present, reports `conclusive: false` with a `why` that says the loss is inside the serving stack (it produced neither usable bytes nor usable text) while stating plainly that this CANNOT say whether the model's own output was sound. `unreadable.blame` is then "unknown" rather than an accusation the data does not support. Consequence for surya_repair_accents: it CANNOT work on vLLM 0.27.1 -- there is nothing to rebuild from, and that is not a tuning problem. It is kept because its guards refuse correctly: the ASCII-skeleton check catches the mismatch, so a page comes back with its markers rather than with the accents quietly deleted, and the repair will work on any stack that does report the bytes. So on vLLM 0.27.1 the real options are `chandra` (reads the same pixels perfectly) or the `coderai-vllm-ocr` image pinned to the vLLM surya validates against. tests: +5 in test_ocr_token_trace.py (42 total) pinning that empty bytes are refused by both the reassembly and the repair, that the blame goes to "unknown" rather than the serving path, and that a FULLY reported clean stream still produces the genuine verdict. 643 passed across the ocr/front/config/admin selection with the same 8 pre-existing failures. Co-Authored-By:
-