- 19 Oct, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authoredReverts the advice in 8b976170, which was wrong. A pod serving ~32 pages at once is a fact about what the SERVER should dispatch to one pod, not about what the CLIENT should offer. Offering only what the pods can hold means that the instant a page finishes there is nothing queued to replace it, so every pod idles for a round trip between every page -- losing exactly the throughput the queue exists to protect. The split is now explicit in the guide: the client offers 64 x pods, the server dispatches ~32 to each pod (discovered from the pod, v0.4.38) and holds the rest queued and ready. The client cannot overload a pod by offering more, and does not need to track what a pod can take. Admission is sized for exactly that offer: 4096 dispatched + 4096 queued = 64 x 128 pods, with 429 only beyond it. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
-
Stefy Lanza (nextime / spora ) authored
Telling a client to send exactly 32 x pods was me asking it to compensate for a server that could not absorb being wrong. v0.4.38 bounds per-pod dispatch from what each pod will accept and answers 429 with Retry-After and the four live numbers when the queue in front of the fleet is full. The guide now says that: 32 x pods is the efficient window, not a requirement, and a 429 is a signal rather than a failed page. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A client must be able to send too much. The server's job is to carry what it can and say so about the rest -- not to depend on the client having been told the right number, which is what the previous fix amounted to. TWO THINGS WERE MISSING. 1. NOTHING BOUNDED WHAT ONE POD WAS SENT. `max_inflight_per_pod` defaults to 0, meaning unlimited, so out of the box every page handed to the pool is dispatched. A client offering more than a pod can serve therefore does not queue in the orchestrator, where the queue would be counted, reported as demand and bounded -- it queues in the POD'S LISTEN BACKLOG, and an overflowing backlog makes a healthy pod refuse connections. That is what cost eight A40s yesterday. The right width is a property of the pod, the model and the card, not of the install, so the pod is asked rather than configured. A refusal halves what we send that pod, FROM THE IN-FLIGHT COUNT AT THE REFUSAL -- its observed breaking point, not a guess -- and successes walk it back up one slot at a time (multiplicative decrease, additive increase: overshooting costs refused connections and a misdiagnosed fleet, undershooting costs a little latency). Floored at 4, never above the operator's own ceiling, per pod so one overloaded pod does not throttle the fleet, and inert on an install where no pod ever refuses. Live numbers: 64 configured, 58 in flight at the refusal, width becomes 29 -- the pods were measured serving 29-32. 2. A FULL QUEUE ANSWERED SLOWER INSTEAD OF SAYING NO. The queue itself is deliberate: when a pod finishes a page there must already be another ready to hand it, or the fleet idles between dispatches and is paid for the gap. So the server carries `ocr.max_concurrency` in flight PLUS `ocr.max_queued_pages` waiting (0 = follow max_concurrency) -- and past that answers 429 with Retry-After and the numbers a client needs to size its window (X-OCR-Page-Width, X-OCR-Queue-Limit). Before, a client offering several times the capacity got no error at all: it got replies that took longer and longer, which reads as "the pages are getting slower" rather than "stop sending", so it sent more. Measured 2026-10-19: 522 pages offered against a fleet carrying ~250, mean page latency 238s against a 19s service time, nothing in any reply saying the server was over capacity. Refused BEFORE the upload is read. The client hears at once instead of after transferring a document the server will not look at, and the server stops buffering megabytes it is about to discard -- in-flight memory here is concurrent requests times upload size, the product that has OOM-killed this endpoint before. Every page now takes its slot through one counted path, which is what makes the queue depth knowable. Writing that exposed a real defect in what it replaced: `self._sem or asyncio.Semaphore(4)` built a FRESH semaphore per page, so an unconfigured manager bounded nothing at all -- forty pages of one document ran at once against what every line of code called a limit of four. Caught by the existing pod-pipeline gate test. 38 tests. Full suite 3817 passed, 15 pre-existing failures unchanged. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The guide said `in_flight = 64 x pods_ready`. 64 was the configured ceiling; the measured capacity is ~32 (`Running: 29-32 reqs, Waiting: 0` on every pod of a live fleet), and the 0.557 pages/s per pod the guide quotes was itself measured at a concurrency of 32. Offering 64 does not queue inside the orchestrator. It queues in the pod's socket backlog, and an overflowing backlog makes a healthy pod refuse connections -- which on 2026-10-19 got eight A40s terminated mid-work with 522 pages on them. v0.4.37 stops the termination; this stops the overload that triggered it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`discard()` terminated the pod, on the reasoning that a connection can only be refused by a pod that no longer exists. A pod whose listen backlog is full refuses connections while being perfectly alive and busy -- and refuses them on /healthz too, so the "is anything still there?" probe added to protect against exactly this agrees it has vanished. Observed 2026-10-19, mid-campaign, ~58 concurrent connections per pod: 08:36:13 pod 2n69g0vt47h0hl terminated (connection refused) 08:36:30 pod c79lv31ayh28l1 terminated (connection refused) ... eight healthy A40s in 69 seconds ... 08:37:22 pod n9fpbskd7owl2h terminated (connection refused) 522 pages were in flight on them. The warm floor then re-rented eight more, which booted ~12 minutes, began refusing, and were killed again: $3.72 of pods billed -- more than the campaign's whole spend -- on boots that never produced a page, while throughput read 0.577 pages/s against a measured 0.557 PER POD. A pod is now QUARANTINED instead: out of rotation, still ours, still counted against max_pods because it is still billing. `maintain` re-probes it when the quarantine expires (45s) and * it answers -> straight back into rotation, which is the common case and the whole reason this stopped terminating; * it has work in flight -> left alone whatever the probe says, and it accrues no strikes while excused, so it does not die the moment the last page drains. Killing a pod with pages on it loses them AND pays for them; * it fails 3 probes with nothing in flight -> terminated. ~2 minutes of a pod that exists, bills and will not answer while carrying no load is dead in every sense that matters, so this does not trade a terminating bug for a leaking one. A quarantined pod also counts toward the warm-floor deficit. Without that, eight set-aside pods read as a deficit of eight and the pool rents ~700s of boot to cover a 45s gap, ending with two cards where the floor asked for one -- the re-rent loop in one line. The snapshot reports `quarantined` as its own state with the seconds remaining, and `pods_quarantined` in the summary. It is NOT counted as booting, which told an operator a settled fleet was still starting and told a client sizing from pods_ready to wait for pods already present. `discard` has exactly one caller (RunpodBackend._unreachable, reason "connection refused"), so nothing else loses a termination it needed. 19 tests. Full suite 3777 passed, 15 pre-existing failures unchanged. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The front forwards to an engine over a client built with `pool=None` -- wait forever for a free connection -- and httpx defaults to `max_connections=100`. Together those two defaults cap everything the front forwards at 100 in flight and make the 101st wait indefinitely. Nothing anywhere reported a limit, because nothing had set one. Measured on a live orchestrator mid-campaign, client offering 512 pages: established to the front 514 forwarded to the engine 100 <- the cap pages/s 0.42 (fleet sized for ~3.3) mean page latency 238 s (service time measured at ~19 s) cost per page 12x the measured baseline The pods were rented, healthy, and unreachable. The 200 s of that latency that is not service time was pages queueing for a connection. It also wedged the control plane, which is how this was finally found. /v1/runpod/status is not under /admin/api/, so it took the same unbounded path and never answered at all -- http=000 at 30 s, three times in a row, while /healthz answered in 2.1 ms. Two consequences, both worse than a blind admin page: a lease could not be RELEASED while the fleet was busy, so pods kept billing; and a client sizing its in-flight window from `summary.pods_ready` -- which is what the throughput guide tells it to do -- was flying blind as a direct result of the thing it was measuring. * the forwarding pool is sized from `ocr.max_concurrency` plus headroom, floored at 256 and capped at 8192. An unreadable config falls back to the floor, never to httpx's default: silently reinstating the limit this removes would be the worst possible fallback. * waiting for a connection is bounded at 30 s, so exhaustion surfaces as a retryable 502 rather than a socket that never answers. The READ timeout stays unbounded -- a page may legitimately take minutes, and confusing the queue with the work would truncate real pages. * control-plane paths (fleet status, scale, release, healthz, metrics) take a separate 16-connection pool that request traffic may not use. A pod that cannot be released because the release cannot get a connection keeps billing. This supersedes part of v0.4.34's diagnosis. The control pool and the bounded snapshot there were real fixes to real defects, but they were in the ENGINE app, and the wedge was in front of it. 18 tests. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The throughput guide told a client to lease pods and said nothing about bounding what that lease may grow to, because there was no way to. Adds the staging recipe (one call per stage, no restart), what the three $/hr figures now mean and which one to alarm on, and that a `degraded` status document is stale rather than an idle fleet. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Found by exercising the new ceiling against a live orchestrator: max_pods=128 effective_max_pods=8 client_max_pods=8 client_pods=0 client_lease_expires_in_s=0 The lease had 1800 s left and was the reason the fleet had stopped growing at 8. `client_lease_expires_in_s` was gated on the FLOOR, which is 0 for a cap-only lease -- "grow no further than 8, hold nothing warm", which is what a client with a long tail of work asks for. It now reports whichever half of the lease lasts longer. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/runpod/status is the one surface an operator has for seeing WHY a server is busy. During a pod boot wave it was both unanswerable and, when it did answer, wrong about the only number anyone was watching. IT COULD NOT ANSWER. Measured on a live orchestrator: /healthz 200 in 2.9 ms (no thread needed) /metrics 200 in 180 ms (no thread needed) /v1/runpod/status no answer in 25 s (needs a thread) /v1/runpod/scale no answer in 25 s (needs a thread) Nothing was broken. Those two are the only endpoints that hand work to a thread, and they drew it from the DEFAULT executor -- the pool the request path also draws on. 25 s of silence reads as a dead server, and it also meant a client lease could not be RELEASED while the fleet was busy, so pods kept billing with no way to stop them but the console. Three changes, each closing one way that happened: * a CONTROL pool (16 threads, fixed) that nothing in the request path may use -- status, scale and release run there. Its whole value is being empty when everything else is full, and a test pins that no request-path module touches it. * the PDF split moves to a two-wide PDFIUM pool. pdfium admits exactly one caller, so hundreds of concurrent documents each took a general thread and then queued on the same global lock: the serialisation was unchanged while the threads they held were the control plane's. Queueing two-wide costs nothing that was not already being paid. * the snapshot's three sources now run concurrently, each with its own 5 s deadline, and whatever is late is reported as `degraded` rather than waited for. pools_schedule_status() was the half never bounded in v0.4.21 -- pods_status() gave up on a busy pool and this one waited for it -- and the ledger snapshot reads without the lock it shares with the writer that fires on every termination. Enforcement still takes that lock properly: only the snapshot may degrade. IT COULD NOT BE BELIEVED. `live_hourly_usd` summed only HEALTHY pods. RunPod bills from creation and a pod spends ~12 minutes booting without ever being healthy, so a fleet entirely in paid boot reported as free: reported $0.59/hr actual $4.72/hr reported $0.00/hr actual $7.67/hr (13 pods, 8 serving) A client polling it concluded nothing was being spent and kept feeding work. It now counts every pod in the listing, and reports the serving and booting halves separately -- their ratio is how much of the bill is boot, which is the figure that says a ramp is too steep, and it did not exist before. Both admin pages show it; "0 pod(s) · $7.67/hr" was a real reading and is self-contradictory. 29 tests. Full suite 3741 passed, 15 pre-existing failures unchanged. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/runpod/scale set a FLOOR. "Give me 8 pods" meant "hold at least 8 ready", and the autoscaler stayed free to grow the pool to the configured max_pods for as long as work kept arriving. A client running a staged batch -- 8 pods for the first stage, 32 for the next -- had no way to bound a stage at all, and nothing in the reply said so. Observed 2026-10-18: a run that asked for 8 pods was at 13 and climbing toward 128. Every pod past the plan is ~700s of paid boot producing nothing, which is why the realised cost per page came out 16.5x the measured baseline while every component behaved exactly as written. A lease may now also carry `max_pods`, under the same rule `gpu_types` already follows there: it may only ever NARROW the operator's ceiling, a larger number is clamped and reported in `notes`, and it dies with the lease. `pods: 0` with a `max_pods` is a cap with no floor -- what a client with a long tail of work wants; `pods: 0` alone releases both. The ceiling has to be read by EVERY path that decides whether to rent, or it is a race between the grower that honours it and the one that does not: _pods_for_demand, _grow_reason, _pods_needed and the warm-floor top-up all now read effective_max_pods(). A leased floor is clamped to it too, since a floor above a ceiling is a fleet that oscillates for as long as the lease lasts. A configured max_pods of 0 still means zero -- that is the operator having turned renting off, not a lease. 22 tests, including a guard that no growth path reads mcfg.max_pods directly, so a path added later cannot silently escape every cap. Co-Authored-By:
-
- 18 Oct, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authored
A page dispatched to a pod holds a pod-pool thread for its whole life, so the pool's width decides how many pods an orchestrator can actually feed: pods_feedable = pod_pool_width / max_inflight_per_pod With _MAX_POD at 2048 and a 64-page-per-pod ceiling, that was 32 pods — whatever max_pods said. An operator who set max_pods=128 got 96 pods' worth of nothing, and no line anywhere explained it: not ocr.max_concurrency (4096 here, so it looked generous), not max_pods, not the cgroup (pids.max 16384 with 79 in use on the box this was measured on). A silent fleet ceiling is a worse failure than a slow one, because every symptom points at the pods. Raised to 8192, which feeds 128 pods at 64 pages each. Past that an async client really is the right answer rather than more thread stacks. Nothing that protects the box changes. `pool_sizes()` still clamps against `task_budget()` — the cgroup's pids.max minus what is running minus headroom — still keeps _TASK_HEADROOM in reserve so a busy install can still be diagnosed, and still gives the pod pool away before the general pool, because an install that cannot answer /healthz cannot be fixed. Tests pin all three. One existing test asserted `pod == 2048` for a configured width of 2048. That was only true because _MAX_POD clamped there; the property it meant to protect is that the configured width reaches both pools, which is `want * 2`. Restated that way, so the next raise does not break it again. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A rented pod has no config.json and no models.json -- it creates defaults and environment is the only channel it has. CODERAI_OCR_CHANDRA_INSTANCES and CODERAI_OCR_OLMOCR_INSTANCES were sent to every pod and read by nobody, so those engines fell back to the dataclass default of ONE instance however many the operator configured, while the admin UI showed the configured value. Measured on one A40 with chandra_instances=128 and 32 pages in flight: [ocr] engine 'chandra': 1 instance(s) ready vLLM: Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6% Avg generation throughput: 54.5 tokens/s against surya's ~2000 tokens/s and ~32 concurrent sequences on the same card and the same pages. It read as "chandra is 36x slower than surya"; it was one page at a time. Surya worked only because its own env rows had been mapped -- the gap was invisible for exactly as long as surya was the engine being tested. Adds the two instance counts, plus chandra's and olmOCR's page budgets (MAX_TOKENS, MAX_CONTINUATIONS, MAX_RETRIES, DPI), which were sent and unread for the same reason: seeding an engine's wiring but not its tuning gives a pod running that engine's library defaults, which is indistinguishable from the outside from the wrong pod image. The test is deliberately generic -- for each engine and each tuning knob, if the orchestrator sends it the pod must read it. A list of expected names would have passed before this fix on surya's rows alone and said nothing about the other two engines, which is how this survived. NOTE: config.py runs ON the pod, so a pod only gets this once its image carries it. coderai-vllm-ocr must be rebuilt before chandra serves more than one page at a time. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.4.29 added a guard against a pool holding one pod twice and asserted, in its message and in the code, that this was what billed two A40s against a max_pods=1 configuration. That attribution is not supported. The $1.18/hr incident was two DISTINCT pod ids, four minutes apart in the account listing, and its cause was the warm-floor top-up counting only healthy pods against the cap -- fixed in 0.4.27. The duplicate was a later and separate observation: a snapshot carrying two rows for one pod id, taken without recording the model key, so two POOLS reporting one shared pod was never ruled out. That case `_add_pod` does not address, since it dedupes within a pool. The guard is kept, because two paths appended pods with no identity check between them and that is a hazard on its own. What is removed is the claim that it explains a billing event, and the invented detail of a pool "reporting 2 in pool, 2 healthy, 2 at cap with 128 pages in flight while the card ran 31" as though it had been read off that duplicate. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two paths append pods -- a fresh rent and an adoption -- and neither checked whether the pod was already in the pool. On a restart both fire for the same machine within the same second: [runpod] adopting pod j74fxkxr9vs3fg's bearer token [runpod] reclaim: took back a pod for 'datalab-to/surya-ocr-2' ... and the pool then held one A40 twice. Every capacity question is answered by walking self.pods, so the duplicate is not cosmetic: * the pool claimed 2 x max_inflight_per_pod = 128 slots on a card that was serving 31, reporting "2 in pool, 2 healthy, 2 at cap" with 255 pages queued -- and refusing work the pod could have taken; * max_pods is enforced against that same count, so a cap of ONE was satisfied by two entries for one pod, which is how a second real pod came to be rented beside it and billed at $1.18/hr for a configuration that asked for $0.59; * the in-flight staleness sweep added in 0.4.28 cannot help here, because both entries hold legitimate fresh timestamps. A duplicate is not a leak. Both sites now go through `_add_pod`, which is the only remaining way into the pool and refuses a pod id it already holds. The EXISTING handle is kept rather than replaced: it carries the in-flight accounting for work already dispatched to that pod, and dropping it would strand every slot it holds. What can legitimately change is refreshed onto it -- the address, since RunPod remaps ports and the pod id is the identity, not the url. A test asserts there is exactly one `self.pods.append(` in the file: the guard is worth nothing if a third path is added later that bypasses it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`pod.inflight` is raised in `_pick` and lowered only in `release`, which makes it a ratchet. Any dispatch path that fails to release raises it for the life of the pod, and at max_inflight_per_pod the pod is refused all work while its card sits idle -- reported as "at capacity", which is the most misleading thing this file can say. Observed on a deliberately single-pod fleet: the pool reported inflight=64 of 64 with 63 pages queued, while the pod itself reported `Running: 31, Waiting: 0` at 10.6% KV cache. Half the slots did not exist. A single test document then waited 900s for a pod that was half idle, and timed out. Two such leaks have now been found and fixed -- a lease dropped during acquisition (0.4.23) and a page whose client disconnected. The reason this change exists is the third one, which has not been found: a counter that can only rise is one bug away from taking a pod out of service permanently, and every leak so far has been invisible precisely because the pool reported confidence about a number nothing could correct. So each slot now records when it was taken, and the LIST is the authority: * slots held longer than INFLIGHT_TTL_S (900s) cannot belong to a live page and are returned. Deliberately generous -- a safety net, not a timeout. Returning a slot that IS in use would hand a pod more work than its cap allows, while holding a phantom a few minutes longer costs only those minutes. * a counter standing above its own recorded slots is counting slots with no evidence of ever being taken, and is pulled down to the evidence. This is not conditional on anything having expired -- the first version of the sweep only examined pods with a stale entry, which would have skipped a pod whose counter had drifted upward, and skipped every pod already in the pool when this code arrives carrying a counter and no timestamps: exactly the locked-out pod the sweep exists to free. The list is created on demand rather than required at construction, so duck-typed pod stand-ins are not forced to carry a bookkeeping field, as with the per-card memo. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pool configured max_pods=1, min_pods=1 held two A40s: a pod adopted at 17:41, a restart at 17:43, a second pod rented at 17:46. $1.18/hr for a configuration that asked for $0.59, and the operator's explicit ceiling exceeded by the very code path whose job is to hold exactly one pod warm. The warm-floor top-up in maintain() computed in_hand = len([p for p in self.pods if p.healthy]) + self._booting and tested BOTH the deficit and the max_pods cap against it. After a restart the pool adopts the pod it had before. That pod is briefly not yet healthy, and it was not booted by this process, so `_booting` is 0 and `in_hand` is 0 -- room under a cap of one, beside a pod that is already running and already being charged for by the minute. They are two questions and they need two counts. The deficit is in pods that can SERVE, so it keeps counting healthy ones: an unhealthy pod does not satisfy a warm floor whose entire purpose is that the next request finds a pod ready. The cap is in pods that are CHARGED FOR, so it counts every pod in the pool whatever its health -- which is what `_should_grow` has always done for its own cap. This path disagreed with it, and the disagreement was the bug. The existing test pinned the single `in_hand` by its source text; it now pins both counts, including that the cap is the billing one, since that is the part whose regression costs money. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The diagnostic added in 0.4.25 tested `gate * 4 < ocr.max_concurrency`. Those are different units -- a request against a page -- and for the 4096-page budget this install runs it demanded a gate of 1024: memory-unsafe, and four times more than the derivation in the same function will ever choose. It therefore fired against the 256 that derivation had just picked, telling the operator to raise a value coderai itself had chosen as correct: [queue] max_parallel_requests=256 max_size=2048 [queue] WARNING: ... admits only 256 request(s) ... Raise server.max_parallel_requests ... The floor and the cap are now named once, as _GATE_FLOOR and _GATE_CAP, and both the derivation and the warning read them, so the two rules cannot disagree again -- a separate literal in each place is how they came to. The warning now tests the gate's own size rather than its ratio to a page budget, which is also the only thing worth saying: a handful of admitted requests cannot feed a pod pool whatever the page budget is. Since an unwritten key is already derived above the floor, it now reaches only an explicitly small value, and says so. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
server.max_parallel_requests defaults to 2 in flight and 6 queued. That is right for one local card answering chat. For an OCR install dispatching pages to rented pods it is the throughput ceiling, and nothing points at it: the pool is sized in pages (ocr.max_concurrency, max_inflight_per_pod), a pod is handed its own much larger limit through env, and the front in front of all of it keeps admitting two requests. Measured on one A40 with ocr.max_concurrency=4096 and the pod allowed 64: the pod sat at `Running: 1, Waiting: 0` -- asking for work and not getting it -- while 219 pages took 1210s, the same as reading them one at a time. Raising the gate took the pod to 8-11 concurrent sequences on the same documents. The `[queue] max_parallel_requests=2` line was printed at every boot throughout and read past, because a number with no consequence attached does not look like a problem. Two changes, neither of which overrides a deliberate choice: * When OCR is enabled and the operator has NOT written the key, the gate is derived from the page budget they did ask for (page_budget/8, floored at 16 and capped at 256) rather than taking a default chosen for a different deployment. An explicit value is obeyed however small, and so is an env override, since env is the only channel a rented pod has. The cap is there because the remaining bound is memory, not CPU: an admitted request holds its whole upload plus the per-page PDFs split from it. * When the gate is explicitly set below what the configured page budget needs, a startup diagnostic says so and names the setting to change, instead of leaving the operator to infer it from pods that look idle. ConfigManager now records which server keys were actually written, because an explicit 2 and an absent 2 are indistinguishable from the parsed value and must be treated differently. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/admin/api/system/reload built a bare `Request({})` in order to reach `app.state.config_mgr`, which raises on the first attribute: the endpoint answered 500 to every call it ever received and had never reloaded anything. An operator editing a config file by hand therefore had no way to apply the edit except restarting the server. That matters most for a pod pool. A pool reads its ceilings -- max_pods, gpu_types, the cost caps -- out of models_data on every acquisition, so an edit the process has not re-read is an edit that does not apply, and the operator's only lever was a restart that drops whatever is in flight. The handler now takes the injected Request and falls back to the manager that set_config_manager installed, because the front proxy and the engine do not both populate app state. It reloads BOTH files and reports how many model entries it re-read, so a caller can see that the reload reached models.json and not only config.json. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`placement.choose` runs in a thread and raises `pod.inflight` the moment it picks a pod. If the awaiting coroutine was cancelled after the executor had produced the Lease but before the value was delivered, the object was dropped with the counter already raised -- and nothing released it, because the page was gone and no `finally` ran. A client that gives up on a batch cancels every page of it at once, so the leak arrives in thousands rather than one at a time. Measured on a live run, once `max_inflight_per_pod: 64` made it visible: the pool reported all 52 healthy pods at their cap -- 3,328 pages "in flight" -- while `ss` inside the container showed 3 to 5 open sockets per pod, the cards sat at `GPU KV cache usage: 0.2%` with `Running: 1 reqs`, and nothing was preempted. Roughly 60 of every 64 pages did not exist. `_pick` then refused every pod, correctly, reporting "all 52 serving pod(s) are at their in-flight cap", and the run fell to 0.33 pages/s. It had been leaking since the pod pipeline was written. With `max_inflight_per_pod: 0` an inflated counter blocks nothing, so the only symptom was DEMAND -- the in-flight counts of 2,700 and 4,096 that earlier runs reported, and that the scaler dutifully rented pods to serve. The cap did not cause this; it converted a silent accounting error into a hard stop, which is how it was finally found. Three runs were spent on its consequences. `_acquire_lease` shields the acquisition, so it completes even when the page waiting on it is cancelled, and a done-callback hands the lease straight back. Orphans are counted and said once a minute, because the entire cost of this bug was that it was silent. Note on the analysis: the gap between `lease = await ...` and the `try` that releases it is NOT a window -- CancelledError is only delivered at an await point, and there is none in between. The acquisition itself was the only one. Proven rather than argued: the same test, same cancellation, against the unshielded form and the shielded one -- granted=1 released=0 LEAKED=1, against granted=1 released=1 LEAKED=0. Co-Authored-By:
-
- 17 Oct, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authored
Suggested by a third party and worth keeping: run layout detection, count the blocks, and pick full-page or block mode from the count. Recorded rather than built, because our own evidence says the repetition it would prevent came from the 3072-token cap (532 of 594 continuations were segment 2, and the loops appear AT the truncation boundary, where a continuation re-sends the page its own output), and v0.4.22 raises that budget. The argument that survives the budget fix is GPU utilisation: one full-page generation is a single sequence and cannot be batched with itself, which is consistent with pods idling at 1.4% KV cache and with an A100 PCIe delivering only 1.14x an A40. SURYA_INFERENCE_BACKEND=vllm is already set, so the batching it would rely on is in place. The doc states the three measurements that decide it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A per-request `max_tokens` wins over the engine's own default and over the operator's configured budget. That is right -- only the caller knows whether it is sending a four-page contract or a fifty-page judgment -- and it stays the default. But it cost a run, and the operator had no say. A client sent `max_tokens: 3072` on every page of a corpus whose pages need just over that. Each page therefore generated to the cap, stopped mid-text, and paid a continuation that RE-SENT the first 3072 tokens to retrieve the last few hundred; by the second cut-off the model was often repeating, so the page was refused a resume and re-run whole at a higher temperature. Measured across 594 continuations on the live fleet: 532 pages reached segment 2, 44 segment 3, 11 segment 4, 7 segment 5. That distribution is the signature of a budget MARGINALLY too small, which is the most expensive kind -- nearly every page pays double, rather than a few pages paying a lot. The cards sat at `GPU KV cache usage: 1.4%` with `Running: 0-4 reqs` while the pool grew to 65 pods at $47/hr, because the time was going on sequential round trips and not on inference. Adding pods could not help and did not. So `<engine>_max_tokens_floor` (surya, chandra, olmocr): an operator may state a minimum. Off by default -- an install that silently overrode a caller would be lying about its API -- never LOWERS a budget, because a caller asking for more knows something the floor does not, and never invents one where the caller asked for nothing (omitted means "use the configured value", which may well be higher) or asked for 0 (which means unlimited, so raising it would cap it). Applied once at the top of `ocr_document`, before either pipeline sees the opts, so a local read and a pod forward cannot disagree about the budget. Said once per engine per five minutes with both numbers and the reason, because a 5,000-document run would otherwise write the line 30,000 times, and because an operator should not have to find this commit to learn why a small budget is expensive. Seeded to pods as `CODERAI_OCR_<ENGINE>_MAX_TOKENS_FLOOR`. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
During a boot wave /v1/runpod/status stopped answering entirely, and a client reasonably read that as "the front is dead" -- while /healthz answered in 2.9 ms throughout and the front recorded zero event-loop stalls. It was never the loop. The snapshot takes the pool locks, a provisioning wave holds them, and the request simply waited until the caller gave up. That cost more than observability. It is the one surface an operator has for seeing WHY a server is busy, so losing it exactly when the server is busy sent two investigations to the wrong layer, and it silently turned a monitoring sampler's output into zeros -- 48 pods billing at $35/hr, recorded as an empty fleet, because a hung read and an empty fleet wrote the same row. So every lock in `pods_status` is taken with a timeout (`_lock_for_read`, 2 s). A pool whose lock is busy contributes one `state: "busy"` row naming itself; a locked registry answers one such row rather than an empty list, because "no pods" and "ask again" are opposite actions and must not look alike. Writers keep the plain lock: a reader that skips it reports less, which is the point, while a writer that skipped it would race. Note for anyone testing this: `_pools_lock` and `pool._cv` are RLocks, so a test that takes one in the thread that then reads simply re-enters and passes. The holder in production is always another thread, so the tests hold them from one too -- three of them passed for the wrong reason until they did. Also, in the same endpoint: `caps` has been silently empty in every answer it has ever given. `from codai.config import config_manager` -- which does not exist, the manager lives in `codai.admin.routes` -- raised every time and the except swallowed it, so an operator polling for the install's spend ceilings never received them. Identical mistake, identical shape, to the one that pinned both thread pools to their floors on every install that ever ran. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.4.19 fixed the lease -- pages now reach the pool, where in-flight had been stuck at zero -- and the run still produced nothing: 56 pods, 27 healthy, 1,040 pages in flight, ZERO `POST /v1/ocr` in any pod's log, and /v1/runpod/status unanswerable while /healthz answered in 2.9 ms. The ramp was eating the process. Measured on that run: coderai-cpu at 75%, event-loop stalls of 57.4 s and 44.9 s, and 406 candidate rejections in 120 seconds with every GPU type re-probed 33 to 72 times. Two duplicated costs, both paid once per BOOT and so multiplied by 32 parallel boots: * Each boot thread swept the whole ~70-candidate list itself, so each independently rediscovered that A100 SXM has no capacity in any of seven regions. The verdict is now shared and short-lived (`_unavail`, 90 s): one thread learns it, the other 31 skip it. On this run's numbers that is 2,240 probes reduced to 70. A successful deploy clears the entry, and the TTL is short because capacity returns without telling us. * Each boot fetched and parsed the FULL GPU price catalogue (`list_gpu_types` was uncached), so 32 boots meant 32 identical large JSON bodies parsed within seconds. Cached 60 s process-wide, each caller getting its own copy so a mutating caller cannot corrupt it, with `fresh=True` for a caller showing prices to a person rather than picking a card in a storm. Both costs are Python, so they hold the GIL, and CPython gives it to whoever is running: the event loop carrying the page forwards simply did not get scheduled. That is why in-flight climbed while no socket was ever opened to a pod -- and the undelivered queue then read as demand, so the scaler rented more pods, which starved the loop harder. The fleet grew because it could not serve and could not serve because it was growing. No blocking call is involved, which is why the 0.4.17 static check passes and the loop watchdog was the thing that found it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Third run, third failure, and this one was mine. A 5,000-document run dispatched ZERO pages to 50 healthy pods in 41 minutes: not one `POST /v1/ocr` in any pod's log, every GPU at `KV 0.0%`, 811 pages waiting, and the scaler reading that unservable queue as demand and renting to 73 pods at $43/hr. `/healthz` answered in 1.6 ms the whole time, so the loop fixes from 0.4.16/0.4.17 were holding and nothing looked wrong from outside. 0.4.18 bounded a lease attempt at 20 s and ran it on a 256-thread pool. That made the LEASE STAGE the ceiling on everything: 256 threads at one attempt per 20 s passes ~13 pages a second however many pods are up, and a thread that drew the growth branch sat through an entire ~9-minute boot. Against 811 pages each retrying every couple of seconds, nothing got through. So `acquire(grow_inline=False)`: answer now, or start growth in the background and raise `NoPodRightNow`. The caller waits on its event loop, where waiting is free, and `POD_LEASE_ATTEMPT_S` drops 20 s -> 2 s for the pools whose acquire predates the flag. Threaded through `placement.choose(grow_inline=)` by signature inspection, so a candidate that cannot take it keeps working. Demand had to survive that. `_waiting` only ever counted THREADS parked in acquire, and the non-blocking path has none, so the pool would have stopped growing. It now counts DENIALS over a window equal to the caller's retry interval, which makes the count distinct waiting PAGES rather than the attempts they have made between them -- a page retrying every 2 s appears once in any 2-second window. A longer window multiplies demand by the retry rate, which is how the same run read its own retries as new work. Two more things found on the way: * `pod_weight`/`pod_capacity`/`pod_threshold` depend only on a card's NAME, but were recomputed per POD per acquire -- `_pick` once per free pod, `_pods_for_demand` once per serving pod, `_preferred_vram_gb` walking every pod inside each -- all under the pool lock. Memoised per card: measured 286us -> 69us per acquire at 70 pods, 528us -> 102us at 128. The memo self-initialises via getattr, because a pool built without __init__ must not have an optimisation turn into an AttributeError. * `_pick` returned a bare None for four different causes -- no pods, none healthy, all draining, all at their in-flight cap -- which need opposite responses, and the caller could only report the queue it could not drain. It now says which, once per 15 s, with every count in the line. I also mis-read `futex_do_wait` on 324 of 352 threads as lock contention. A condition-variable wait IS a futex wait, so that is what pages legitimately waiting for a pod look like. The pod logs settled it instead: zero POSTs means the dispatch never happened, not that it was slow. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A 5,000-document run leased 48 warm pods and then read 278 pages ON THE ORCHESTRATOR -- a box with no GPU -- while every one of those cards sat idle. Each page logged "the pod-side pipeline could not run (can't start new thread)". Zero documents were read on the pods. The container's task budget was 2048 (podman's --pids-limit default) and the pod pool had sized itself 2048 from ocr.max_concurrency: 4096. It reached 1992 threads; the next one could not exist. Nothing had looked for a limit, so nothing reported one. Three separable mistakes, all three fixed: * `thread_pool.task_budget()` reads what the process is actually allowed -- cgroup v2 pids.max walked up to the root (a limit on any ancestor binds as hard as one on the leaf), minus pids.current because the other processes in the cgroup already spend it, and RLIMIT_NPROC -- and `pool_sizes()` fits inside it, keeping 256 tasks in reserve so a rasteriser worker, an isolated-venv engine or a `podman exec` to look at the box can still fork. The general pool is protected ahead of the pod pool: an install that cannot answer /healthz cannot be diagnosed. On the numbers above it now sizes 256 + 1506 instead of asking for 2304. * A lease took a POD-POOL thread per in-flight page, so the page width an install wants decided how many thread stacks it needs. A lease is a bounded wait (POD_LEASE_ATTEMPT_S), so a separate 256-thread lease pool serves thousands of pages by cycling. It also stops a cold start filling every thread with leases waiting for pods, leaving none to carry the pages those pods were for. * "Cannot create a thread" was treated as "the pods cannot run the pipeline", and the fallback for that is the local pipeline -- which needs MORE of this box than the path that just failed for want of it. Local exhaustion now answers 503 naming both ways out (the task budget, or ocr.max_concurrency) and leaves the pages for the cards. AND THE TEARDOWN, from the same run: the client's own release sweep answered 500 five times in three seconds, because terminating a pod that was already gone raised. By the time a fleet teardown reaches its last pod some are always gone -- reaped for idle, shed by a scale-down, or released by a retry of the same call -- so a teardown that was working looked like one that had failed. Already gone is the goal, not a failure: it answers 200 with `already_gone: true`, while a real failure still raises so a pod left billing is never reported as stopped. Also: a 400 on an upload path now logs why. A malformed or truncated multipart body is rejected before any handler runs, so 19 of 137 batches failed in that run with no record anywhere but the access line -- not the filename, not the size, not even whether it was the body or a field. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.4.16 fixed one blocking call on the loop by finding it the hard way: a 5,000-document run rented 127 pods and completed nothing for ten minutes, and the only honest signal in the system was that /v1/runpod/status had stopped answering. Nothing in the process said "the loop was blocked", though it had been, for minutes at a time. Both halves of that are addressed here. MEASURED: codai/api/loop_watch.py samples the loop's own behaviour -- a sleep that asks for 0.5 s and returns at 4 s says the loop was busy elsewhere for 3.5 s and served NOTHING in that window. Installed in both the engine and the front (the front is the process the client talks to, so a stall there is the one it experiences), logged per stall with its duration, and exported on /metrics as coderai_loop_lag_seconds, _lag_worst_seconds, _stalls_total and _blocked_seconds_total. It cannot name the culprit, but it turns "the server is slow" into "the server was not running", which is a different investigation. CHECKED: tests/test_no_blocking_calls_on_the_loop.py walks every coroutine in codai/ for calls that block the calling thread. What makes it precise is that it does not look inside a nested def or lambda -- `to_thread(lambda: f())` and `run_on_pod_pool(f, x)` are both correct, so only a call reached without crossing a function boundary is on the loop. Its own ten tests prove it catches what it claims and ignores os.path.join, asyncio primitives and coroutines handed to ensure_future. It found 20 more, all fixed: * ocr/gateway.py: `placement.choose()` in async chat_completions -- the SAME 1200 s pool acquire as 0.4.16, on the chat path instead of the page path, and with no retry loop above it to make the wait visible. * api/ocr.py: a sync urlopen of a caller-supplied URL, so an OCR step was as slow as whoever served that URL, for every request in the process. * eleven `subprocess.run()` ffmpeg/ffprobe calls across faceswap, video, pipelines, voice_clone, custom_pipelines and the outfit video path. A media encode is seconds to minutes, and all of it was everyone's wait. * two HuggingFace API fetches in the admin routes. * two `thread.join()`s on the CUDA streaming path, and a `time.sleep(0.1)` per generated image. The allowlist in that test is empty and should stay empty: an entry in it is a request that cannot be served while the named call runs. Also tools/loadprobe.py, for the outside view: /healthz does no work and returns the instant the loop is free, so its latency IS loop lag as a client sees it. p99 healthz flat while documents crawl is a capacity problem; p99 healthz in seconds is not, and no extra capacity would have helped. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A 5,000-document run rented 127 pods, reached $79/hr and completed zero documents in ten minutes. The front had stopped answering entirely: /v1/runpod/status timed out on every port, which is also what the client had been reporting as thousands of timeouts. The pods were fine. Probed from inside the container, both transports answered /healthz in milliseconds, vLLM was loaded on each one, and the GPUs sat at KV 0.0% with no surya worker line, because no page had ever reached them. `placement.choose` is synchronous and ends in a pool's `acquire()`, which waits on a condition variable for up to its default 1200 s and may rent a pod INLINE -- a candidate sweep over every region, then a boot wait, in whichever thread asked. It was called straight from `_read_on_pods`'s coroutine, so it ran on the event loop and took the whole process with it for as long as one page waited for a pod. It has been there since the pod pipeline was written (0.3.70) and cost nothing while acquire() returned at once: a pool with a free pod answers in microseconds. It turns fatal exactly when the pool is the thing under strain, which is when it matters most. So the lease runs on the pod pool now, and because a pod takes MINUTES to boot, three things follow: * One attempt is bounded (`POD_LEASE_ATTEMPT_S`, 20 s), threaded through `placement.choose(acquire_timeout=)` to `pool.acquire(timeout=)`. An unbounded attempt parks a thread for a whole boot, per page; bounded, the waiting happens in `_lease_a_pod`'s own loop where it is free. * `ocr.pod_pipeline_wait_s` goes 60 s -> 600 s. The 60 s was chosen because a waiting page held a thread, so a long ceiling meant thousands of parked threads. It no longer does, and at 60 s a cold start handed its whole first wave of pages to a GPU-less orchestrator to rasterise -- the exact failure `prefer` exists to prevent. * The per-page PDF split moves to the DEFAULT executor, not the pod pool: pdfium permits one caller at a time, and putting a split behind thousands of leases waiting for a boot is a cold start that cannot produce the next page while it waits for the pods that page is for. The forward itself stays on the event loop, which is the point of 0.4.8 and is now covered by a test that fails if it moves back. All of this is invisible to a test that awaits one page at a time: one page blocking the loop for 0.5 s looks exactly like one page taking 0.5 s. The new tests measure the longest gap between two ticks of a coroutine running alongside, and the thread-seconds held while waiting out a boot. Five of them fail on 0.4.15. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
chandra and olmOCR each set `meta.confidence` and `meta.confidence_measured`. surya set neither: its predictor reports confidence per LINE, so a caller switching engines was handed a different kind of answer, and nothing could tell "no line reported a confidence" from "the model was certain it was unsure". Now it reports both. Weighted by CHARACTERS rather than per line, because the other two average over tokens -- that is, over text -- and a plain line-mean lets a two-word line count as much as a full one: on [(100 chars, 0.95), (2 chars, 0.1)] it answers 0.9333 instead of 0.525. `confidence_measured` is the half that earns its place, and the reason the aggregate reads the RAW value rather than the per-line field: that field has to coerce a missing confidence to 0.0 for its own sake, which would make nothing reported -> 0.0 a real zero -> 0.0 indistinguishable. One such 0.0 has already produced a confidently wrong verdict in this subsystem. So: nothing reported -> (0.0, False) a real zero -> (0.0, True) mixed -> unreported lines skipped, never counted as zero Both keys reach the caller with no new plumbing: `_page_from_dict` forwards unknown keys by EXCLUSION (a choice made after that reassembly silently dropped `accents_repaired`, and then `model_attempts`), and both are already in the pod->orchestrator whitelist, so a page read on a rented pod reports what a local one does. Tested, including that `False` is not mistaken for the empty value that exclusion filter skips. The aggregate is extracted as `_page_confidence()` so it is testable without surya installed. Worth recording what this is NOT: on the page where the accent bug was found, surya reported ~96 % per line while every accent was a replacement character -- confident, and right to be, having emitted the correct tokens. Confidence is not a corruption detector; the byte trace is. 13 new tests. 3447 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Italian accents were lost again on surya: a measured page came back with 25 U+FFFD, **zero** 0xC3 lead bytes and a surviving 3-byte em-dash, while chandra read the identical pixels with 40 0xC3 bytes and every accent intact. So the 2-byte UTF-8 sequences were lost in DETOKENISATION, not misread from the page -- which is what `ocr.surya_repair_accents` exists to undo, and it had been shipped and verified months earlier. It had no input. The repair rebuilds the text from the model's own token IDS, the only lossless channel: vLLM 0.27.1 reports `bytes: []` for exactly the byte-level tokens that lose their characters (257 on one page, not one byte >= 0x80), so the byte fallback is dead there, and ids arrive only when vLLM is launched with `--return-tokens-as-token-ids`. That flag WAS configured -- `vllm.extra_args` on the renting install -- and the pods never got it. A pod launches its own vLLM from its own seeded config, the seeder sent only CODERAI_VLLM_ENABLE_LORA and _VENV, and there was no pod-side reader for `extra_args` or `return_token_ids` either. The pod's launch line proves it: `--trust-remote-code` present (the launcher adds that itself), the flag absent. The repair therefore had no working channel at all, and said nothing, because its diagnostics could not escape the pod either (below). Fixed at both doors, because the operator had already done their part: * `--return-tokens-as-token-ids` is FORCED for a byte-level OCR model (surya/chandra/olmOCR, matched on served name or path), deduplicated against `extra_args` so the original fix cannot pass it twice. A plain LLM is untouched unless asked. A guard that depends on every install remembering a flag is not a guard. * the operator's own `vllm.return_token_ids` now travels: CODERAI_VLLM_RETURN_TOKEN_IDS is seeded and read on the pod. And the reason a verified fix could regress in silence: * **the worker could not be heard.** `stderr=None` let it inherit the engine's raw fd 2, while a pod's log ring tees the Python `sys.stdout`/`sys.stderr` OBJECTS -- a subprocess holding the descriptor writes straight past the tee. `grep -r "\[surya\]"` across 343 drained pod logs returned nothing. It now gets its own pipe with a daemon pump that re-prints through the teed stream. NOT merged into stdout: that pipe carries the newline-JSON protocol. * **the diagnosis was opt-in.** The byte trace -- the only thing that separates "the model emitted broken bytes" from "our serving path mis-assembled correct ones" -- was gated on `surya_token_trace`, so a page full of markers came back unexplained unless someone had set a flag BEFORE the run. It now runs whenever the markers are present, which costs nothing on a clean page. * **nothing announced it.** A corrupt page now prints one line naming what was lost, whether the repair ran, recovered nothing, or is off, and the byte trace's verdict. A missing hook target is loud and on stdout. 21 new tests. 3432 passed, 13 pre-existing failures (broker, studio, manual client) untouched. NOT yet observed on a real page: the chain is proven in tests and the missing input restored, but the flag is applied at POD LAUNCH, so it takes this image plus a freshly rented pod. Co-Authored-By:
-
- 16 Oct, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authored
Three things, and the third is the reason the first two were impossible. **A mixed fleet was distributed as if every pod were the same.** `PodPool._pick` was `min(free, key=inflight)` and growth was `load / scale_up_inflight_per_pod` -- both card-blind. An A100 and an A40 were handed the same number of pages, so the fast card drained its share and idled while the slow one straggled; and a document completes only when its LAST page lands, so the slowest pod set the latency the client saw. The scaler had the matching fault in reverse: a fleet of bigger cards was still sized as if each were the preferred one, which is the same arithmetic that reached 94 pods at 10 % utilisation. Now: weighted least-connections, an in-flight cap scaled per pod, and growth that divides demand by the fleet's real capacity. **`gpu_limits`: per-card numbers, nothing multiplied.** "gpu_limits": { "NVIDIA A100-SXM4-80GB": {"parallel": 186, "queue": 744, "scale_up": 93}, "NVIDIA A40": {"parallel": 64, "queue": 256, "scale_up": 32} } A scaled single number cannot stay right: the model-level figures describe ONE card, and which card that was came from the first entry of `gpu_types` -- so changing the RENTAL preference silently re-scaled the concurrency, and a figure measured on an A40 came to describe an A100 by a factor nobody had measured. A table states what was measured, per card. The share of load is DERIVED from it (two cards with depths 32 and 93 are given work in that ratio) rather than configured twice, with `gpu_weights` and then the VRAM ratio as fallbacks for cards nobody has measured yet. Editable per model in the admin UI; a row with a card name and no numbers is dropped, because "not measured yet" must not be saved as a limit of zero. `parallel`/`queue` are applied AFTER the candidate card is chosen. The launch plan is built before, so its env carried the model-level number whatever card won -- and the gate is env, set once, so a pod rented on a 3x card spent its whole life admitting a 1x card's work. **An adopted pod recorded `gpu="(adopted)"`.** A pool that cannot name a pod's card cannot size it, weight it, or report the fleet: a whole day's status output read `by_gpu: {"(adopted)": 32}`. `list_pods`/`get_pod` now ask RunPod for `gpuCount` and `machine { gpuDisplayName gpuTypeId gpuType { memoryInGb } }` (every field probed against the live schema first), the registry stores the card and its size so adoption needs no API call, and an unreadable card costs a weight of 1.0, never a pod. **And a catalogue entry could crash an engine.** Adding a `vision_models` row for a model served on RunPod quarantined the cpu engine for ninety minutes: its id is not a path, not a URL and not a repo id, so resolution fell through to "assume a repo id", then to "assume a URL", and `requests.get("qwen38-vision")` raised MissingSchema -- five exits in 120 s, QUARANTINED, taking /v1/ocr and every pool with it while `/healthz` still answered `ok: true` because the FRONT was up. Two independent guards: * a model pinned to a backend that serves it ELSEWHERE is never resolved, downloaded or cached here -- which is also what lets one row run on rented pods and load locally on a box with a GPU, the pin being the whole difference and the section not overriding it; * a name that is none of the three resolvable kinds is refused with a sentence naming all three and pointing at `backend`. The pin does NOT make local weights useless: `resolve_model_source` is configurable (`auto | hf | url | upload`) and reads the entry directly, so an id still means "the pod downloads it" and a local path still means "push it". Tested, so a later change cannot couple the two. 43 new tests. 3411 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
**The accent regression, reported from production hours after 0.4.7 shipped.** Italian accents stopped surviving surya again. The cause was this afternoon's scan passthrough, not the old detokenisation fault: its guard used an ABSOLUTE 150-DPI floor instead of the DPI the page would have been rendered at. surya's own default is 250 (`surya_dpi`), and Aruba leaves it unset, so every scanned page holding a 150-200 DPI JPEG went to the model at native size -- 25-40 % less linear resolution than the render it replaced -- and surya-ocr-2 is measured to read badly below 250. The reasoning error is the part worth keeping: passing a scan through was always meant to be "the same pixels, without the work". Fewer pixels is not a saving. The floor is now the TARGET DPI, and `page_image_min_dpi` (default 0) can only RAISE it. Verified against a 250-DPI engine: a 150 or 200 DPI scan is rendered up as before; 250 and 300 DPI scans still pass through, which is where the shortcut was always legitimate. It also honours the REQUEST. `ocr_document` resolves `use_dpi = int(dpi) if dpi else engine_dpi(engine)` and hands that one value to the pod forward, the local render and (on the pod) this guard -- so "is this scan detailed enough" is asked about what the caller asked for, never about a default they overrode. Pinned, because a future change reading the engine default would silently hand a 200-DPI scan to a request that wanted 300. **And the rasterisation hop now encodes, instead of shipping raw pixels.** Measured on A4 at 250 DPI (6.04 Mpix, surya's own default): raw RGB over the pipe 18.1 MB + 109.7 ms for the parent to encode it for the engine anyway PNG level 1 in the worker 0.72 MB 104.7 ms, in the WORKER the parent's lazy open 0.12 ms The encode is not added, it MOVES: off the single parent process onto the raster pool, which is parallel. The pipe carries 25x less, the second encode disappears, and the parent holds 0.72 MB per page in flight instead of 18.1 MB -- at 48 pages that is 35 MB instead of 870 MB, which is the quantity this whole subsystem keeps getting into trouble over. PNG, not JPEG, unless the operator asked: a page about to be read must not pass through a lossy codec by default. Raw RGB stays the fallback, so the protocol still reads on an older parent. Seven existing tests had encoded the wrong floor and were corrected; one of them hardcoded pixel counts that assumed A4 while the fixture is 609 pt, so it measured 244 DPI while claiming 250. 11 new tests. 3368 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
coderai serves a LoRA on a vision model and cannot train one. Recorded as a plan because the gap turned up while configuring qwen38 as a vision model and is worth writing down while the reasons are fresh: what exists (loras.py is a DIFFUSION trainer; text_loras.py only serves and packages), what to build (SFT from messages-with-media, video as frames plus a token budget, QLoRA against a 4-bit base, the vision tower frozen because vLLM cannot serve an adapter on it), and which questions need a measurement rather than an opinion. No code. Nothing here is scoped to one language, document type, card or deployment size. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.4.10 fixed `_served_names` and left `_sibling_lora_specs` reading `text_models` only -- found while verifying the built image, which is the point of verifying the built image. That second function decides which adapters vLLM registers AT LAUNCH. So a QLoRA declared on a `vision_models` entry was never collected when a text request started the engine: vLLM came up with no `--enable-lora`, and then answered under the adapter's name with BASE weights. Nothing failed, nothing was logged, and the reply looked exactly like a reply from the adapter. Its own docstring records that happening once already, for the sibling reason. Both reads are now one `_catalogue_entries()`: every section, one reader, so a third caller cannot drift from these two. A test asserts `md.get("text_models")` appears nowhere in the module. Also pinned, because it decides what a client must ASK for: an entry that owns an adapter registers it under `_entry_name`, which prefers `alias` over `id` (a seeded pod entry carries an alias and no id). An entry with BOTH therefore splits -- the alias serves base+adapter, the id serves the bare base, and nothing says so. An entry that owns an adapter should carry ONE name, which is how diogene-alpha and digesta-alpha were already declared and is now how the qwen38 vision entry is too. 13 tests. 3358 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A model's TYPE in coderai is the models.json SECTION it is listed in (see main.py's _CATEGORY_TYPE_PREFIX, and `vision_models` is built from that list alone). So serving one set of weights both as text and as vision means two entries in two different lists -- which is exactly what a VLM like Qwen3.5-27B-AWQ wants, since the same pod can answer chat and image requests from one copy of the weights. `_served_names` collected the names of those weights from `text_models` only. The vision entry's name was therefore never passed to the launch, the engine did not answer to it, and `_ensure_service_once` relaunches an engine that does not answer -- which for a 27B sharing a card is not a slowdown but an out-of-memory crash, with the first engine the one that dies. The sibling function `service_key_for` had already learned this lesson ("ONE engine per set of weights, not one per name"); this is the other half of it. Now every category is scanned. Not a longer list of blessed sections: any category may name these weights, and which list an operator chose says nothing about what one engine can serve. Adapter names are still excluded (vLLM would hold a model and an adapter under one name), different weights are still ignored, a model with no adapters is still named exactly as before, and a catalogue that cannot be read still cannot stop an engine starting. 7 tests. 3352 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.4.8 moved the forward off the default 32-thread pool and the measured result on the live orchestrator was 33 -> 483 pages in flight, 1.65-2.2 -> 7.99 pages/s, and a control-plane request that used to time out at 40 s answering in 12 ms. But a thread is still the wrong unit for the wait: ~60 s per page against a real corpus means 15 documents/second at ~3.5 pages each is ~3,200 outstanding forwards, against a pod-pool ceiling of 2048. Threads cannot get there, and should not try. So the hot path is a coroutine. `pod_http` gains `apost`/`aget` on httpx -- already in the stack (codai/cluster/nodes.py, codai/main.py), so no new dependency -- with everything the sync path learned kept intact: * the pinned TLS context, unchanged: a direct-TCP pod serves HTTPS with a certificate signed by THIS install's CA at an ip the certificate cannot name, so the CA is trusted and the hostname check is off. NOT verify=False, which would accept anybody; * one client per event loop AND per pinned-ness, because a client belongs to the loop its sockets were opened on and the two trust settings must never share one; * connection limits from the same `_pool_size()` the sync session uses; * a proxied pod (http://, or https:// to a name) stays an ordinary request. `_apost_page_to_pod` and `_apod_still_alive` mirror the sync twins, and both twins now share ONE request builder and ONE response reader -- two copies of that field list is how an engine name, a dpi or a token budget comes to apply on one transport and not the other. Every distinction 0.4.2-0.4.6 established is preserved and tested on the new path: an ANSWER (even 503) is a verdict about the pod and belongs on the refusal path, nothing-came-back is PodUnreachable, a dead pooled socket is retried on a fresh one, and a timeout is never retried -- resending a page the pod is still working on is a second card doing the same work and a double bill. `ocr.pod_http_async` (default on, admin API + settings control) puts the forward back on the pod thread pool. That path is kept working deliberately, not as dead code: this is a new transport on the hottest path in the subsystem, and an operator who hits something odd should change one setting rather than wait for a release. Four existing test files had to be repaired, and the reason is worth keeping: they stubbed only the SYNC twin, so the moment the async default landed they stopped testing their own stubs and began dialling 10.0.0.1 for real -- 300 s hangs where there should have been failures. Their helpers now stub both twins via `_stub_page_post`, and the policy tests say `pod_http_async: False` out loud because they stub `pod_http.post` specifically. A test that quietly measures something else is worse than one that breaks, and this change produced four of them. 21 new transport tests. 3345 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A 512-page semaphore, a 48-wide admission gate on every pod, 31 A40s at `Running: 1` with `Waiting: 0`, and the orchestrator delivered 1.65 pages/s. Raising `ocr.max_concurrency` 128 -> 512 changed nothing. Nor did replacing the pods' 2-wide gate with 48. The reason both were inert is that neither was ever the limit: 32 threads / 19.4 s per page = 1.65 pages/s Two defects, and the second is why the first was invisible for every release this code has shipped in: * **The per-page forward ran on the DEFAULT executor.** `asyncio.to_thread` hands work to the loop's default pool -- 32 threads, shared with every other blocking call in the process -- while `codai/api/thread_pool.py` exists for exactly this call, documents it ("a per-page OCR forward to a pod (30-300 seconds each)"), and offers `run_on_pod_pool()`. Nothing called it. It was dead code next to the problem it was written for. * **Every sizing helper read a name that does not exist.** `thread_pool._configured_concurrency()` and `pod_http._pool_size()` both did `from codai.config import config_manager`; the shared manager lives in `codai.admin.routes`. The ImportError was swallowed by a bare except, so both answered 0 and took their FLOORS -- 32 threads, 64 connections -- on every install that has ever run, whatever the config said. `ocr.max_concurrency` has never once reached a thread pool. So the two knobs an operator would reach for were the two that could not help, and the number in the boot line ("default pool 32 workers") was the floor wearing the clothes of a configured value. Fixed: * the page forward and the liveness probe go to the pod pool; * sizing reads the holder that exists, tries both, and `sized_from_config()` reports which -- the boot line now says FLOORS out loud when it could not read the config, because "32 workers" means something entirely different when it is chosen than when it is a fallback nobody saw; * `_MAX_POD` 512 -> 2048: ~100 pages/s at ~20 s per page is ~2000 forwards outstanding, and each is a thread blocked on a socket. Past that an async client is the answer, not more stacks; * `configure()` grows the pod pool when `max_concurrency` changes, so raising it mid-run widens the threads and not just the semaphore. Growing only: shrinking would have to stranded threads mid-page. With max_concurrency 2048 the pools are now (256, 2048) rather than (32, 64). Credit where it is due: the client team measured 41-42 established sockets inside the container's network namespace and a 1.65 pages/s ceiling, and said the serialisation was on the orchestrator's OCR path. It was. The contrary reading here came from running `ss` on the HOST, outside the rootless netns, which can only see forwarded sockets -- a measurement of the wrong thing, reported with more confidence than it had earned. 11 new tests, including the one-liner that pins the original bug (`assert not hasattr(codai.config, "config_manager")`) and a static check that the forward is not back on `asyncio.to_thread`. 3324 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Measured on A4 at 200 DPI (3.4-3.9 Mpix), best of three: pdfium page.render() 21-46 ms PNG encode, PIL default (level 6) 86-353 ms 1846 KB PNG encode, level 1 (lossless) 123 ms 2117 KB JPEG q95 8.4 ms 596 KB the page's own embedded JPEG 0.1 ms 147 KB So most of what is called rasterisation is not rasterisation, and on the surya path it was paid TWICE per page: once in subprocess_engine to hand the page to the isolated-venv worker, once inside surya's own client to build the data URI (surya/inference/backends/openai_client.py:32, with "image/png" hard-coded into the URI it builds). Against roughly 5 s on the GPU per page -- 12 pages/min/pod at Running: 0-3 reqs, measured on two live pods -- that is CPU the pods need in order to hold 48 pages in flight. Two ways out, both shipped, both default: * A page that already IS an image is sent as the PDF holds it. A scan is usually one full-page DCTDecode object, and extract_embedded() takes it in 0.1 ms with no render, no encode and no resample. Every guard in there is a way those bytes would have been a WRONG page rather than a slower one: one image object and nothing else (a stamp or a text layer must not be dropped), DCTDecode alone (Flate is raw pixels, CCITT and JBIG2 are bitonal, JPX is JPEG 2000), grey or RGB only (a CMYK JPEG out of a PDF decodes inverted), /Rotate 0 and an axis-aligned positive matrix (otherwise the bytes are a sideways page, which reads as gibberish rather than as an error), 98 % coverage anchored at the origin, and an effective DPI within [min_dpi, max_dpi_factor x target]. * When a page must be encoded, it is encoded once, at zlib level 1. Lossless, 2.6x faster, and the bytes are carried forward so the request reuses them. The carrier is an attribute on the PIL image, recorded with the size it was true for. Any PIL transform returns a NEW object and so drops it, which is the design: a missed passthrough costs milliseconds, a wrong one shows the model a plausible page at the wrong resolution. chandra and olmOCR therefore scale BEFORE converting, since convert() was quietly invalidating every page. End to end, one page from PDF bytes to data URI: a page that must be rendered 610 ms -> 234 ms (jpeg: 68 ms) a scanned page 610 ms -> 6.9 ms, 1895 KB -> 147 KB New: ocr.page_image_format (auto | png | jpeg), page_image_jpeg_quality, page_image_min_dpi, page_image_max_dpi_factor, with the admin API and the settings form. `png` always re-encodes, which is what a controlled comparison needs; `jpeg` is lossy and stays opt-in, because whether it costs accuracy is a question about a corpus and not about a codec. The rasteriser worker loads page_image by path: it may not import codai.ocr (377 MB via litellm), and the guards must be the SAME ones the parent applies or a pool outage would change what the model is shown. ocr_worker.py may not import even that -- it runs in the engine's own venv -- so its copy of the carrier convention is tested against page_image's to keep the two from drifting. 42 new tests; 3306 passed with the 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pooled keep-alive connection that a pod closes while idle fails on its next use: pod OCR at https://.../v1/ocr is unreachable: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) _post_page_to_pod turned every transport failure into OcrError(503), 503 is in _POD_CANNOT_RUN_PIPELINE, so one dead TCP connection was read as "these pods cannot run the OCR pipeline" and set the whole pool aside for five minutes. It happened twelve times in four hours on a production run: the GPU-less orchestrator rasterised every page of every document for those windows and climbed to 40 GB at ~0.8 GB/min, while sixteen rented A40s sat idle and willing. An ANSWER may condemn an image; a socket may not. So: * PodUnreachable, a distinct error for "nothing came back", which never sets the pool aside. A pod that ANSWERS 404/501/503 still does. * One page is retried on a fresh connection (nothing was sent, so there is no side effect) and then on another pod — but a pod is dropped from the pool only after it fails to answer /healthz. Writing a pod off because an idle connection was reused has cost seven healthy cards before. * A timeout is NOT retried: the pod may be ninety seconds into a 130-second read, and sending the page again doubles the work on that card. * An active cooldown is no longer extended. It used to be, silently: every further refusal pushed the expiry out while only the first one printed, so a five-minute window could last twenty with nothing in the log. Two more ways to end up local, both of which were SILENT: * The gate compared the request's engine against the three VLM engine names. /v1/ocr has always also accepted the MODEL a caller knows in place of the engine driving it — resolve_engine maps the Surya-2 checkpoint to 'surya' and the catalogue advertises those ids as aliases — so a caller naming the model got no pod-side pipeline and no message. The switch has to be read the way a READ reads it, for the same reason it has to be read the way a LAUNCH reads it. * Serve mode not 'model', an empty model id, an engine with no pod pipeline, a standing refusal: each returned "" or None without a word. The log of the run that prompted this holds 8,278 successful pod reads and nothing at all explaining the thousands of documents in between. Every one of them now says which way the decision went, rate limited rather than said once, because a cooldown is minutes and a run is hours. And, since reading a page here means rasterising here, ocr.pod_pipeline: prefer (default) wait up to ocr.pod_pipeline_wait_s for a pod, then read it here — a pool with nothing free is usually one still booting. A timed-out wait is not paid again for five minutes, so the ceiling cannot become a per-page tax. auto read it here at once (0.4.1-0.4.5 behaviour). require never rasterise on this box: 503, saying why. Tests: 32 new (tests/test_ocr_pod_pipeline_policy.py) covering the alias, all four silent routes, the three policies and the transport/answer line. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.4.3 got placement working on a pod: it resolved the model and minted an attach lease, which it had never managed before. Then surya said: SURYA_INFERENCE_URL=http://127.0.0.1:8000/v1/ocr/attach/<lease>/v1 is not reachable at /health ...about an address the pod was answering perfectly well. A pod on the direct path serves HTTPS on its ONLY port, and an OCR engine there is a SUBPROCESS: it attaches by URL and goes through the socket like any other client. self_base() handed it `http://`, so it probed a plain request at a TLS listener. Fixed with verification ON end to end, not off: * the leaf certificate now carries DNS:coderai-pod, DNS:localhost, IP:127.0.0.1 and IP:::1. It had one SAN, `coderai-pod`, that the comment said was "never checked" -- true for the orchestrator, which reaches a pod by a bare IP with the hostname check off. A pod reaching ITSELF has no such excuse, so it uses a name the certificate actually holds. * boot.sh maps 127.0.0.1 coderai-pod and exports CODERAI_SELF_BASE=https://coderai-pod:8000; the plain-HTTP case (behind the RunPod proxy) exports the loopback http base explicitly instead of relying on a default. * the CA's PUBLIC certificate now travels with the leaf, appended to CERTIFI'S ROOTS into one bundle. That order is not cosmetic: a bundle that lost the public roots would break every HuggingFace download on the pod, which is worse than the bug. Exported only once built and non-empty, and it says so plainly when it cannot be. This covers chandra and olmOCR too -- in served mode they attach through the same gateway URL from the main venv, so one trust store serves all three. Co-Authored-By:
-