- 21 Oct, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authoredEvery tool-using request to a pod-served model answered an EMPTY assistant message: content "", finish_reason "stop", no tool_calls. Asked directly the pod answered the call it had generated; through the orchestrator the same request answered nothing. Both the LoRA (diogene-alpha) and the base model were affected, so no adapter was involved. A proxy backend's generate_chat returns a STRING. The remote server owns the chat template, so it is the side that recognises a tool call and reports it structurally -- `{"content": null, "tool_calls": [...]}`. `message.content or ""` dropped that on the floor; the caller then had an empty string, found no call to parse in it, and answered the client with an empty message. The calls now travel beside the text, where usage and logprobs already do: get_last_tool_calls() on the backend, forwarded by the manager, preferred over re-parsing text in both the streamed and buffered chat paths. A backend that generates locally still keeps its call inside the text and answers None, so that path is untouched. Streaming needed the same: a streamed call arrives as deltas with no content at all, so merge_tool_call_deltas folds the fragments -- keyed by the call's index, never by arrival order, since concatenating the arguments of the wrong call yields valid JSON for a call nobody made. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
-
- 20 Oct, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
2026-10-20, qwen38: a restart generated a fresh random pool token (none is configured), adopted a pod still carrying the old one, and rented a second pod with the new one. A pool has one api_key, so whichever token it held answered 401 on the other pod — "Invalid API key" on every request until both pods were destroyed. The adoption guard ("only swap while we hold no pod") could not help: the rent and the adoption happened in the same minute. - PodHandle.api_key: the token THIS pod was launched with, or adopted carrying. pool.pod_key(pod) resolves it, and the pool remembers the tokens of pods it no longer holds so a quarantined one stays reachable. - Adoption keeps the adopted pod's token on that pod instead of replacing the pool's, so pods with different tokens coexist. - _acquire_endpoint returns (base, release, bearer): the key is handed out WITH the address. Reading it afterwards from the pool was wrong twice — the pool's token could change mid-acquire, and _last_pod is one attribute shared by every concurrent request, so it could name another request's pod. - The OCR worker's majority vote over the registry's keys is gone; each pod's key rides on its own handle. - The pool's own token is minted once and persisted beside the pod registry (0600), so a restart no longer orphans every pod it had rented. A configured api_key still wins, and an unwritable directory still serves. Also tried and reverted: treating an orphaned mid-boot pod as "on the way" to stop a restart renting beside it. That reintroduces the twenty-minute freeze of test_runpod_restart_orphans. Renting twice costs cents; a pool that will not rent serves nothing. The reasoning is recorded in sibling_is_provisioning. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
RunPod's live GraphQL limits for our key are 180/min, 3,600/h and 43,200/day. The day limit is the one that ran out, and the ban lasts until midnight UTC. Averaged over a day it allows 1,800/h, so the hourly cap is not the binding one. This process now counts its own GraphQL calls over a rolling hour. Past CODERAI_RUNPOD_GQL_HOURLY_BUDGET (default 1500; 0 = off), _gql_banned() reports true, so everything REST v2 can do goes to v2 before RunPod ever answers 429. The budget only reroutes. It never refuses a GraphQL call on its own, so spot rentals, which v2 cannot do, still work. The first time the budget is reached, and every 10 min while it stays reached, it says so. Co-Authored-By:
-
- 19 Oct, 2026 32 commits
-
-
Stefy Lanza (nextime / spora ) authored
Verified against RunPod's v2 OpenAPI spec and live headers on 2026-10-20. GraphQL quota is 180/min, 3,600/h, 43,200/day; ours read x-ratelimit-remaining-day 0 with Retry-After 3939. REST v2 has a separate per-user quota (180/min, 7,200/h, 86,400/day) and read "day";r=86399 at the same moment. v2 then terminated a pod (204) that REST v1 (500) and GraphQL (429) could not. v1 is deprecated and retires 2026-11-15. - All REST traffic now goes to https://api.runpod.io/v2: get/list pods (with pagination), terminate (DELETE, 404 counts as gone), create (POST /v2/pods with gpu{id,count}, cloud, disk, ports, env, entrypoint/cmd, dataCenterIds, mounts persistent/network/global), network volumes, and the priced GPU catalogue. - v2-first with GraphQL as fallback for the high-volume reads (status, listing, boot polling) and for terminate. GraphQL stays primary for create and the catalogue because only it offers spot bidding and spot prices. They switch to v2 during a GraphQL ban. v2 refuses spot rather than renting it on-demand. - v2 phases map to the pool's words (PROVISIONING/STARTING -> RUNNING, ERROR -> FAILED). Runtime memory and CPU utilisation are kept per pod. - v2's own 429 and Retry-After are honoured, and its live RateLimit header is kept. - The global volume is the documented mounts.global on v2. A refused one is retried without it, the same guard as the GraphQL path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Observed through the 3.3 h ban: REST reads kept working, but every REST write (create, and a body-less DELETE) failed with 500 "unexpected end of JSON input". REST writes appear to go through RunPod's GraphQL backend and share its rate limit, so REST cannot rent around a ban. The fix is not to earn one. - wait_ready polled GraphQL get_pod every 5 s per booting pod, which is hundreds of calls a minute in a 64-pod scale-up. Direct-TCP pods (the default) now poll REST and open a TCP connect to the mapped port. RunPod refuses that connect until the server inside listens, which is a stronger signal than GraphQL listing the port at container start. GraphQL stays the fallback outside a ban. - list_pods (the reaper and the survivor sweep, every minute) is REST-first, with includeMachine=true. Card name and VRAM come from the catalogue. - GraphQL calls are counted by operation and the rate is logged every 10 min, so the volume is visible before RunPod bans the key. - A REST write that fails this way during a ban says why. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod rate-limited GraphQL for ~3.3 h on 2026-10-19 while REST kept answering. Every pool operation stopped: renting, the GPU catalogue that prices cards, watching a pod boot, the reaper, and teardown. While the ban (v0.4.54's backoff) is in force: - create_pod goes to REST. That fixes _create_pod_rest, which used direct_tcp and global_volume_id without accepting them, so it had never worked. - get_pod, list_pods and terminate_pod use REST and return the same shapes. REST portMappings and publicIp become the GraphQL-style ports list. A 404 on DELETE means the pod is already gone, which counts as success. - list_gpu_types returns the last catalogue saved to disk. It is now saved beside the pod registry on every successful fetch. An unpriced card is still never chosen, so ceilings hold. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Measured on a rented A40 under 128 pages in flight. The engine stopped answering. _ensure_service_once treated any service that failed a 3 s health probe as gone: it dropped it from the table, without stopping the process, and launched another. A second caller during a start did the same, so two EngineCores came up one second apart. For twenty minutes vLLM was relaunched every few seconds and killed with -9 mid-start. This is also the earlier "10.22 of 47 GiB free" crash. - A service records "starting" until it is ready. Another caller waits for it (_StillStarting -> _wait_for_start) instead of launching a duplicate. - A live process that misses the probe gets a second 15 s probe. If that fails too it is BUSY and is handed out, not replaced. - A dead one is stopped properly before it is replaced. - The OCR gateway's /health no longer runs placement.any_candidate (which can start vLLM) on the event loop. Every surya engine probes it, and it froze a pod for up to 408 s. The no-blocking-calls guard now covers placement.any_candidate and placement.candidates, and fails on the old code. - RunPod 429s are honoured: retryAfter is in seconds (observed 11,871, about 3.3 h), and no GraphQL call is made until it passes (capped at 6 h). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A page holds its thread for the whole read, a minute or more. _Pool.recognize used asyncio.to_thread, which puts that wait on the default executor: capped at 256 and shared with model lookups and the admin pages. On a rented A40 the engine had 512 lanes and vLLM sat at 9% KV cache, so the thread pool, not the GPU, set the pod's ceiling. Past 256 the model lookup each page's chat request needs (off the loop since v0.4.52) would also have queued behind the page waits. Page reads now use the pod pool. It is sized from ocr.max_concurrency (up to 16384, within the task budget), and on a pod nothing else uses it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The v0.4.51 stack dump named the OCR pod freeze. chat_completions called multi_model_manager.get_model_for_request() on the loop. For a model not loaded yet that goes _load_default_model -> load_model -> vllm_worker.ensure_service(), which waits for vLLM in a time.sleep(2) loop. On a pod it froze the whole process for 324 s at boot, /healthz included, and for minutes whenever it decided to (re)start vLLM, so the orchestrator took busy pods for dead ones. chat_completions and completions now resolve the model in a thread. The manager's load lock already prevents duplicate loads. The no-blocking-calls AST guard now also forbids get_model_for_request, _load_default_model and ensure_service in a coroutine; it fails on the old code and passes on the new. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Rented OCR pods froze for 108-784 s under load ("[loop] blocked for 784.6s"), and there is no shell on a pod to attach a profiler. A daemon thread now watches the sampler's heartbeat. After CODERAI_LOOP_DUMP_AFTER seconds (default 3) of silence it prints the loop thread's current stack, again every 30 s while the stall lasts, at most 20 per hour. It exits when its loop closes. If the top frame is the selector, the loop was starved of the GIL rather than blocked in a call. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
require only bound the lease. A page that timed out on a pod, or a pool that refused, still sent the whole document to the local pipeline. Live, that was 4 surya processes at ~9 GB each and swap full on a GPU-less orchestrator, with those pages slower than a retry on a pod. Under require, that case is now a 503, so the client retries and the page goes back to the cards. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A single pod whose vLLM failed to start (another vLLM still held 37 of its 47 GB) answered 503 to every page. _read_on_pods set the whole pool's pod pipeline aside for 300 s on that one answer, so every page in that process was read on the orchestrator. Live, that meant ~10 pages/s from 34 pods, with the orchestrator's CPU doing the reading. Now a refusal takes that pod out of rotation (lease.dead -> quarantine) and the page tries another pod. The pool is set aside only when a page's attempts were refused by at least two different pods, i.e. when it looks like the image and not one card. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
One Python process forwards ~16-23 pages/s however many pods it has. Measured on a 35-pod fleet: front 99% CPU, engine 101%, 14 of 16 cores idle, pod GPUs at 0-7% KV cache. 128 pods at ~0.65 pages/s each need ~85. ocr.workers (or CODERAI_OCR_WORKERS) = N starts N --engine-only processes (codai/ocr_fleet_workers.py, a supervisord program) and writes nginx includes that send exact /v1/ocr and /v1/ocr/batch to them, least_conn on a shared zone, front as backup. Everything else still goes through the front. 0 (the default) leaves routing exactly as before. The main engine stays the only process that rents, scales, quarantines, adopts or terminates. Workers (CODERAI_OCR_WORKER=i/N) follow the pod registry, take a balanced sticky slice (runpod_fleet_share.assign), and report per-pod in-flight/used/suspect back. The controller merges those reports into every scale/idle/teardown decision through _load(). Drains go to workers as flags. Each worker takes 1/N of the per-process ceilings (max_concurrency, max_queued_pages, *_instances, rounded up) and never writes config.json or models.json. nginx retries another worker only on a connect error, never on a timeout (that would read a page twice, billed twice), with no try cap so the backup is reachable. worker_connections 1024 -> 16384. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
v0.4.46's sweep loops "while _adopt_shared_pod() adopts something". find_shared_pod knew nothing of the pool calling it, so it returned the same pod every iteration; _add_pod ignored the duplicate, the call still returned True, and the sweep never reached the next candidate. Live, on the deploy that introduced it: one pod "adopted" 15-18 times in a second and four pods left untracked behind it, each wasted pass costing up to two 15-second probes inside the maintenance loop. find_shared_pod now takes the caller's own pod ids and skips them, and adoption reports True only for a pod the pool did not already hold. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
test_an_unbounded_budget_leaves_the_sizes_alone assumed the pod pool always reaches _MAX_POD; since v0.4.45 the ceiling is 16384 and an unbounded budget gives exactly the configured 2 x 4096. And the env-sizing test from v0.4.43 passed alone but not in the suite, because an earlier test left a config manager set; it now clears it itself. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three defects, all measured on a live 32-pod fleet on 2026-10-19. QUARANTINE KEPT HEALTHY PODS OUT. The re-check skipped its probe while a pod had pages in flight -- meant to stop a busy pod being terminated, it also stopped it being RE-ADMITTED. A pod that tripped once on a connection got no new pages until all 64 drained, minutes later. Two of four serving pods sat quarantined while answering /healthz in 0.1 s. Now the probe always runs and a pod that answers is back at once. A HUNG POD HELD ITS PAGES FOR 15 MINUTES. The same rule meant a pod that answered nothing was never terminated while it had pages charged, so they waited for the in-flight timeout. 64 pages sat on one. Now three failed probes (~2 min) terminate it regardless; busy pods answer /healthz in a tenth of a second at full load, so this cannot catch one. A RESTART UNDER LOAD ORPHANED 29 PODS ($17.11/hr). Adoption happened only on demand -- a warm floor, or a waiting request -- and the restart dropped the client's lease, so nothing asked. And when asked, the probe was /v1/models with a 4 s timeout, through an engine still finishing 64 pages. Now every maintenance pass sweeps the registry (rate-limited to once a minute) and adopts every surviving pod whatever the floor says; the engine probe gets 15 s, and a pod slow there but alive on /healthz is adopted when the token it was launched with is on record. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A 128-pod fleet at 64 pages per pod is 8192 pages on the cards, and keeping the queue full behind them doubles that. Three constants each capped the chain below it, and the lowest one decides the fleet's speed whatever the others say: front -> engine forwarding pool _FWD_MAX 8192 -> 32768 engine -> pods async client _MAX_FLEET_POOL 16384 -> 32768 pod thread pool (sync transport) _MAX_POD 8192 -> 16384 All three are still sized from ocr.max_concurrency rather than set to their ceiling, and the pod pool is still clamped to the process's task budget, so this only stops a constant from being the binding limit. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`_pool_size()` is a PER-POD figure -- what `requests` means by `pool_maxsize`, which keeps one pool per host. The async client that forwards every page to every pod is ONE httpx.AsyncClient for the whole fleet, and httpx's `max_connections` is a TOTAL across hosts. Passing the per-pod 256 to it capped every page forward in the install at 256. Measured 2026-10-19 on a 31-pod fleet: 1,583 pages holding a pod lease, 284 TCP connections to pods in total -- six pods with 25-49 connections, most with exactly one. Renting more pods added cards and no throughput. Pages that waited too long for a connection failed with an empty-message timeout, reported as the pod being unreachable, and healthy pods were quarantined for it. The async client is now sized from the page ceiling the install is configured for (ocr.max_concurrency + headroom, capped at 16384): 5120 at the current 4096. The sync per-host pool is unchanged; per host is the right meaning there. An existing test asserted the shared client's limit equalled the per-pod figure -- it pinned the bug, and now asserts the fleet total. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod has no config.json -- its settings arrive as env vars -- and the thread pools are built before the config manager has read them. So on every pod `_configured_concurrency()` returned 0 and the default pool took its floor of 32 workers. Every page on a pod runs in that pool. The pod said so in its own log, every boot: [threads] default pool 32 workers, pod pool 64 (FLOORS -- the config was not readable from here yet ...) [vllm] Maximum concurrency for 32,768 tokens per request: 72.20x [ocr] engine 'surya': 8 instance(s) ready, 64 lane(s) each Measured 2026-10-19: 64 pages dispatched per pod, `Running: 32` on every card, GPU KV cache at 4-10%. Raising the orchestrator's dispatch width could never help; the cap was inside the pod. The sizing now falls back to CODERAI_OCR_MAX_CONCURRENCY, the env var the orchestrator already sends every pod, so a pod sizes its pools the way an install with a config file always did. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Operator decision. A pod that refuses a connection is still quarantined and re-probed rather than terminated (v0.4.37), but its dispatch width is no longer cut in half. The queue in front of the fleet absorbs the surplus; narrowing the pod only starved a card that was measured at 11-13% KV cache with nothing waiting. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The previous revision told a client to retry 404 and 502 indefinitely. That is wrong and would spin forever in two real cases: a document that reliably 502s never succeeds and blocks the corpus behind it, and a 404 that is not a restart (wrong URL, route removed) never clears. Both end with pods billing and nothing being produced -- the exact failure the advice was written to prevent. It was an over-correction. On 2026-10-19 a restart answered 618 requests with 404 over ~40 seconds, the client treated 404 as fatal and stopped, and nine pods sat ready at $5.31/hr until a human noticed. The fix for that is "do not treat 404 as fatal immediately", not "never give up". What separates the codes is what each says about the server: 429 healthy and full, and it stated when to return -> honour Retry-After, no attempt cap, alert if 15 min pass with no page completing. 404 restarting -> retry with backoff for up to 5 minutes, then STOP and alert. A route that is gone does not come back by being asked again, and a real "no such endpoint" would have answered 404 to the first request rather than the ten-thousandth. 502 the pod failed this page -> 5 attempts, then PARK the page and move on. ~3-5% of pages 502 at least once; one unparkable document must not stop a million-page run. 400/413 the document is the problem -> park, never retry. Plus a circuit breaker, which matters more than any per-page rule: stop feeding if more than 20% of the last 200 pages ended parked or erroring. No per-page policy can see "the fleet is costing money and producing nothing"; only the aggregate can. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
v0.4.40 fixed the WRITER: a boolean sent through the pod env builder's int loop was dropped, because `False == 0`. The reader had the same bug in mirror image. The env row was declared with `bool` as its cast: ("CODERAI_OCR_EARLY_LOOP_ABORT", "early_loop_abort", bool) and the typed loop does `_cast(_val)` -- so `bool("0")` is True, and a pod sent `CODERAI_OCR_EARLY_LOOP_ABORT=0` turned the feature ON. One flag, two opposite coercions, both wrong in the same direction: it could only ever be enabled. Caught by the A/B test it was blocking -- the "abort off" run reported 8 aborts. Booleans belong in the `_flag()` loop, which reads the STRING ("1", "true", "yes", "on" and nothing else), because that is the only way an environment variable can express false. Moved there, with a guard that no CODERAI_* env row anywhere casts with `bool`. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three additions from a live run on 2026-10-19. A 404 ON /v1/ocr/batch MEANS THE SERVER IS COMING BACK UP, not that the endpoint is gone -- a real 404 would have been there from request one. A restart answered 618 requests with 404 over ~40 seconds, the client treated it as fatal and stopped feeding, and nine pods sat ready and billing $5.31/hr until a human noticed. The guide now gives the retry policy explicitly: retry 404 and 502 indefinitely with backoff, 429 after Retry-After, and only other 4xx retires a page. The rates section says how its numbers were taken and what they are not. 0.557 pages/s per pod is concurrency / mean latency from a controlled bench; the same run yields 0.222 and 1.242 by two other defensible definitions, and only the first extrapolates. It also records that a live run got 0.051 per pod at identical settings, cause not yet known, so the figure is not read as a promise. And the status fields worth watching, including the three added today: pods_quarantined (set aside and coming back, neither gone nor booting), pods[].capacity / soft_limit (what each pod is actually being fed), and live_hourly_booting_usd (the share of the bill that is paid boot). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod env builder filters out any value equal to 0: if _val not in (None, "", 0, 0.0): env[_var] = str(_val) and in Python `False == 0`. A boolean sent through that loop can only ever be switched on: `ocr.early_loop_abort: false` was silently discarded and the pod ran with the feature enabled. That is the worst shape a configuration bug can have -- the setting is in the config, the operator believes it applied, and the behaviour is the opposite of what it says. It is also how this was found: the early loop abort could not be A/B tested at all, because there was no way to send a pod "off". There was already a boolean loop beside it, writing "1"/"0" explicitly for `surya_repair_accents` and `surya_token_trace`. The new flag simply was not added to it. It is now, along with the three thresholds -- which ARE ints and belong in the int loop, since 0 there properly means "use the default". The int filter is deliberately unchanged: for an int, 0 means "use the engine's own default", and forwarding it would override that default with a meaningless zero. 7 tests, including a guard that no boolean declared on OcrConfig is sent through the int loop, so the next one cannot repeat this. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A VLM that loses its place on a page repeats the last thing it saw until it runs out of tokens, and the result arrives looking like a successful transcription. Every engine here already runs `detect_repeat_token` over the FINISHED reply and treats a looping one as retryable -- the waste is that the reply has to finish first, so a page that starts looping on its tenth token still emits its whole budget before anyone looks at it. Measured on 100 pages of a real judicial corpus (surya, A40, cap 4096 / 1 continuation): 4 pages looped and accounted for 18.9% of the run's entire generation time. The worst spent 450 s emitting 17,586 characters of one repeated clause. Replayed through the shipped guard: 19.3% of generation time not spent, 0.560 -> 0.690 pages/s (+23%), worst page 450 s -> 67 s. Over all four measured rungs, 12 of 12 looping pages caught and ZERO false positives across 366 clean pages. WHY THIS IS SAFE TO HAVE ON BY DEFAULT: it is the same function with the same thresholds that already judges the finished reply. A page abandoned here is a page the post-hoc check would have rejected anyway, reached sooner. Nothing new decides anything. Two measured margins against cutting a page that merely LOOKS repetitive for a while: nothing is judged before 2000 characters (the largest genuine page in the corpus is 5,054, median ~1,900), and the detector must fire on 2 consecutive checks. Swept over 393 page reads -- at 1200/1 there was one false positive, a page with a repetitive middle and genuine text after it; at 2000/2 none, with all 12 real loops still caught. Closing the stream is the point: the server cancels the sequence when the client goes away, so the tokens not waited for were never generated. chandra and olmocr own their HTTP call, so they stream it directly. surya does not -- it builds and sends its own request -- so the CLIENT it is handed is wrapped instead, in the worker hook we already own, using SURYA'S OWN detector so the abort and surya's retry decision can never disagree. The worker keeps its no-codai-imports rule. Four defects found while writing it, each worse than the feature: * one check per CALL made a persistence above 1 unreachable for a caller that fed a whole reply at once, and made every verdict a function of the server's chunk framing. Now one check per step boundary crossed. * falling back to a blocking call after a failed stream re-ran the page: two generations billed, and the second reply mistaken for the continuation of the first. Only a failure that consumed NOTHING is retryable now (StreamUnsupported); a server that ignores `stream` has its complete body read rather than discarded, so the common case costs no extra call at all. * a local `from codai.ocr.base import OcrError` shadowed the module-level one for the whole function, so `except OcrError` raised UnboundLocalError and swallowed a mid-stream transport failure as retryable -- the one case that must never be retried. * the surya hook installs on `_trace or _repair`, so the abort depended silently on `repair_accents`. Now named in that condition. A server proved unable to stream is not asked again (one wasted request per server per process, not one per page). 4xx to a streaming request means "cannot stream" and falls back; 5xx is a real error and is not retried against a broken server. 29 tests. Full suite 3849 passed, 15 pre-existing failures unchanged. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reverts the advice in 8b976170, which was wrong. A pod serving ~32 pages at once is a fact about what the SERVER should dispatch to one pod, not about what the CLIENT should offer. Offering only what the pods can hold means that the instant a page finishes there is nothing queued to replace it, so every pod idles for a round trip between every page -- losing exactly the throughput the queue exists to protect. The split is now explicit in the guide: the client offers 64 x pods, the server dispatches ~32 to each pod (discovered from the pod, v0.4.38) and holds the rest queued and ready. The client cannot overload a pod by offering more, and does not need to track what a pod can take. Admission is sized for exactly that offer: 4096 dispatched + 4096 queued = 64 x 128 pods, with 429 only beyond it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Telling a client to send exactly 32 x pods was me asking it to compensate for a server that could not absorb being wrong. v0.4.38 bounds per-pod dispatch from what each pod will accept and answers 429 with Retry-After and the four live numbers when the queue in front of the fleet is full. The guide now says that: 32 x pods is the efficient window, not a requirement, and a 429 is a signal rather than a failed page. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A client must be able to send too much. The server's job is to carry what it can and say so about the rest -- not to depend on the client having been told the right number, which is what the previous fix amounted to. TWO THINGS WERE MISSING. 1. NOTHING BOUNDED WHAT ONE POD WAS SENT. `max_inflight_per_pod` defaults to 0, meaning unlimited, so out of the box every page handed to the pool is dispatched. A client offering more than a pod can serve therefore does not queue in the orchestrator, where the queue would be counted, reported as demand and bounded -- it queues in the POD'S LISTEN BACKLOG, and an overflowing backlog makes a healthy pod refuse connections. That is what cost eight A40s yesterday. The right width is a property of the pod, the model and the card, not of the install, so the pod is asked rather than configured. A refusal halves what we send that pod, FROM THE IN-FLIGHT COUNT AT THE REFUSAL -- its observed breaking point, not a guess -- and successes walk it back up one slot at a time (multiplicative decrease, additive increase: overshooting costs refused connections and a misdiagnosed fleet, undershooting costs a little latency). Floored at 4, never above the operator's own ceiling, per pod so one overloaded pod does not throttle the fleet, and inert on an install where no pod ever refuses. Live numbers: 64 configured, 58 in flight at the refusal, width becomes 29 -- the pods were measured serving 29-32. 2. A FULL QUEUE ANSWERED SLOWER INSTEAD OF SAYING NO. The queue itself is deliberate: when a pod finishes a page there must already be another ready to hand it, or the fleet idles between dispatches and is paid for the gap. So the server carries `ocr.max_concurrency` in flight PLUS `ocr.max_queued_pages` waiting (0 = follow max_concurrency) -- and past that answers 429 with Retry-After and the numbers a client needs to size its window (X-OCR-Page-Width, X-OCR-Queue-Limit). Before, a client offering several times the capacity got no error at all: it got replies that took longer and longer, which reads as "the pages are getting slower" rather than "stop sending", so it sent more. Measured 2026-10-19: 522 pages offered against a fleet carrying ~250, mean page latency 238s against a 19s service time, nothing in any reply saying the server was over capacity. Refused BEFORE the upload is read. The client hears at once instead of after transferring a document the server will not look at, and the server stops buffering megabytes it is about to discard -- in-flight memory here is concurrent requests times upload size, the product that has OOM-killed this endpoint before. Every page now takes its slot through one counted path, which is what makes the queue depth knowable. Writing that exposed a real defect in what it replaced: `self._sem or asyncio.Semaphore(4)` built a FRESH semaphore per page, so an unconfigured manager bounded nothing at all -- forty pages of one document ran at once against what every line of code called a limit of four. Caught by the existing pod-pipeline gate test. 38 tests. Full suite 3817 passed, 15 pre-existing failures unchanged. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The guide said `in_flight = 64 x pods_ready`. 64 was the configured ceiling; the measured capacity is ~32 (`Running: 29-32 reqs, Waiting: 0` on every pod of a live fleet), and the 0.557 pages/s per pod the guide quotes was itself measured at a concurrency of 32. Offering 64 does not queue inside the orchestrator. It queues in the pod's socket backlog, and an overflowing backlog makes a healthy pod refuse connections -- which on 2026-10-19 got eight A40s terminated mid-work with 522 pages on them. v0.4.37 stops the termination; this stops the overload that triggered it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`discard()` terminated the pod, on the reasoning that a connection can only be refused by a pod that no longer exists. A pod whose listen backlog is full refuses connections while being perfectly alive and busy -- and refuses them on /healthz too, so the "is anything still there?" probe added to protect against exactly this agrees it has vanished. Observed 2026-10-19, mid-campaign, ~58 concurrent connections per pod: 08:36:13 pod 2n69g0vt47h0hl terminated (connection refused) 08:36:30 pod c79lv31ayh28l1 terminated (connection refused) ... eight healthy A40s in 69 seconds ... 08:37:22 pod n9fpbskd7owl2h terminated (connection refused) 522 pages were in flight on them. The warm floor then re-rented eight more, which booted ~12 minutes, began refusing, and were killed again: $3.72 of pods billed -- more than the campaign's whole spend -- on boots that never produced a page, while throughput read 0.577 pages/s against a measured 0.557 PER POD. A pod is now QUARANTINED instead: out of rotation, still ours, still counted against max_pods because it is still billing. `maintain` re-probes it when the quarantine expires (45s) and * it answers -> straight back into rotation, which is the common case and the whole reason this stopped terminating; * it has work in flight -> left alone whatever the probe says, and it accrues no strikes while excused, so it does not die the moment the last page drains. Killing a pod with pages on it loses them AND pays for them; * it fails 3 probes with nothing in flight -> terminated. ~2 minutes of a pod that exists, bills and will not answer while carrying no load is dead in every sense that matters, so this does not trade a terminating bug for a leaking one. A quarantined pod also counts toward the warm-floor deficit. Without that, eight set-aside pods read as a deficit of eight and the pool rents ~700s of boot to cover a 45s gap, ending with two cards where the floor asked for one -- the re-rent loop in one line. The snapshot reports `quarantined` as its own state with the seconds remaining, and `pods_quarantined` in the summary. It is NOT counted as booting, which told an operator a settled fleet was still starting and told a client sizing from pods_ready to wait for pods already present. `discard` has exactly one caller (RunpodBackend._unreachable, reason "connection refused"), so nothing else loses a termination it needed. 19 tests. Full suite 3777 passed, 15 pre-existing failures unchanged. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The front forwards to an engine over a client built with `pool=None` -- wait forever for a free connection -- and httpx defaults to `max_connections=100`. Together those two defaults cap everything the front forwards at 100 in flight and make the 101st wait indefinitely. Nothing anywhere reported a limit, because nothing had set one. Measured on a live orchestrator mid-campaign, client offering 512 pages: established to the front 514 forwarded to the engine 100 <- the cap pages/s 0.42 (fleet sized for ~3.3) mean page latency 238 s (service time measured at ~19 s) cost per page 12x the measured baseline The pods were rented, healthy, and unreachable. The 200 s of that latency that is not service time was pages queueing for a connection. It also wedged the control plane, which is how this was finally found. /v1/runpod/status is not under /admin/api/, so it took the same unbounded path and never answered at all -- http=000 at 30 s, three times in a row, while /healthz answered in 2.1 ms. Two consequences, both worse than a blind admin page: a lease could not be RELEASED while the fleet was busy, so pods kept billing; and a client sizing its in-flight window from `summary.pods_ready` -- which is what the throughput guide tells it to do -- was flying blind as a direct result of the thing it was measuring. * the forwarding pool is sized from `ocr.max_concurrency` plus headroom, floored at 256 and capped at 8192. An unreadable config falls back to the floor, never to httpx's default: silently reinstating the limit this removes would be the worst possible fallback. * waiting for a connection is bounded at 30 s, so exhaustion surfaces as a retryable 502 rather than a socket that never answers. The READ timeout stays unbounded -- a page may legitimately take minutes, and confusing the queue with the work would truncate real pages. * control-plane paths (fleet status, scale, release, healthz, metrics) take a separate 16-connection pool that request traffic may not use. A pod that cannot be released because the release cannot get a connection keeps billing. This supersedes part of v0.4.34's diagnosis. The control pool and the bounded snapshot there were real fixes to real defects, but they were in the ENGINE app, and the wedge was in front of it. 18 tests. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The throughput guide told a client to lease pods and said nothing about bounding what that lease may grow to, because there was no way to. Adds the staging recipe (one call per stage, no restart), what the three $/hr figures now mean and which one to alarm on, and that a `degraded` status document is stale rather than an idle fleet. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Found by exercising the new ceiling against a live orchestrator: max_pods=128 effective_max_pods=8 client_max_pods=8 client_pods=0 client_lease_expires_in_s=0 The lease had 1800 s left and was the reason the fleet had stopped growing at 8. `client_lease_expires_in_s` was gated on the FLOOR, which is 0 for a cap-only lease -- "grow no further than 8, hold nothing warm", which is what a client with a long tail of work asks for. It now reports whichever half of the lease lasts longer. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/runpod/status is the one surface an operator has for seeing WHY a server is busy. During a pod boot wave it was both unanswerable and, when it did answer, wrong about the only number anyone was watching. IT COULD NOT ANSWER. Measured on a live orchestrator: /healthz 200 in 2.9 ms (no thread needed) /metrics 200 in 180 ms (no thread needed) /v1/runpod/status no answer in 25 s (needs a thread) /v1/runpod/scale no answer in 25 s (needs a thread) Nothing was broken. Those two are the only endpoints that hand work to a thread, and they drew it from the DEFAULT executor -- the pool the request path also draws on. 25 s of silence reads as a dead server, and it also meant a client lease could not be RELEASED while the fleet was busy, so pods kept billing with no way to stop them but the console. Three changes, each closing one way that happened: * a CONTROL pool (16 threads, fixed) that nothing in the request path may use -- status, scale and release run there. Its whole value is being empty when everything else is full, and a test pins that no request-path module touches it. * the PDF split moves to a two-wide PDFIUM pool. pdfium admits exactly one caller, so hundreds of concurrent documents each took a general thread and then queued on the same global lock: the serialisation was unchanged while the threads they held were the control plane's. Queueing two-wide costs nothing that was not already being paid. * the snapshot's three sources now run concurrently, each with its own 5 s deadline, and whatever is late is reported as `degraded` rather than waited for. pools_schedule_status() was the half never bounded in v0.4.21 -- pods_status() gave up on a busy pool and this one waited for it -- and the ledger snapshot reads without the lock it shares with the writer that fires on every termination. Enforcement still takes that lock properly: only the snapshot may degrade. IT COULD NOT BE BELIEVED. `live_hourly_usd` summed only HEALTHY pods. RunPod bills from creation and a pod spends ~12 minutes booting without ever being healthy, so a fleet entirely in paid boot reported as free: reported $0.59/hr actual $4.72/hr reported $0.00/hr actual $7.67/hr (13 pods, 8 serving) A client polling it concluded nothing was being spent and kept feeding work. It now counts every pod in the listing, and reports the serving and booting halves separately -- their ratio is how much of the bill is boot, which is the figure that says a ramp is too steep, and it did not exist before. Both admin pages show it; "0 pod(s) · $7.67/hr" was a real reading and is self-contradictory. 29 tests. Full suite 3741 passed, 15 pre-existing failures unchanged. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/runpod/scale set a FLOOR. "Give me 8 pods" meant "hold at least 8 ready", and the autoscaler stayed free to grow the pool to the configured max_pods for as long as work kept arriving. A client running a staged batch -- 8 pods for the first stage, 32 for the next -- had no way to bound a stage at all, and nothing in the reply said so. Observed 2026-10-18: a run that asked for 8 pods was at 13 and climbing toward 128. Every pod past the plan is ~700s of paid boot producing nothing, which is why the realised cost per page came out 16.5x the measured baseline while every component behaved exactly as written. A lease may now also carry `max_pods`, under the same rule `gpu_types` already follows there: it may only ever NARROW the operator's ceiling, a larger number is clamped and reported in `notes`, and it dies with the lease. `pods: 0` with a `max_pods` is a cap with no floor -- what a client with a long tail of work wants; `pods: 0` alone releases both. The ceiling has to be read by EVERY path that decides whether to rent, or it is a race between the grower that honours it and the one that does not: _pods_for_demand, _grow_reason, _pods_needed and the warm-floor top-up all now read effective_max_pods(). A leased floor is clamped to it too, since a floor above a ceiling is a fleet that oscillates for as long as the lease lasts. A configured max_pods of 0 still means zero -- that is the operator having turned renting off, not a lease. 22 tests, including a guard that no growth path reads mcfg.max_pods directly, so a path added later cannot silently escape every cap. Co-Authored-By:
-
- 18 Oct, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authored
A page dispatched to a pod holds a pod-pool thread for its whole life, so the pool's width decides how many pods an orchestrator can actually feed: pods_feedable = pod_pool_width / max_inflight_per_pod With _MAX_POD at 2048 and a 64-page-per-pod ceiling, that was 32 pods — whatever max_pods said. An operator who set max_pods=128 got 96 pods' worth of nothing, and no line anywhere explained it: not ocr.max_concurrency (4096 here, so it looked generous), not max_pods, not the cgroup (pids.max 16384 with 79 in use on the box this was measured on). A silent fleet ceiling is a worse failure than a slow one, because every symptom points at the pods. Raised to 8192, which feeds 128 pods at 64 pages each. Past that an async client really is the right answer rather than more thread stacks. Nothing that protects the box changes. `pool_sizes()` still clamps against `task_budget()` — the cgroup's pids.max minus what is running minus headroom — still keeps _TASK_HEADROOM in reserve so a busy install can still be diagnosed, and still gives the pod pool away before the general pool, because an install that cannot answer /healthz cannot be fixed. Tests pin all three. One existing test asserted `pod == 2048` for a configured width of 2048. That was only true because _MAX_POD clamped there; the property it meant to protect is that the configured width reaches both pools, which is `want * 2`. Restated that way, so the next raise does not break it again. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A rented pod has no config.json and no models.json -- it creates defaults and environment is the only channel it has. CODERAI_OCR_CHANDRA_INSTANCES and CODERAI_OCR_OLMOCR_INSTANCES were sent to every pod and read by nobody, so those engines fell back to the dataclass default of ONE instance however many the operator configured, while the admin UI showed the configured value. Measured on one A40 with chandra_instances=128 and 32 pages in flight: [ocr] engine 'chandra': 1 instance(s) ready vLLM: Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6% Avg generation throughput: 54.5 tokens/s against surya's ~2000 tokens/s and ~32 concurrent sequences on the same card and the same pages. It read as "chandra is 36x slower than surya"; it was one page at a time. Surya worked only because its own env rows had been mapped -- the gap was invisible for exactly as long as surya was the engine being tested. Adds the two instance counts, plus chandra's and olmOCR's page budgets (MAX_TOKENS, MAX_CONTINUATIONS, MAX_RETRIES, DPI), which were sent and unread for the same reason: seeding an engine's wiring but not its tuning gives a pod running that engine's library defaults, which is indistinguishable from the outside from the wrong pod image. The test is deliberately generic -- for each engine and each tuning knob, if the orchestrator sends it the pod must read it. A list of expected names would have passed before this fix on surya's rows alone and said nothing about the other two engines, which is how this survived. NOTE: config.py runs ON the pod, so a pod only gets this once its image carries it. coderai-vllm-ocr must be rebuilt before chandra serves more than one page at a time. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.4.29 added a guard against a pool holding one pod twice and asserted, in its message and in the code, that this was what billed two A40s against a max_pods=1 configuration. That attribution is not supported. The $1.18/hr incident was two DISTINCT pod ids, four minutes apart in the account listing, and its cause was the warm-floor top-up counting only healthy pods against the cap -- fixed in 0.4.27. The duplicate was a later and separate observation: a snapshot carrying two rows for one pod id, taken without recording the model key, so two POOLS reporting one shared pod was never ruled out. That case `_add_pod` does not address, since it dedupes within a pool. The guard is kept, because two paths appended pods with no identity check between them and that is a hazard on its own. What is removed is the claim that it explains a billing event, and the invented detail of a pool "reporting 2 in pool, 2 healthy, 2 at cap with 128 pages in flight while the card ran 31" as though it had been read off that duplicate. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two paths append pods -- a fresh rent and an adoption -- and neither checked whether the pod was already in the pool. On a restart both fire for the same machine within the same second: [runpod] adopting pod j74fxkxr9vs3fg's bearer token [runpod] reclaim: took back a pod for 'datalab-to/surya-ocr-2' ... and the pool then held one A40 twice. Every capacity question is answered by walking self.pods, so the duplicate is not cosmetic: * the pool claimed 2 x max_inflight_per_pod = 128 slots on a card that was serving 31, reporting "2 in pool, 2 healthy, 2 at cap" with 255 pages queued -- and refusing work the pod could have taken; * max_pods is enforced against that same count, so a cap of ONE was satisfied by two entries for one pod, which is how a second real pod came to be rented beside it and billed at $1.18/hr for a configuration that asked for $0.59; * the in-flight staleness sweep added in 0.4.28 cannot help here, because both entries hold legitimate fresh timestamps. A duplicate is not a leak. Both sites now go through `_add_pod`, which is the only remaining way into the pool and refuses a pod id it already holds. The EXISTING handle is kept rather than replaced: it carries the in-flight accounting for work already dispatched to that pod, and dropping it would strand every slot it holds. What can legitimately change is refreshed onto it -- the address, since RunPod remaps ports and the pod id is the identity, not the url. A test asserts there is exactly one `self.pods.append(` in the file: the guard is worth nothing if a third path is added later that bypasses it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`pod.inflight` is raised in `_pick` and lowered only in `release`, which makes it a ratchet. Any dispatch path that fails to release raises it for the life of the pod, and at max_inflight_per_pod the pod is refused all work while its card sits idle -- reported as "at capacity", which is the most misleading thing this file can say. Observed on a deliberately single-pod fleet: the pool reported inflight=64 of 64 with 63 pages queued, while the pod itself reported `Running: 31, Waiting: 0` at 10.6% KV cache. Half the slots did not exist. A single test document then waited 900s for a pod that was half idle, and timed out. Two such leaks have now been found and fixed -- a lease dropped during acquisition (0.4.23) and a page whose client disconnected. The reason this change exists is the third one, which has not been found: a counter that can only rise is one bug away from taking a pod out of service permanently, and every leak so far has been invisible precisely because the pool reported confidence about a number nothing could correct. So each slot now records when it was taken, and the LIST is the authority: * slots held longer than INFLIGHT_TTL_S (900s) cannot belong to a live page and are returned. Deliberately generous -- a safety net, not a timeout. Returning a slot that IS in use would hand a pod more work than its cap allows, while holding a phantom a few minutes longer costs only those minutes. * a counter standing above its own recorded slots is counting slots with no evidence of ever being taken, and is pulled down to the evidence. This is not conditional on anything having expired -- the first version of the sweep only examined pods with a stale entry, which would have skipped a pod whose counter had drifted upward, and skipped every pod already in the pool when this code arrives carrying a counter and no timestamps: exactly the locked-out pod the sweep exists to free. The list is created on demand rather than required at construction, so duck-typed pod stand-ins are not forced to carry a bookkeeping field, as with the per-card memo. Co-Authored-By:
-