- 12 Oct, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authoredcoderai_engine_inflight / coderai_requests_active are load-bearing: the hourly autoupgrade's idle guard reads them to decide whether a restart is safe, and with MAX_DEFERRALS=0 a non-zero count makes it SKIP for ever. So one request counted in and never out silently stops a host taking upgrades, and the only symptom is a version that stops moving. Observed on the Digesta orchestrator today: the gauge read 13 against two real requests, the 09:01 tick logged "SKIP: 7 piece(s) of work in progress" and left the image alone, and only a manual restart cleared it. The structural leak is in the streaming proxy path. Starlette awaits a response's background task only AFTER the body is sent -- `await self.stream_response(send)` then `await self.background()` -- so when the client disconnects mid-stream the send raises, the exception propagates, and `BackgroundTask(_release)` never runs. The relay generator's `finally` DOES run in that case (Starlette closes the iterator) but not when the body is returned without streaming, so both now call `_release`, which is idempotent via a flag. `exit_request` is idempotent per rid as well, and that is required rather than decorative: two release paths firing for one request would otherwise decrement an UNRELATED in-flight request, and the count would under-report real work -- worse than over-reporting, because then a restart lands during a request. Every request is now timestamped, not just the inference ones that carry `meta`. Without that a leaked non-inference request was invisible as well as unreleasable: the count said 13 and the Tasks page showed nothing, which is how this went unnoticed. And a reaper underneath all of it (`server.inflight_max_age_s`, default 3600, 0 disables), called from the two places that ASK whether anything is in flight: * the /metrics scrape, because that is the exact moment a stale count does its damage -- the scrape that would report a lie is the right place to correct it, not to hope somebody notices; * the supervisor's drain wait, which would otherwise sit out the whole grace period for a request that finished an hour ago. An hour, not a minute: a cold RunPod pod can make a single OCR page take 14 minutes, and reaping a live request would be a worse bug than the one being fixed. Training is unaffected -- it is tracked as a JOB and the request that starts it returns at once. NOT fixed here, noted for later: `_proxy_passthrough` has the same BackgroundTask-only release for the system worker. It never calls enter_request, so it cannot leak the gauge -- it leaks an httpx stream, which is a smaller and separate problem. tests/test_front_inflight_leak.py: 14 tests, including double-release, late release after a reap, thread safety, and the source invariants for the two release paths. 299 passed across the front/metrics/registry/config/admin selection; the 9 failures are pre-existing (broker_config x5, broker_streaming, studio_composed_surfaces, whisper_server x2) and touch no file changed here. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
-
Stefy Lanza (nextime / spora ) authored
v0.3.25 shipped the trace as a bare list of findings, and defined the empty list as "the bytes were sound, so the loss is in the serving path". That is wrong whenever the upstream returned no logprobs: there was nothing to analyse, and the emptiness would have read as a confident conclusion drawn from missing data -- exactly the class of mistake the trace exists to stop. So the trace is now a self-describing envelope: {"calls": 7, "with_logprobs": 7, "conclusive": true, "kinds": {"invalid-utf8": 29}, "why": "...", "findings": [...]} and `unreadable.blame` is one of "the model", "the serving path" or "unknown". `conclusive` is false when no calls were recorded (the hook never fired) or when none carried logprobs, and then blame is "unknown" rather than defaulting to either culprit. surya asks for logprobs by default (SURYA_INFERENCE_LOGPROBS = True) and the OCR gateway forwards the body verbatim precisely so they survive, so the normal case is conclusive. But an upstream that declines them is a configuration, not a bug, and it has to be visible instead of silently inverted into an answer. Also: a trace analysis that throws now reports {"conclusive": false, "why": "trace analysis failed: ..."} rather than no key at all, so "the analysis broke" and "the bytes were fine" cannot look the same either. tests/test_ocr_token_trace.py: 37 (was 30), including the inconclusive paths. 825 passed across the OCR/config/admin/runpod selection, same 10 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`ocr.surya_token_trace` (off by default) reports the per-token bytes at every place a surya page lost a character. It exists to answer one question that the finished text cannot: whose fault is it? invalid-utf8 the reassembled stream does not decode -- a lone UTF-8 lead byte with nothing to complete it. The MODEL emitted a broken sequence; the text faithfully renders a wrong answer. model-emitted the stream decodes and a token IS the replacement character. The model chose it. no findings the bytes were sound and the markers appeared anyway, so the loss happened AFTER them -- OURS, in the serving stack. That last case is why an empty trace is a result, not a failure. CORRECTION, and the reason this was needed. v0.3.21 claimed U+FFFD could not be a byte-level artefact, on the grounds that `a`-grave was a single token so one marker had to mean the model chose it. That was wrong. Measured: 'a-grave' utf8=[195, 160] ids=[2131, 2258] pieces=['A-tilde', 'l-bar'] decode([2131]) = U+FFFD <- a lone lead byte, exactly ONE marker decode([2131, 2104]) = 'e-grave' surya-ocr-2's vocabulary is BYTE-level, so one marker per accented letter is precisely the signature of a byte-level failure -- the thing it was claimed to rule out. The vocab lookup was misread too: `vocab['a-grave'] = 2160` does not mean the letter is token 2160; in a byte-mapped vocabulary that key is the glyph standing for byte 0xA0, which alone also decodes to one marker. The "one marker vs two" discriminator does not exist. Digesta's charset instinct was closer to the mark than the rebuttal. base.UNREADABLE and docs/ocr.md now carry the correction rather than the refuted argument. What IS established, measured on the Aruba install at the same DPI through the same transport on perfect synthetic Italian (crisp vector glyphs, 34pt, 250 DPI): surya 29 U+FFFD markers, zero accents surviving chandra 0 markers, all 12 accents intact So the document is exonerated. Also notable: the em dash survived surya intact while every two-byte accent died, and surya emitted U+1EEA (Vietnamese) in place of one -- no decoding bug can fabricate a character that was never in the input. Implementation notes: - surya DROPS this data. Its recognition predictor does pass it on (`BlockOCRResult(..., raw_logprobs=out.logprobs)`) but the model never declares the field and pydantic defaults to extra="ignore", so it is constructed and silently discarded -- their own comment calls it "phase 2 use". Verified by construction: the attribute does not exist on the result. So the value is taken one layer earlier, wrapping `_generate_one`, which still has it. The wrapper only reads and swallows every error: a diagnostic that could break a page would be worse than no diagnostic. - The scan reassembles the WHOLE byte stream and walks it as UTF-8 rather than judging tokens pairwise. Per token is meaningless here -- one character legitimately arrives as two or three tokens, each "invalid" alone -- and the first version of this did judge them pairwise and flagged every accent and every em dash, the exact false positive that would make it worthless. - Capped at 40 findings per page, two tokens of context each side, because a lead byte is only wrong in the light of what followed it. - Clean pages are byte-identical to before: the key appears only when the text actually contains a marker, and it is reported ONCE document-wide under `unreadable.tokens` rather than repeated in every page's meta. - The flag is in _pool_params, because the hook patches surya's backend at LOAD -- without that, toggling it would appear to save and take effect never. A third discriminator needing no code: serve the same weights through llama.cpp (`datalab-to/surya-ocr-2-gguf`, `surya_serve: "llamacpp"`). A different detokenizer on identical weights would convict vLLM outright. tests/test_ocr_token_trace.py: 30 tests. 818 passed across the OCR/config/admin/ runpod selection with the same 10 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A fifth OCR engine. Chandra (datalab-to/chandra-ocr-2) is from the same lab as Surya and is driven the way olmOCR is — one page image over an OpenAI chat endpoint, three serving modes (model | vllm | server) — but its `ocr_layout` prompt asks the model to answer as one <div> per layout block carrying data-bbox (normalised 0-1000) and data-label. So it is the only engine here that returns text, boxes, labelled regions, tables AND page metadata in one pass, with no separate detector. Why it was worth adding now: it is a DIFFERENT MODEL FAMILY from surya-2 (Qwen3-VL's tokenizer, against surya-2's own 65k WordLevel vocabulary) reached over the same transport, at the same pixel ceiling, from the same rasterised page. That makes it a cheap controlled experiment on the U+FFFD accent loss: accents intact under chandra and lost under surya points at the model; both losing them points back at the document's pixels, and then pdfimages -list is the next thing to read. Implementation notes: - NO new pip dependency. The two prompts are mirrored verbatim from chandra/prompts.py (down to the Python repr of the allowed-tag list, which is trained-on wording), the input scaling is chandra's own scale_to_fit reimplemented, and the reply is split with stdlib html.parser rather than pulling bs4 and markdownify into the main venv for a tag split. - Chandra's pixel cap is 3072x2048 = 6,291,456 — the SAME numbers as surya-2 — and it snaps to a 28-pixel grid for the Qwen vision tower. So chandra_dpi defaults to 250 for exactly the reason surya_dpi does. Boxes are de-normalised against the image the MODEL saw, and the size the caller rasterised is reported as meta.source_size when they differ: de-normalising against the pre-scaling size would put every box on a scaled page in the wrong place. - A repetition loop answers 200. A VLM that loses its place repeats the last thing it saw until the token budget runs out, and the result looks like a successful page. chandra_max_retries (2) re-asks with a raised temperature — a greedy decode falls into the same loop twice — and a page that still loops is returned flagged with meta.repeat_detected rather than handed on silently. - The parse never loses page. No blocks at all, content outside any block, an unclosed final div (routine at the token limit), a bbox that will not parse: each keeps the text. A bbox that cannot be read yields NO region rather than a full-page one, because a wrong box looks like a real answer. - Table cells get a space, not a newline, so a table does not flatten to "VoceImporto". License: narrower than Surya's GPL. The code is Apache-2.0 but the WEIGHTS are a modified OpenRAIL-M — research, personal use and companies under $2M funding/revenue, no use competing with Datalab's API. So it is gated on chandra_accept_license, and the pod path differs from surya's on purpose: asking a pod for surya by name IS the GPL acceptance, while chandra's is only ever inherited from a decision already recorded in this install's config. A pod cannot accept a licence on an operator's behalf. lang= is accepted and does not steer it (language_aware: false), like surya-2 and olmOCR: 90+ languages, no language input. So it keeps ONE pool whatever is asked — a language-keyed pool is a second set of loaded instances holding VRAM. Wired through: engine registry, licence gate, per-engine DPI, the vLLM stop path (its VRAM is the subprocess's, not a pool's), the capability catalogue (which now reports boxes/regions/tables from chandra_layout, because with layout off there genuinely are none), model-name aliasing, the automatic fallback chain, pod planning + env seeding, the frontproxy's local-OCR check, models.json capability advertisement, the pipeline step's engine list, admin save/load and the Settings OCR card. tests/test_ocr_chandra.py: 61 tests. 788 passed across the OCR/config/admin/ runpod selection with the same 10 pre-existing failures (broker_config x5, studio_composed_surfaces x3, whisper_server x2). test_ocr_fanout's catalogue test asserted a frozen engine set; it now asserts against the registry, which is what "lists every engine" was always supposed to mean. Also fixed while in there: the Settings page used `parseInt(v) || default` for numbers whose zero is meaningful, so a deliberate 0 saved as the default. New _num() helper only falls back for an empty or unparseable field. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/ocr has accepted `lang` since the beginning, with the comment "reserved": it was parsed and dropped. A client asking for Italian got whatever the install was configured for and no way to tell. It is honoured now, on /v1/ocr, /v1/ocr/batch and the pipeline step, with precedence request > ocr.<engine>_lang > ocr.lang > auto. The request beats the engine-specific setting deliberately: that setting is an install's default, while a caller naming a language knows something about the document in hand. Asking for `auto` explicitly is NOT the same as sending nothing, so an install configured for Italian can still be asked to work it out. The default is auto-recognition rather than any particular language, because a server that has not been told what it is reading should not assume one. For surya-2, olmOCR and docTR that is simply what they already do: they are multilingual and have NO language input to constrain -- surya-2's entire prompt is "OCR this block image to HTML." and its own module warns the wording is the training-time contract, so there is nowhere to put a language and nothing to gain. Rather than let `lang` look effective everywhere, every response now carries what was applied AND whether the engine that served could act on it ("lang": "it", "lang_applied": false), and /v1/ocr/engines reports the same per engine so a client can choose before sending. PaddleOCR is the exception that shapes the rest. It loads a language-specific recognition model and its options are sent once at load, so an instance is built for one language and cannot be re-languaged. Hence: `auto` becomes ocr.paddle_auto_lang (default `latin`, a SCRIPT family rather than detection -- PaddleOCR's own default is `ch`, a worse guess for most documents than no guess), and paddle gets one pool per language, keyed on the model it is BUILT with so `auto` and `latin` share a pool instead of loading it twice. The multilingual engines keep a single pool whatever is asked. That is not tidiness: a language-keyed pool is a second set of loaded instances holding VRAM, and a production install runs 24 surya instances. Both pool-iteration sites already use .values(), and _Pool keeps the plain engine name, so eviction and the VRAM releaser handle the new keys unchanged. ocr.lang's shipped default moves "it" -> "auto"; installs that set it explicitly (Digesta production does) are unaffected. ocr.surya_langs stays as the legacy setting it is -- it reaches only the <=0.17 recognition API and does nothing on the pod path. 21 tests. The 8 failures in test_broker_config / test_studio_composed_surfaces / test_whisper_server_local_models are pre-existing, verified at HEAD. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
ocr: olmOCR has to trust a pod's certificate itself, and U+FFFD is one token not two bytes (v0.3.22) olmOCR could not reach a rented pod at all: CERTIFICATE_VERIFY_FAILED. A pod on the direct-TCP path serves HTTPS with a certificate signed by THIS install's own CA (codai/api/pod_tls.py), at a public IP that was not known when the certificate was issued, so stock `requests` fails it twice over. Surya never hit this because surya attaches through the fan-out gateway, which learned to trust pod certificates in 0.3.16. olmOCR talks to the upstream the dispatcher chose DIRECTLY, so it needed the same trust of its own -- and the gap only surfaced when olmOCR was asked to cover for surya, which is exactly when a fallback engine must not be the thing that is broken. It now goes through codai/api/pod_http.py, which verifies https-to-a-bare-IP against our CA with the hostname check off and hands every other URL -- localhost, a named server, another coderai -- to requests untouched. Swept the rest of the tree: every other raw requests call targets a localhost worker or RunPod's own public API, where pod_http would be a pass-through anyway. test_ocr_olmocr's HTTP-status test patched sys.modules["requests"] wholesale, which now breaks pod_http's own import of it; it patches pod_http.post instead and asserts the same two statuses. Also records what the accent investigation actually established, because the natural reading of U+FFFD is wrong and expensive. It is NOT a charset bug: U+FFFD is a real entry in surya-ocr-2's vocabulary (id 65421 of 65425, the only id that decodes to it), the tokenizer is WordLevel and character-level for Latin so an accent cannot be split across tokens, and the decisive detail is the COUNT -- a byte-level mis-decode yields TWO markers per accented letter (a-grave is two UTF-8 bytes, each one undecodable), and what arrives is ONE. One marker is one token. Every other hop was checked: _html_to_text is regex plus html.unescape and survives literal and entity forms; the isolated-venv worker emits json.dumps with ensure_ascii=True, so the pipe carries pure-ASCII escapes and the locale cannot matter; the gateway forwards raw bytes. And it corrects the previous commit's claim. Raising DPI is right on its own merits -- 200 really was feeding surya 62% of the pixels it accepts -- but a production document came back with the same markers at 200 AND at 300, which is evidence against too-few-pixels being the whole story: a perception failure would be stochastic, not every accent every time. The docs now say what distinguishes the remaining candidates (the document's embedded scan resolution, which `pdfimages -list` reports and which no rasterisation DPI can improve, versus the serving stack) and how to tell them apart: OCR perfect synthetic Italian through the same path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Accented letters came back from surya as U+FFFD -- "piu`" as "pi?", "facolta`" as "facolt?", nine of them in one page, as real EF BF BD bytes. That is exactly what a byte-boundary decoding bug looks like: a` is C3 A0, and byte-level BPE splitting it across two tokens is the classic cause. It is not that. U+FFFD is a REAL ENTRY in surya-ocr-2's vocabulary -- id 65421 of 65425, the only id that decodes to it, with <unk> a separate token at 65424. The tokenizer is WordLevel and character-level for Latin text, so a` is a single token (2160) and an accent CANNOT be split across two: no partial-UTF-8 artefact is possible, and per-token decoding is byte-identical to decoding the whole string. Surya reads text only from message.content, never from the per-token logprobs, and the fan-out gateway forwards the upstream body as raw bytes. So the model predicted "there is a character here I cannot identify" and every hop delivered that faithfully. Two things follow. It is a RESOLUTION problem. Surya caps its own input at 3072x2048 = 6,291,456 pixels in scale_to_fit, which is called with DEFAULTS ONLY -- no surya setting raises it, IMAGE_DPI_HIGHRES included. Measured, for A4: 200 DPI reaches the model as 3.89 Mpixels, 250 as 6.03, and 255 saturates it; 300 and 400 are downscaled back to the same 6.23. The shared ocr.dpi = 200 was therefore handing the model 62% of the pixels it accepts, and a diacritic sits above the x-height at ~3 pixels tall. So DPI is now chosen per engine (ocr.surya_dpi, default 250) and resolved BEFORE rasterising; paddle and docTR have their own optima and keep ocr.dpi. The lookup is generic (f"{engine}_dpi"), so another engine only needs the field. Resolving the engine first also refuses an unknown one before a large PDF is rasterised rather than after. It must not be SILENT. "pi?" indexes, searches and reads wrong for ever, and for judicial documents a known unknown beats a quiet corruption. Lines, pages and the document now count their unreadable glyphs and locate each one with the confidence of the line it sits in -- the finest granularity that exists, since surya reports one mean token probability per block, not per character, and that is also the signal worth acting on: a low-confidence line with an unreadable glyph is a line to re-read at a higher DPI. The key is OMITTED when there are none, so a clean response is byte-for-byte what it always was, spots are capped with a truncated count, and the text itself is never rewritten -- stripping the marker would destroy the only record of where the loss is. 20 tests, including the arithmetic that chose 250, kept so a later edit has to face it. The 8 unrelated failures in test_broker_config / test_studio_composed_surfaces / test_whisper_server_local_models are pre-existing -- verified identical in a clean worktree at HEAD. Co-Authored-By:
-
- 11 Oct, 2026 20 commits
-
-
Stefy Lanza (nextime / spora ) authored
/v1/models answers the moment a vLLM process starts — minutes before it can serve, because the checkpoint still has to be downloaded (every pod re-downloads it without a network volume) and the KV cache built on first use. The pool called that ready, every client read it as usable, and the whole model load landed inside somebody's first request, where it is indistinguishable from a hang. That is what cost today. One OCR page took 807s end to end — ~400s image pull, ~400s weights — while the client timed out at 180s and then 300s and concluded the serving was broken. It was not: nobody had ever waited long enough, and the only reason we know is a probe run with a 1400s timeout, which came back 200 with text and conf 0.9019. So the pool now sends one token against the served model before calling the pod ready. The pod bills through the load either way; this only decides whether the wait is visible as a pod that is not ready yet, or hidden inside a request that looks stuck. `pods_ready` becomes a signal a client can act on, which is what every client already assumed it was. A failed warm-up is reported and the pod used anyway: it may still serve (an engine that takes no chat completions, a model whose warm-up shape we guessed wrong), and rejecting a pod we have already paid to boot over a diagnostic request would be worse than the hidden latency this removes. Off with warmup_on_boot=false or warmup_timeout_s=0. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A rented pod is the one machine in the system whose logs cannot be read. RunPod publishes no pod-log API — its REST spec lists 23 routes and none serve logs, which is why every attempt returns HTTP 400 — and boot.sh writes its phases to a file and then execs uvicorn, whose output goes to the container's stdout and nowhere else. /boot therefore covered the way up and nothing after it, and _dump_pod_logs only ever ran on a failure, printing into the orchestrator's log. That cost an evening. Three pods were investigated in one session and two were already destroyed by the time there was a question to ask — one reclaimed by RunPod mid-request, taking the only evidence with it — so "the serving hangs" stayed a hypothesis for an hour. It was not hanging: a page took 807s, most of it the weights download every pod repeats because global volumes are console-only, and every client had given up at 180-300s. So the pod keeps a bounded ring of its own stdout and stderr (codai/api/podlog) and serves it at /logs with a cursor, and the pool drains that on every scaler tick into <config_dir>/pod-logs/<pod_id>.log — kept AFTER the pod dies, which is the whole point. It also drains once more before tearing a pod down, because the interesting lines are the last ones and they are only readable while the pod exists. A pull, not a push: the orchestrator already reaches pods to serve requests, so the bearer token and pinned TLS apply unchanged and a pod with no outbound route still gets its log kept. Bounded at both ends, since neither disk is free, and a gap in the pod's ring is written into the file instead of passing silently. The stdout tee writes through to the real stream on every failure path — a logging aid that could break printing would be worse than none. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
RunPod reclaimed an A40 mid-document today. The pool went on reporting it healthy, so every page chose the same corpse: the in-flight count climbed, the pod was never dropped, and nothing drained. From the client that reads as "requests neither complete nor time out", and it is what wedged the test. RunpodBackend has always handled this — _unreachable() drops the pod and the request goes again. The fan-out gateway never did: it returned 502 and left the dead pod in the pool as healthy. So a lease can now report its upstream dead, which drops the pod, clears the cached candidate set and releases the slot; the page is retried ONCE against whatever replaces it, and a third attempt would just be a retry loop against a real outage. Only connect-level failures count as dead. A read timeout is a model taking its time over a dense page, and discarding a healthy pod for that would be an outage of its own. Also: the first version of that handler could not work. It used a metaclass with __instancecheck__ so `except _DEAD_UPSTREAM` would match httpx's errors — but `except` matches on the class hierarchy at C level and never consults __instancecheck__, so it caught nothing. A test caught it; the check is now an explicit isinstance() where it cannot lie. And the registry path again: 0.3.15 keyed the fallback off XDG_CONFIG_HOME, which the container launcher exports only into the processes IT starts. A diagnostic script entering the container another way printed a THIRD registry path and created a second pod TLS CA. The fallback now reads CODERAI_CONFIG_DIR, the variable the launcher actually sets, so every process agrees however it was started — and since pod_tls.tls_dir() derives from the registry path, the CA follows. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
refuses our token is not an asset (v0.3.17) Both from watching a real OCR run. THE LOST INTENT. A client asked for the fleet back while a page was still being read. The pods correctly stayed — work is never dropped to save money — but the intent was then forgotten: the pool fell back to idle_timeout_s (900s there) and billed for a fleet nobody had asked to keep. It took a second teardown call, four minutes later, to actually release it. The pool now remembers: pods kept only because they are working are retired the moment they fall idle, the same "do not wait out the timeout" treatment a closed schedule window already gets. Asking for warm pods again cancels it. Also fixes the reply and the log, which said "floor 1" when the 1 came from work in flight. That reads as though the operator had asked for the pod — the opposite of what it meant. Demand and configuration are now reported separately ("needed" and "floor"), because they are two different reasons a pod survived. THE ADOPTED POD THAT 401s. After a restart the pool adopts the pod it left running instead of renting another, and carries that pod's bearer token across through the registry — when one was recorded. With none recorded it fell back to our own token and never checked the result, so a pod launched with a different one answered 401 to every request for the rest of its paid life. Seen today, right after a restart: "reusing the pod already running" followed by 401 on every OCR page. The health probe that found it proves nothing here — a coderai pod answers /healthz to anybody. Adoption now probes an AUTHENTICATED path with the token it would use, and a pod that refuses it is terminated rather than adopted: with one registry per deployment (0.3.15) an unrecorded key is a lost key, so that pod can never serve anyone, and paying for it is the only outcome worse than killing it. An unreachable pod is NOT treated as unauthorised — that stays with the health machinery, which has a budget for it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
First real document, first real pod: it came up at https://194.68.245.79:22198 and every page came back 502 — OCR upstream pod failed: [SSL: CERTIFICATE_VERIFY_FAILED] unable to get local issuer certificate A direct-TCP pod serves the certificate coderai's OWN CA signed, on a bare IP that certificate can never name. codai.api.pod_http has always known this: it pins to pod_tls.ensure_ca() and switches hostname checking off. The fan-out gateway is async and could not reuse that requests-based module, and I gave it a bare httpx client — so the one hop that now carries every OCR page was the one hop with no pod trust. It keeps TWO clients per event loop, chosen by pod_http.is_pinned_url: the pinned context trusts only our CA, so using it for everything would reject a cluster node whose certificate a public CA signed, and sharing one client would let whichever upstream connected first decide the policy for both. Worth recording: this was never fan-out-specific. With serve_fanout off, surya attaches straight to that same https:// IP with its own httpx and fails identically — `model` mode has never worked against a TLS pod. The gateway is what makes it fixable in one place. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two things, both from today's live incident. THE SPLIT REGISTRY. _pod_registry_path() fell back to ~/.coderai whenever config_manager was not initialised in the calling process. The container's launcher puts the real config dir one level deeper — it exports XDG_CONFIG_HOME="$CODERAI_CONFIG_DIR" and uses "$CODERAI_CONFIG_DIR/coderai", which is also where platform_paths.user_config_dir() lands. So a process with the config manager wrote one registry and a process without it read another, and the pods each owned were orphans to the other. That is what made the documented mutual-kill loop recur: pod created 19:24:56, terminated as untracked 19:24:57. The fallback is now the same XDG-aware path the launcher computes, and the resolved path is logged once — a registry that splits silently is invisible until it costs a GPU. CLIENT-FORCED TEARDOWN. `pods: 0` released a client's lease but gave nothing back: the pods then waited out scale_down_after_s, or idle_timeout_s before that. The automatic path is slow on purpose — it infers from a momentary reading, and a cold pod costs minutes to replace — but a client that has just finished a batch is not inferring. `teardown: true` sheds the surplus at once, skipping both the window and the one-per-pass limit, and answers with what happened. It still cannot scale through the operator's min_pods, and a pod with requests on it drains rather than dying under them. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Observed live on the Digesta orchestrator while testing OCR: pod created 19:24:56, "terminated STALE pod … not tracked by any pool" 19:24:57. The pool then waited out its 900s boot timeout on a pod that no longer existed, every OCR request 503'd, and the next attempt rented another. This is the race the source already documents from an earlier incident — "the nvidia engine created a pod and the radeon engine terminated it four seconds later, then the reverse — a mutual kill loop that rented and destroyed pods until stopped" — and the defence against it was supposed to be the age guard, "never reap a pod younger than this, whatever the registry says". It was written as `age is not None and age < REAP_GRACE_SECONDS`, and list_pods() returned {id, name, status, cost_per_hr, image} with no timestamp of any kind. So age was None for every pod, the condition was never true, and the guard abstained on exactly the pods it existed to protect. A guard that fails open is worse than none: it reads as covered. Two changes. The guard now fails CLOSED — an undatable pod is left alone and says so — and the listing carries runtime.uptimeInSeconds so it can measure instead of abstaining. Not reaping costs one more cycle; reaping a live pod costs the work on it and bills in a loop that never serves a request. What made it recur here is worth recording: this install has TWO config dirs, so the cross-process registry that shields a new pod exists twice and each process reads the one the other does not write. That is the deeper fault and is not fixed by this commit. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two faults, one cause: `local` was both the default and the only engine mode that would quietly settle for the CPU. Surya's own settings pick the device and fall back cuda -> mps -> cpu (surya/settings.py TORCH_DEVICE_MODEL), so `local` on a machine with no CUDA ran a VLM pipeline on the processor: minutes a page, a result that looks perfectly normal, and nothing anywhere saying why. paddle and docTR already refuse in exactly that situation — surya was the exception. It now refuses too, with a 503 that names both ways out, and the worker is TOLD its device rather than left to guess. Detection fails closed: torch is asked first (it is what surya will ask), nvidia-smi second, and if neither can answer the answer is no. And `surya_serve` no longer defaults to `local`. An install that said nothing got the one mode that runs the model on this box — on a GPU-less orchestrator whose whole job is renting one, that is the worst possible default. It now defaults to `model`, which adapts to wherever the checkpoint is served and asks for surya_model_id when it has not been told. Safe to change: both production installs set the mode explicitly (`model` on Aruba, `vllm` on zeiss), so nothing was relying on the old default. Running on the CPU is still fully supported — surya_use_gpu = false — it just has to be a decision now. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pool rents at whatever the market offered when each pod was started, so one fleet can hold a $0.35 card and a $0.79 card doing identical work. Shedding the cheap one saves less than half as much, and which to prefer is the operator's call, not ours — hence a checkbox rather than a new default. With scale_down_costliest_first the dearest surplus pod goes first, tie-broken by the idlest and then the coldest cache. Cost decides WHICH pod goes, never whether one goes, and a dear pod with requests on it is drained rather than dropped — so choosing it costs time, not work. Worth weighing before switching on: an on-demand pod costs more than a spot pod, so "dearest first" prefers to keep the interruptible ones. Also fixes the default tie-break, which was inverted when scale-down was written: `-last_used` picked the pod used MOST recently, discarding the warmest prefix cache of the set. It now retires the coldest. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Only half the scaler existed. The pool adds a pod once every pod it has carries scale_up_inflight_per_pod requests, and pods went away again only through the idle reaper — which fires on a pod with NO work and no request for idle_timeout_s. That misses the expensive case. A pool that grew for a burst and is now serving a trickle keeps every pod: each is touched often enough that last_used never ages out, each is a fraction busy, and the bill stays at the peak for as long as the trickle lasts. With max_pods raised to 64 that is the difference between a few dollars an hour and a few hundred. So the pool also asks how many pods the CURRENT concurrency justifies and retires the excess. The rules that matter are not "fewer pods": * work is never dropped to save money. The pod retired is the least loaded, and if it still has requests it is marked draining — no new work, and terminated only once it has finished what it had; * one pod per pass, after the surplus has persisted for scale_down_after_s, and the window restarts after each retirement. A dip of a few seconds must not cost a pod wanted again seconds later: a cold one takes minutes; * a quiet pool is NOT this routine's business. With nothing in flight, teardown stays entirely with idle_timeout_s — a number the operator chose, weighing the bill against a cold start — so this never silently shortens it. A draining pod is also excluded from the growth decision, which reads the LEAST loaded pod: an emptying drainer would otherwise answer "no need to grow" with the one pod that cannot take the work. Two existing tests had to change. A _Pod stub stood in for PodHandle without the new field, and a source-text assertion used a 600-character window that a new field plus its comment pushed data_center out of — now asserted against the whole class, which is what it was always about. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The gateway built an httpx.AsyncClient for every request, so each page paid a fresh TCP and TLS handshake to the upstream. With surya_instances: 24 that is 24 streams reconnecting per page, against a round trip that already dominates the cost of reading one. One pooled client with keep-alive instead, so a document's pages reuse the connection. Correctness-neutral; it only ever cost time. Keyed by the event loop, because an AsyncClient binds to the loop that created it and coderai runs more than one (the OCR pool keeps its own). The key is the loop OBJECT held weakly, not id(): keying by id would leak an entry per dead loop and, since CPython reuses addresses, could hand a new loop a client bound to the closed one that used to live at that address — a fault that would only appear under the load this change exists to handle. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The fan-out gateway lived at /v1/ocr/engine/<lease>/… and the new catalogue at /v1/ocr/engines. They do not collide, but both auth gates exempt the gateway by PREFIX — so any later route under /v1/ocr/engine* would have inherited that exemption silently, and the two names are indistinguishable at a glance in a middleware condition. The gateway is now /v1/ocr/attach/<lease>/…: internal by name, and nothing a public path family will grow into. Tests assert /v1/ocr/engines still requires ordinary authentication at both gates, and that a plausible future sibling (/v1/ocr/engine-stats) does too. Free to rename because nothing consumes the lease URL yet — it is minted per attachment at run time and never written down. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Naming an engine an install had not enabled was a 400, and there was no way to find out beforehand — the client had to know, or guess and handle the refusal. GET /v1/ocr/engines answers it from this install's own config: every engine with `enabled` and, when it is not, a `reason` that names the setting to change (an unaccepted licence is reported as such rather than as merely disabled, because that is a different checkbox). Each row also carries `kind` — an engine owns its weights, a pipeline drives a model — the `aliases` that also resolve to it, and `returns`, so a caller that needs coordinates can see that olmOCR has none rather than discovering it in an empty field. The top level reports both what the operator wrote as the default and what it resolves to. Where the model runs is deliberately not in the answer. It is configuration, it can differ between two requests, and a client that could see it would start depending on it. A test asserts the payload mentions no pod, node or address. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
docTR called .cuda() when torch happened to see a card and otherwise simply stayed on the CPU, so a GPU-less install read every page many times slower with nothing in the response to say why — the result looked normal, it just arrived minutes late. Nothing else in coderai quietly relocates GPU work, and paddle refuses outright in exactly this situation; docTR was the odd one out. `doctr_use_gpu` is now a requirement: no CUDA device means a 503 that names the setting. Running docTR on the CPU is still fully supported and unchanged — it just has to be asked for, with doctr_use_gpu = false. The test drives the real load() with docTR and torch stubbed, rather than re-implementing the placement rules beside it; reintroducing the fallback makes it fail, which is the only reason to trust it. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three corrections to the fan-out, all the same mistake in different places: the code was deciding things the configuration had already decided. Nodes are no longer discovered. Asking every cluster node what it holds and using whoever answered is configuration by probe: it puts work on a machine nobody chose and moves it the moment a different node replies first. A node serves a model because the model is PINNED to it — `engine` on the entry, the same hard constraint the front router already honours, comma-separated when several nodes should share the work. A pinned node that does not hold the model is reported, not silently replaced. The decision is not an OCR decision, so it does not live in codai/ocr: any model, any caller that needs an endpoint rather than a proxy hop, gets the same answer from codai/models/placement.py. For requests that pass THROUGH coderai this was already true — the router honours the pin, treats a node as an engine by name, reaches RunPod through the backend. This is that decision for a caller that must hold an address itself, which is why OCR needed it. And "how busy is too busy" is the model's own scale_up_inflight_per_pod, on the model page, instead of the parallel ocr.serve_spill_inflight knob I had added — one number per model, already there, already meaning exactly this. A model with no RunPod block has nothing to rent, so load alone never moves its pages. Finally, the caller: resolve_engine accepted the four pipeline names only, so a client naming the model it can see in /v1/models got "unknown OCR engine". Both now work — engine=surya and engine=datalab-to/surya-ocr-2 reach the same place — because which pipeline reads a page, and where its model runs, are ours to know and not the client's. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
An OCR pipeline is handed one server address and drives it itself, which is what keeps its page images, its strict json_schema and its logprobs intact. One address cannot spread load, and three things followed from that: * a pool grows on in-flight requests counted as they pass through it. An attached engine never passed through, so the pool saw no load however hard OCR ran, never grew, and the engine had one pod's URL anyway; * several cluster nodes serving the model resolved to whichever answered first, for ever, however busy it became; * a local model configured to burst to RunPod never did: that trigger lives on the proxy path an attached engine bypasses. So the engine gets a lease on an address of ours, and codai/ocr/dispatch.py chooses per request: a pool that is the model's configured home takes every page (picking the least-loaded pod, renting when they are all at the threshold); otherwise this machine and the nodes holding the model compete on load — including the load a node reports for its whole install, so a node busy with someone else's work is left alone — and the overflow past ocr.serve_spill_inflight rents instead of queueing. An explicit `backend` pin still wins over discovery: that was a decision, and load-aware picking must not quietly repeal it. The hop is allowed to exist only because it decides nothing: body bytes in, body bytes out. The losses it is there to avoid came from PARSING a request and re-emitting it through the backend abstraction. Two gates stood in the way and neither could be satisfied the usual way: the engine has no API token, and surya's /health and /v1/models probes send no headers at all (bare httpx GETs), while refusing to attach if /health is not 200. The lease is therefore read from the URL, and the engine-only internal gate exempts that path alone — tested to be no wider than that. olmOCR's model mode asks the same dispatcher, and still takes the in-process route when the answer is local, where the model manager's loading, eviction and queueing belong. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Surya asks every request for logprobs and averages exp(logprob) over the generated tokens to score each line. coderai accepted the parameter, never sent it, and hardcoded `logprobs: None` in every response builder. Surya's fallback is what makes that expensive: with no logprobs it does not report "unknown", it reports a literal 1.0 (recognition/__init__.py:248, :302, layout/__init__.py:97). Every line of every page comes back perfectly confident, the guesses included, and nothing downstream can tell which judicial document needs a human to look at it. So the round trip is wired where it can exist: the proxy backends (vLLM, RunPod) forward `logprobs`/`top_logprobs` and hand back what the server measured, buffered and streamed, accumulating the streamed pieces into one response-shaped object. The manager asks the backend's signature rather than negotiating by TypeError, which would have called it twice for a kwarg it never had. A request that did not ask is sent exactly as before, and a backend that cannot answer still returns null. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Everywhere else in coderai an engine is what RUNS a model. `ocr.default_engine` borrows the word for two different kinds of thing: paddle and docTR are runners with their own weights, while surya-2 and olmOCR are pipelines over a model that vLLM runs. Only the second kind has to be told where a server is — and that is the same question the model layer already answers for everything else. So answer it in all the places a model can actually be served. A cluster node was the one that was missing: a node is a whole coderai registered with the head as a remote ENGINE, so nothing in the backend layer knows it exists, and a VLM hosted there came back as "not served over HTTP" from an install that was serving it perfectly well. The node's own catalogue is the authority, so it is asked. An explicit backend pin still wins — discovery is the fallback, not an override — and the nodes are not probed at all when clustering is off, which would have added a timeout to the first page of every document. Also: one test here was passing for the wrong reason. `from codai.models import manager` reads the package attribute and never consults sys.modules, so a fake installed there was simply not seen. Patched as attributes of the real module, which is what made the precedence test fail honestly. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three silent drops on the way to a rented pod, all found while wiring Surya to the model layer, all of the same shape: coderai accepted a field, forwarded a reduced version of it, and the pod answered something plausible. * Images. The front flattens multipart content to the literal text "[image_url content]" unless the backend claims vision, and only the llama.cpp backend ever did. So every page sent to a VLM served on RunPod — olmOCR in "model" mode, or Surya through a coderai-vllm pod — arrived as that placeholder, and the model answered fluently about nothing. A proxy is the wrong place to judge: forwarded, a text-only server returns a plain 400 and a VLM server reads the page. * Structured output. Only {"type":"json_object"} survived, so a json_schema — the strict form current clients send, Surya's guided decoding among them — was dropped and the reply came back as free text the caller could not parse. The streamed path ignored the field entirely, json_object included. * extra_args. Whitelisted per engine on a model's runpod block since pods shipped, parsed, stored, and never put on a command line. Setting it looked like tuning and did nothing, which is worse than refusing it: the pod boots, serves, and ignores the decision. A page-reading model wants prefix caching and the Qwen-VL pixel bounds, and until now the only way in was docker_args, which replaces the whole generated line. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
OCR on the Digesta install answered 503: "vLLM isolated venv not found at /cache/vllm_venv". That host has no GPU at all — renting A40 pods is its whole job — so the venv it was told to build could never have run, while the Surya-2 checkpoint it needed was already a configured model (backend: runpod, engine: coderai-vllm) nothing in the OCR path could ask for. The model layer was never the problem; `ocr.surya_serve` was asking the wrong question. It enumerated server PROCESSES — torch in a local venv, our own vLLM, a URL typed in by hand — when the answer the operator had already written down was a MODEL. olmOCR has had that bridge since it shipped ("model" mode, through our own chat path); Surya never got one, because Surya2 drives its server directly with guided-JSON decoding and logprobs that a proxy hop would drop. So it gets the same bridge, in the only shape it can take: resolve the model through whichever backend owns it and hand the worker the address. Three values, because Surya refuses a server whose /v1/models disagrees with the checkpoint it expects, and a pod URL is useless without its token — which only Surya's vLLM client reads, so a bearer-locked endpoint is always attached to as "vllm". The address is a lease, not a location: a min_pods=0 pod is reaped when OCR goes quiet, and the page that finds it gone re-resolves once — renting a fresh one — instead of every later page failing against a dead URL. A failure against a server that still answers /health is a real error and is reported as one. Co-Authored-By:
-
- 10 Oct, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authored
When zeiss ran out of memory this morning the OOM killer picked coderai-nvidia. Not because it was at fault — a runaway test process was — but because oom_score is essentially "how much memory does this task hold", and an engine with a model loaded is always near the top of that list. The leak carried on; the thing that was serving died; the box needed a hard reset. Linux offers one lever, /proc/<pid>/oom_score_adj, and two rules that decide where it can be pulled from. Both were verified on the two production hosts rather than assumed: raising is free, lowering needs CAP_SYS_RESOURCE — and a docker container does not have it (tested: in-container `echo -500` is denied, `echo 200` succeeds). So the protected base has to come from the engine daemon, via --oom-score-adj on the run, and everything inside can only step back UP from it. rootless podman cannot protect at all: a negative value is clamped to 0. 0 is still worth having on the Digesta host, where coderai currently sits at +200 — inherited from systemd's user manager, which makes it a PREFERRED victim with a kernel score of 800 against zeiss's 670. So: the container gets -500 (CODERAI_OOM_SCORE_ADJ, 0 disables), and inside it codai/util/oom.py states the order as three tiers relative to that base — engines +100, workers +300, training +500. Relative, because the base differs by host and a tier should keep its distance whatever it turns out to be. Engines are marked in the existing _engine_preexec hook, which covers the isolated model workers too: they are spawned by the engine, not the front, so they inherit it. Training is marked from the parent after Popen — a preexec_fn would run Python between fork and exec in a threaded server, which is the documented way to deadlock. The ordering is the point. Protecting everything equally would just move the bullet to an innocent bystander; this way the kernel sheds a training run (hours, restartable) before an engine (loses its in-flight work, and the front notices and respawns it) before the front (loses everything, including the ability to respawn anything). Every helper can only ever make a process MORE killable: nudge() refuses a negative delta rather than attempting a privileged write, because a process that believes it is protected and is not is worse than one that knows it is exposed. All of it is best-effort — a process that cannot adjust its score still serves. 16 tests, including that the number reaches the kernel and moves the score, that a child can be marked from the parent, and that the engine mark sits inside a try (it runs between fork and exec, so anything that raises there fails the spawn). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
zeiss came back from this morning's hard reset with no container at all, and the hourly upgrade then skipped every tick — so the host sat on old code and, worse, stayed DOWN, with nothing in the loop that would ever have recovered it. Both idle probes are HTTP, so an absent server is indistinguishable from a wrong token: inflight() said "unknown" and the fail-closed guard refused to act. The Digesta host, whose container was up, upgraded itself fourteen seconds after the push; zeiss needed hands. container_state() answers the one question the probes cannot — is there a server at all — by name, then by any running container of the image being upgraded. Only a POSITIVE "not there" counts as idle: an engine that cannot be asked (daemon down, wrong socket) stays unknown and still fails closed, because the point is to recognise an absent server, not to assume one. Checked before the token, since a host with no server is in the same position as one with no token. Both installs run their engine's ps as the service user and name the container `coderai`, so the default matches; CONTAINER_NAME covers an install that does not. The second half: every knob reads `VAR="${VAR:-default}"`, which looks like the environment may override it, but the conf is sourced first and assigns unconditionally — so `MAX_DEFERRALS=1 coderai-autoupgrade` did nothing at all, with no line in the log to say why. I hit that landing this morning's restart and had to source the conf from a wrapper to get round it. The environment is now snapshotted before the conf and restored after, for that run only, and a test checks the snapshot list against every knob the script resolves so the next one added cannot quietly fall out. require-idle and max-deferrals are also logged now: "it skipped again" and "it was told to skip" used to look identical. The conf example claimed that without a token "the restart happens whenever the timer says so". It is the opposite — the check fails closed, so a running server is then never upgraded. Corrected, and REQUIRE_IDLE_CHECK documented next to it. tests/test_autoupgrade_recovers_a_down_server.py RUNS the script against a stubbed engine, runner, restart command and health file: the env restore is eval-based and the container check branches on an exit code, neither of which reading the source can confirm. 12 new tests, 56 across the three autoupgrade files. No version bump: nothing in the image changed, and bumping would send both production installs through an image upgrade and a restart for a host-side script. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A full `pytest tests/` run reached 46 GB RSS plus 20 GB of swap on a 54 GB machine. Swap filled, the OOM killer took coderai-nvidia — which was serving — and the box needed a hard reset, which lost the last forty minutes of kern.log with it. atop's history is what identified the process: a pytest worker, carrying the `coderai-front` process title because importing the front sets it, growing 7.8 GB → 34.8 GB → 46.6 GB across three ten-minute samples. The defect is `except asyncio.CancelledError: pass` wrapped around an await whose job is to collect a CHILD task. asyncio delivers a cancellation as a CancelledError at the next await point, so that handler catches two unrelated things: the child reporting it stopped (fine to swallow) and the caller asking us to stop (never fine). run_forever's reconnect handler sat on exactly such an await, so `task.cancel()` was absorbed and the loop carried on. With the reconnect delay mocked to zero by the test driving it, that became a busy loop allocating ~78 MB/s — AsyncMock records every call it receives — and 400,000 iterations in the five seconds I measured. being_cancelled() (Task.cancelling(), 3.11+) tells the two apart, and all four sites now re-raise ours: _stop_heartbeat_task, _cancel_inflight_tasks, the two keepalive collects in handle_message, and BrokerService.stop. The inflight ones matter in production independently of the leak — absorbing a cancel there means a shutdown waits out a whole video generation instead of stopping it, and run_forever simply reconnected forever instead of exiting. run_forever's own cancellation path clears the cancel for the duration of its cleanup and re-raises at the end, so honouring the cancel does not cost the websocket close. The test that drove it now brakes itself: the zero-delay sleep stand-in counts, and raises a BaseException (anything else is caught by the reconnect handler) well past the two reconnects it needs. With the product bug reintroduced the file fails in 1.5 s at 441 MB instead of eating the machine. tests/conftest.py adds what was missing in general: an RSS ceiling (6 GB, CODERAI_TEST_RSS_LIMIT_MB) and a per-test stall ceiling (180 s, CODERAI_TEST_STALL_SECONDS), both dumping every thread's stack to a file before exiting — pytest captures stderr per test and drops the buffer when the process does not unwind, so a watchdog that only printed would have died silently. No new dependency. "A test can consume the host" is a property of the run, not of one test. tests/test_broker_protocol.py could not finish before this — it hung at the 14th test. The whole suite now completes for the first time: 78 s, 1.57 GB peak, 1635 passed. The 25 failures are the 17 that already failed plus the 8 stale register-ack expectations in that file, which were unreachable behind the hang and are a separate question (the fixtures send `accepted: true`; the client has required `status: "ok"` for some time). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Township's clip lengths were tuned when the only video model could hold ~81 frames in one call, so a 70-second match became a dozen short cuts. LongCat was pretrained on continuation and the server renders a long shot in segments, so the same match wants a handful of long takes — and the frame numbers that suit one are wrong for the other by a factor of four. Only the two starter templates knew that, and only because someone typed it in. Worse, nothing snapped a picked count onto what the model can emit: random.randint handed the server lengths like 250, which is not 4n+1 for any VAE and not on LongCat's 93+80k segment grid, so the server rounded and the plan's own numbers became fiction. On LongCat that also silently costs block-sparse attention, because every point of 93+80k is a 16n+13 count BSA can tile and nothing between them is. Server — codai/models/video_geometry.py is the geometry once: native rate, the legal frame grid, how much ONE render holds, whether continuation is native, and the side multiple below which the fast attention path is lost. Published per model on /v1/models (ModelInfo.video) and used by the render path itself, so what the server advertises and what it rounds to cannot drift. The numbers were spread across three modules that could not tell a client; the literals here are pinned to each source by tests, since the LongCat and H3 workers run in 3.10 venvs this cannot import. models.json overrides every field. Township — the five frame fields now default to 0 = AUTO: derive the budget from the model's geometry and this tool's seconds band, then snap both ends and every pick onto the model's grid. Seconds, not frames, because clip count falls out of long_target / (frames/fps). An explicit number still wins, so a saved config keeps exactly the budget it had. Entrances and the face-off get their own shorter band instead of borrowing the fight budget (on LongCat that made every entrance an 11-22s take). Outcome shares are snapped per shot and the total rewritten, the per-render chunk comes from the model (Wan keeps its conservative 50), and the chaining decision reads the published continuation rather than matching the model's name. At the defaults a 70s match goes from 9-11 Wan cuts to 4-6 LongCat takes, each one request with native continuation, with BSA still on. 1607 tests pass; the 17 failures and the test_broker_protocol hang are pre-existing and unrelated (verified with the diff stashed). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An i2v generation died in the first denoising step with a bare AssertionError out of flash_attn_bsa_3d, on a geometry the guard had just declared valid: 93 frames, sides divisible by 64. bsa_problems only ever checked the noise branch. A conditioning pass does not tile the latent once — Attention.forward splits the sequence into a conditioning block and a noise block and calls flash_attn_bsa_3d on each (modules/attention.py:123-134), so the cond depth and the remaining noise depth must divide by the chunk as well. generate_i2v hardcodes one conditioning latent (pipeline:796) and 1 % 4 != 0, which means block-sparse attention has never once been able to run an i2v pass, at any frame count or resolution. The avatar pipeline hardcodes the same 1 in four places, and generate_vc computes a count it never rounds. Only generate_refine gets it right, by padding both counts to the granularity itself (pipeline:1245-1250). (a) The guard now knows about the split. _cond_latent_passes enumerates every conditioning count the request can produce — segment 0 uses the task's own entry point and every later segment uses generate_vc, so even t2v runs a conditioning pass once num_segments > 1, and BSA is one flag for the whole request — and a count that cannot tile puts the pass on dense attention with a line saying which condition failed. (b) bsa_pad_cond, OFF by default, rounds the count up to the granularity on the way into the DiT, which is what generate_refine does for itself, so an i2v or continuation pass can keep BSA. Patched onto the DiT instance rather than the vendored source, the same way the denoising bars are rebound: the count is born inside generate_i2v's loop, so there is no outer argument to correct. Off by default because the pipeline marks only the real conditioning frames clean (timestep[:, :1] = 0), so rounding 1 up to 4 hands the conditioning block three latent frames that are still noise — well-defined attention, but not what the model was trained on, and on a video model that kind of error shows up as temporal drift. Co-Authored-By:
-
- 09 Oct, 2026 8 commits
-
-
Stefy Lanza (nextime / spora ) authored
Probed every API surface coderai can use for attaching a global volume, each with a body valid enough that the extra-key check actually runs: POST /v1/pods key … not in input schema: 'globalNetworkVolumeId' podFindAndDeployOnDemand field not defined by the input type POST /v2/pods additional properties 'globalVolumeId' not allowed templates (mounts) every candidate key refused There is no public API for it: global volumes are console-only, as the docs say. Which makes `global_volume_id` a loaded gun — sending the field fails the create, so configuring a global volume would have stopped the install renting pods at all. Both create paths now recognise that specific rejection, drop the field and retry. The pod boots without the volume and downloads its weights: slower and it costs bandwidth, and far better than serving nothing. Logged once per process, naming the consequence and the override to set when RunPod publishes the field name. An unrelated failure is NOT treated this way — a capacity error retried with the volume stripped would bring a pod up silently wrong. Earlier probes that reported the field as "accepted" were wrong: with an otherwise-invalid body the validators stop before the extra-key check. Worth remembering next time I probe a schema. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`gpu_type` was a filter: "NVIDIA A40" meant an A40 or nothing, so when RunPod had no A40 free the request failed rather than taking a card that would have served it. Now `gpu_types` is an ordered PREFERENCE and `allow_other_gpus` lets the search fall through to anything else that still satisfies min_vram_gb and max_hourly_usd. A preferred card ranks ahead of every fallback whatever selection_criteria says — otherwise "cheaper" would put the fallback first and naming a card would mean nothing. Each ranked option carries `preferred`, and provisioning a card nobody asked for prints "[FALLBACK — no preferred card available]": correct behaviour, but never silent. Off by default. A pinned card is sometimes pinned for a reason, and an install that quietly started renting something else would be a surprise on the invoice. Name matching is exact on the id or display name, case-insensitive, tolerating a missing "NVIDIA " prefix, and deliberately NOT a substring: "A40" would otherwise match "NVIDIA RTX A4000" — 16 GB for a job that asked for 48. GPU names split on commas only, because "NVIDIA RTX A6000" has spaces in it. Also: the admin handler rebuilds the runpod block from a whitelist, so all six keys added today (global_volume_id, data_centers, gpu_types, allow_other_gpus, allow_client_scale, client_scale_ttl_s) were being dropped the next time anyone saved a model from the GUI — taking the warm schedule and the volume with them. They are in the whitelist now, with the two ordered lists kept in order. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Weights in one place, pods wherever a card is free. `global_volume_id` attaches a RunPod GLOBAL volume (beta since Sept 2026): region-independent storage, so unlike `network_volume_id` it does NOT pin the pod's data centre — which is the only reason to use one. Resolved by its own function precisely so the two cannot be confused: the region lookup stays under the network-volume branch, and a test holds them apart. `data_centers` is the other half. A single `data_center` pins one region and a blank one lets RunPod pick any region on earth, so there was no way to say "anywhere in the EU" — which is what data that may not leave the EU requires. The list is tried in order, and each (card, region) pair becomes its own candidate so the capacity fallback that already walks GPU options walks regions too, with no new control flow. The pod handle records the region it actually got, not the pool's first choice. Two honest limits, both documented in the code and the guide. A global volume must be created in the RunPod CONSOLE: POST /v2/network-volumes allows only STANDARD|HIGH_PERFORMANCE and demands a dataCenter. And attaching is console-documented only — no field exists in the Pod API reference, and the v1 REST validator accepts unknown body keys silently, so a wrong name cannot be caught by probing; it yields a pod with no volume that re-downloads its weights. We send globalNetworkVolumeId, overridable by CODERAI_RUNPOD_GLOBAL_VOLUME_FIELD, and it must be verified on a real pod. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The root cause under the last three fixes. The pod registry recorded os.getpid() and both readers skipped an entry whose pid matched ours as "our own, the local pool handles it". Inside a container the engine comes up as the SAME pid on every boot — 74, on the production host — so after a restart the previous container's pod read as ours and was skipped by both find_shared_pod (nothing to adopt) and sibling_is_provisioning (nothing booting). The warm path then rented another A40 beside the one already running. Three restarts this afternoon, three extra A40s, each billing until it was terminated by hand. Entries now carry a per-process nonce and identity is that. The pid is still recorded, and an entry written before the nonce falls back to comparing it, which is the old behaviour and no worse. test_a_surviving_pods_token_is_remembered_across_a_restart simulated the new process by monkeypatching os.getpid — the very thing that does not change. It patches the nonce now, which is why it failed on this commit and passes on it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The last hole in the warm path. sibling_is_provisioning already recognises a pod registered at creation with no URL yet — and after a restart the recorded pid no longer matches, so our own booting pod qualifies — but ensure_ready never asked. Seen live: a pod created at 18:57 was six minutes into its image pull when the 19:03 upgrade restarted the service, and the warm path rented a second A40 beside it. The upgrade runs hourly and an image pull is ~6 minutes, so that is not a rare alignment. Order in ensure_ready is now: a healthy pod of our own, else adopt a ready one, else wait for one that is booting, else rent. Also prune the pod registry in reap_orphans: an id RunPod no longer has stayed "known", which both shielded it from reaping and let the file grow for the life of the install — including pods terminated out of band, of which this afternoon produced five. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A warm floor is re-established at every boot, and ensure_ready() went straight to _provision_one(). The adoption logic — "a pod for this pool is already running, take it over" — existed only inside acquire(), so the warm path never saw it. Three restarts on the production orchestrator this afternoon rented three A40s and left each previous one billing until the reaper's 20-minute grace expired. An hourly unattended upgrade would have done that every hour. find_shared_pod already covers the case: after a restart the recorded pid no longer matches, so our own previous pod is adoptable. It is now one method, _adopt_shared_pod, called from acquire(), ensure_ready() and maintain()'s warm top-up. It also stops lying about the cost. An adopted pod was recorded at $0/hr as "billed by its owner", which is true of a sibling engine's pod and false of one we are inheriting from our own restart — a real A40 hidden from the $/hr cap and from the spend page. register_pod now stores the rate and adoption restores it; an older registry entry without one still adopts, at 0. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
NameError in _provision_one: it read `dc`, a local computed inside _create_with_fallback, from a function that has no such name. The sequence that costs money is this — create_pod succeeds, the pod boots on RunPod, the PodHandle raises, the pool never records the pod, the caller provisions again. Four A40s existed for a warm floor of one before anyone looked. Both paths now call pool._data_center(), which keeps the same precedence (the volume's region, then the model's, then the account's) in one place. The two tests that covered this asserted the broken expression as a SOURCE STRING — 'data_center=(getattr(self, "_volume_dc", "") or dc or "")' — so they passed on a NameError and would have gone on passing. They now call the resolver, including with nothing configured anywhere, which is the shape that raised. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Found while configuring the Digesta catalogue: the launcher reads /root/.coderai/config.json (engine_specs, published port) while the front and the engine read /root/.coderai/coderai/config.json. The RunPod api_key and the spend caps were in the first one, so the engine refused to provision anything and the admin page reported no caps at all. Records what was moved where, and that CODERAI_CONFIG_DIR=/config remains the fix that makes it unnecessary. Co-Authored-By:
-