- 17 Oct, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authoredA 5,000-document run rented 127 pods, reached $79/hr and completed zero documents in ten minutes. The front had stopped answering entirely: /v1/runpod/status timed out on every port, which is also what the client had been reporting as thousands of timeouts. The pods were fine. Probed from inside the container, both transports answered /healthz in milliseconds, vLLM was loaded on each one, and the GPUs sat at KV 0.0% with no surya worker line, because no page had ever reached them. `placement.choose` is synchronous and ends in a pool's `acquire()`, which waits on a condition variable for up to its default 1200 s and may rent a pod INLINE -- a candidate sweep over every region, then a boot wait, in whichever thread asked. It was called straight from `_read_on_pods`'s coroutine, so it ran on the event loop and took the whole process with it for as long as one page waited for a pod. It has been there since the pod pipeline was written (0.3.70) and cost nothing while acquire() returned at once: a pool with a free pod answers in microseconds. It turns fatal exactly when the pool is the thing under strain, which is when it matters most. So the lease runs on the pod pool now, and because a pod takes MINUTES to boot, three things follow: * One attempt is bounded (`POD_LEASE_ATTEMPT_S`, 20 s), threaded through `placement.choose(acquire_timeout=)` to `pool.acquire(timeout=)`. An unbounded attempt parks a thread for a whole boot, per page; bounded, the waiting happens in `_lease_a_pod`'s own loop where it is free. * `ocr.pod_pipeline_wait_s` goes 60 s -> 600 s. The 60 s was chosen because a waiting page held a thread, so a long ceiling meant thousands of parked threads. It no longer does, and at 60 s a cold start handed its whole first wave of pages to a GPU-less orchestrator to rasterise -- the exact failure `prefer` exists to prevent. * The per-page PDF split moves to the DEFAULT executor, not the pod pool: pdfium permits one caller at a time, and putting a split behind thousands of leases waiting for a boot is a cold start that cannot produce the next page while it waits for the pods that page is for. The forward itself stays on the event loop, which is the point of 0.4.8 and is now covered by a test that fails if it moves back. All of this is invisible to a test that awaits one page at a time: one page blocking the loop for 0.5 s looks exactly like one page taking 0.5 s. The new tests measure the longest gap between two ticks of a coroutine running alongside, and the thread-seconds held while waiting out a boot. Five of them fail on 0.4.15. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
-
Stefy Lanza (nextime / spora ) authored
chandra and olmOCR each set `meta.confidence` and `meta.confidence_measured`. surya set neither: its predictor reports confidence per LINE, so a caller switching engines was handed a different kind of answer, and nothing could tell "no line reported a confidence" from "the model was certain it was unsure". Now it reports both. Weighted by CHARACTERS rather than per line, because the other two average over tokens -- that is, over text -- and a plain line-mean lets a two-word line count as much as a full one: on [(100 chars, 0.95), (2 chars, 0.1)] it answers 0.9333 instead of 0.525. `confidence_measured` is the half that earns its place, and the reason the aggregate reads the RAW value rather than the per-line field: that field has to coerce a missing confidence to 0.0 for its own sake, which would make nothing reported -> 0.0 a real zero -> 0.0 indistinguishable. One such 0.0 has already produced a confidently wrong verdict in this subsystem. So: nothing reported -> (0.0, False) a real zero -> (0.0, True) mixed -> unreported lines skipped, never counted as zero Both keys reach the caller with no new plumbing: `_page_from_dict` forwards unknown keys by EXCLUSION (a choice made after that reassembly silently dropped `accents_repaired`, and then `model_attempts`), and both are already in the pod->orchestrator whitelist, so a page read on a rented pod reports what a local one does. Tested, including that `False` is not mistaken for the empty value that exclusion filter skips. The aggregate is extracted as `_page_confidence()` so it is testable without surya installed. Worth recording what this is NOT: on the page where the accent bug was found, surya reported ~96 % per line while every accent was a replacement character -- confident, and right to be, having emitted the correct tokens. Confidence is not a corruption detector; the byte trace is. 13 new tests. 3447 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Italian accents were lost again on surya: a measured page came back with 25 U+FFFD, **zero** 0xC3 lead bytes and a surviving 3-byte em-dash, while chandra read the identical pixels with 40 0xC3 bytes and every accent intact. So the 2-byte UTF-8 sequences were lost in DETOKENISATION, not misread from the page -- which is what `ocr.surya_repair_accents` exists to undo, and it had been shipped and verified months earlier. It had no input. The repair rebuilds the text from the model's own token IDS, the only lossless channel: vLLM 0.27.1 reports `bytes: []` for exactly the byte-level tokens that lose their characters (257 on one page, not one byte >= 0x80), so the byte fallback is dead there, and ids arrive only when vLLM is launched with `--return-tokens-as-token-ids`. That flag WAS configured -- `vllm.extra_args` on the renting install -- and the pods never got it. A pod launches its own vLLM from its own seeded config, the seeder sent only CODERAI_VLLM_ENABLE_LORA and _VENV, and there was no pod-side reader for `extra_args` or `return_token_ids` either. The pod's launch line proves it: `--trust-remote-code` present (the launcher adds that itself), the flag absent. The repair therefore had no working channel at all, and said nothing, because its diagnostics could not escape the pod either (below). Fixed at both doors, because the operator had already done their part: * `--return-tokens-as-token-ids` is FORCED for a byte-level OCR model (surya/chandra/olmOCR, matched on served name or path), deduplicated against `extra_args` so the original fix cannot pass it twice. A plain LLM is untouched unless asked. A guard that depends on every install remembering a flag is not a guard. * the operator's own `vllm.return_token_ids` now travels: CODERAI_VLLM_RETURN_TOKEN_IDS is seeded and read on the pod. And the reason a verified fix could regress in silence: * **the worker could not be heard.** `stderr=None` let it inherit the engine's raw fd 2, while a pod's log ring tees the Python `sys.stdout`/`sys.stderr` OBJECTS -- a subprocess holding the descriptor writes straight past the tee. `grep -r "\[surya\]"` across 343 drained pod logs returned nothing. It now gets its own pipe with a daemon pump that re-prints through the teed stream. NOT merged into stdout: that pipe carries the newline-JSON protocol. * **the diagnosis was opt-in.** The byte trace -- the only thing that separates "the model emitted broken bytes" from "our serving path mis-assembled correct ones" -- was gated on `surya_token_trace`, so a page full of markers came back unexplained unless someone had set a flag BEFORE the run. It now runs whenever the markers are present, which costs nothing on a clean page. * **nothing announced it.** A corrupt page now prints one line naming what was lost, whether the repair ran, recovered nothing, or is off, and the byte trace's verdict. A missing hook target is loud and on stdout. 21 new tests. 3432 passed, 13 pre-existing failures (broker, studio, manual client) untouched. NOT yet observed on a real page: the chain is proven in tests and the missing input restored, but the flag is applied at POD LAUNCH, so it takes this image plus a freshly rented pod. Co-Authored-By:
-
- 16 Oct, 2026 14 commits
-
-
Stefy Lanza (nextime / spora ) authored
Three things, and the third is the reason the first two were impossible. **A mixed fleet was distributed as if every pod were the same.** `PodPool._pick` was `min(free, key=inflight)` and growth was `load / scale_up_inflight_per_pod` -- both card-blind. An A100 and an A40 were handed the same number of pages, so the fast card drained its share and idled while the slow one straggled; and a document completes only when its LAST page lands, so the slowest pod set the latency the client saw. The scaler had the matching fault in reverse: a fleet of bigger cards was still sized as if each were the preferred one, which is the same arithmetic that reached 94 pods at 10 % utilisation. Now: weighted least-connections, an in-flight cap scaled per pod, and growth that divides demand by the fleet's real capacity. **`gpu_limits`: per-card numbers, nothing multiplied.** "gpu_limits": { "NVIDIA A100-SXM4-80GB": {"parallel": 186, "queue": 744, "scale_up": 93}, "NVIDIA A40": {"parallel": 64, "queue": 256, "scale_up": 32} } A scaled single number cannot stay right: the model-level figures describe ONE card, and which card that was came from the first entry of `gpu_types` -- so changing the RENTAL preference silently re-scaled the concurrency, and a figure measured on an A40 came to describe an A100 by a factor nobody had measured. A table states what was measured, per card. The share of load is DERIVED from it (two cards with depths 32 and 93 are given work in that ratio) rather than configured twice, with `gpu_weights` and then the VRAM ratio as fallbacks for cards nobody has measured yet. Editable per model in the admin UI; a row with a card name and no numbers is dropped, because "not measured yet" must not be saved as a limit of zero. `parallel`/`queue` are applied AFTER the candidate card is chosen. The launch plan is built before, so its env carried the model-level number whatever card won -- and the gate is env, set once, so a pod rented on a 3x card spent its whole life admitting a 1x card's work. **An adopted pod recorded `gpu="(adopted)"`.** A pool that cannot name a pod's card cannot size it, weight it, or report the fleet: a whole day's status output read `by_gpu: {"(adopted)": 32}`. `list_pods`/`get_pod` now ask RunPod for `gpuCount` and `machine { gpuDisplayName gpuTypeId gpuType { memoryInGb } }` (every field probed against the live schema first), the registry stores the card and its size so adoption needs no API call, and an unreadable card costs a weight of 1.0, never a pod. **And a catalogue entry could crash an engine.** Adding a `vision_models` row for a model served on RunPod quarantined the cpu engine for ninety minutes: its id is not a path, not a URL and not a repo id, so resolution fell through to "assume a repo id", then to "assume a URL", and `requests.get("qwen38-vision")` raised MissingSchema -- five exits in 120 s, QUARANTINED, taking /v1/ocr and every pool with it while `/healthz` still answered `ok: true` because the FRONT was up. Two independent guards: * a model pinned to a backend that serves it ELSEWHERE is never resolved, downloaded or cached here -- which is also what lets one row run on rented pods and load locally on a box with a GPU, the pin being the whole difference and the section not overriding it; * a name that is none of the three resolvable kinds is refused with a sentence naming all three and pointing at `backend`. The pin does NOT make local weights useless: `resolve_model_source` is configurable (`auto | hf | url | upload`) and reads the entry directly, so an id still means "the pod downloads it" and a local path still means "push it". Tested, so a later change cannot couple the two. 43 new tests. 3411 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
**The accent regression, reported from production hours after 0.4.7 shipped.** Italian accents stopped surviving surya again. The cause was this afternoon's scan passthrough, not the old detokenisation fault: its guard used an ABSOLUTE 150-DPI floor instead of the DPI the page would have been rendered at. surya's own default is 250 (`surya_dpi`), and Aruba leaves it unset, so every scanned page holding a 150-200 DPI JPEG went to the model at native size -- 25-40 % less linear resolution than the render it replaced -- and surya-ocr-2 is measured to read badly below 250. The reasoning error is the part worth keeping: passing a scan through was always meant to be "the same pixels, without the work". Fewer pixels is not a saving. The floor is now the TARGET DPI, and `page_image_min_dpi` (default 0) can only RAISE it. Verified against a 250-DPI engine: a 150 or 200 DPI scan is rendered up as before; 250 and 300 DPI scans still pass through, which is where the shortcut was always legitimate. It also honours the REQUEST. `ocr_document` resolves `use_dpi = int(dpi) if dpi else engine_dpi(engine)` and hands that one value to the pod forward, the local render and (on the pod) this guard -- so "is this scan detailed enough" is asked about what the caller asked for, never about a default they overrode. Pinned, because a future change reading the engine default would silently hand a 200-DPI scan to a request that wanted 300. **And the rasterisation hop now encodes, instead of shipping raw pixels.** Measured on A4 at 250 DPI (6.04 Mpix, surya's own default): raw RGB over the pipe 18.1 MB + 109.7 ms for the parent to encode it for the engine anyway PNG level 1 in the worker 0.72 MB 104.7 ms, in the WORKER the parent's lazy open 0.12 ms The encode is not added, it MOVES: off the single parent process onto the raster pool, which is parallel. The pipe carries 25x less, the second encode disappears, and the parent holds 0.72 MB per page in flight instead of 18.1 MB -- at 48 pages that is 35 MB instead of 870 MB, which is the quantity this whole subsystem keeps getting into trouble over. PNG, not JPEG, unless the operator asked: a page about to be read must not pass through a lossy codec by default. Raw RGB stays the fallback, so the protocol still reads on an older parent. Seven existing tests had encoded the wrong floor and were corrected; one of them hardcoded pixel counts that assumed A4 while the fixture is 609 pt, so it measured 244 DPI while claiming 250. 11 new tests. 3368 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
coderai serves a LoRA on a vision model and cannot train one. Recorded as a plan because the gap turned up while configuring qwen38 as a vision model and is worth writing down while the reasons are fresh: what exists (loras.py is a DIFFUSION trainer; text_loras.py only serves and packages), what to build (SFT from messages-with-media, video as frames plus a token budget, QLoRA against a 4-bit base, the vision tower frozen because vLLM cannot serve an adapter on it), and which questions need a measurement rather than an opinion. No code. Nothing here is scoped to one language, document type, card or deployment size. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.4.10 fixed `_served_names` and left `_sibling_lora_specs` reading `text_models` only -- found while verifying the built image, which is the point of verifying the built image. That second function decides which adapters vLLM registers AT LAUNCH. So a QLoRA declared on a `vision_models` entry was never collected when a text request started the engine: vLLM came up with no `--enable-lora`, and then answered under the adapter's name with BASE weights. Nothing failed, nothing was logged, and the reply looked exactly like a reply from the adapter. Its own docstring records that happening once already, for the sibling reason. Both reads are now one `_catalogue_entries()`: every section, one reader, so a third caller cannot drift from these two. A test asserts `md.get("text_models")` appears nowhere in the module. Also pinned, because it decides what a client must ASK for: an entry that owns an adapter registers it under `_entry_name`, which prefers `alias` over `id` (a seeded pod entry carries an alias and no id). An entry with BOTH therefore splits -- the alias serves base+adapter, the id serves the bare base, and nothing says so. An entry that owns an adapter should carry ONE name, which is how diogene-alpha and digesta-alpha were already declared and is now how the qwen38 vision entry is too. 13 tests. 3358 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A model's TYPE in coderai is the models.json SECTION it is listed in (see main.py's _CATEGORY_TYPE_PREFIX, and `vision_models` is built from that list alone). So serving one set of weights both as text and as vision means two entries in two different lists -- which is exactly what a VLM like Qwen3.5-27B-AWQ wants, since the same pod can answer chat and image requests from one copy of the weights. `_served_names` collected the names of those weights from `text_models` only. The vision entry's name was therefore never passed to the launch, the engine did not answer to it, and `_ensure_service_once` relaunches an engine that does not answer -- which for a 27B sharing a card is not a slowdown but an out-of-memory crash, with the first engine the one that dies. The sibling function `service_key_for` had already learned this lesson ("ONE engine per set of weights, not one per name"); this is the other half of it. Now every category is scanned. Not a longer list of blessed sections: any category may name these weights, and which list an operator chose says nothing about what one engine can serve. Adapter names are still excluded (vLLM would hold a model and an adapter under one name), different weights are still ignored, a model with no adapters is still named exactly as before, and a catalogue that cannot be read still cannot stop an engine starting. 7 tests. 3352 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.4.8 moved the forward off the default 32-thread pool and the measured result on the live orchestrator was 33 -> 483 pages in flight, 1.65-2.2 -> 7.99 pages/s, and a control-plane request that used to time out at 40 s answering in 12 ms. But a thread is still the wrong unit for the wait: ~60 s per page against a real corpus means 15 documents/second at ~3.5 pages each is ~3,200 outstanding forwards, against a pod-pool ceiling of 2048. Threads cannot get there, and should not try. So the hot path is a coroutine. `pod_http` gains `apost`/`aget` on httpx -- already in the stack (codai/cluster/nodes.py, codai/main.py), so no new dependency -- with everything the sync path learned kept intact: * the pinned TLS context, unchanged: a direct-TCP pod serves HTTPS with a certificate signed by THIS install's CA at an ip the certificate cannot name, so the CA is trusted and the hostname check is off. NOT verify=False, which would accept anybody; * one client per event loop AND per pinned-ness, because a client belongs to the loop its sockets were opened on and the two trust settings must never share one; * connection limits from the same `_pool_size()` the sync session uses; * a proxied pod (http://, or https:// to a name) stays an ordinary request. `_apost_page_to_pod` and `_apod_still_alive` mirror the sync twins, and both twins now share ONE request builder and ONE response reader -- two copies of that field list is how an engine name, a dpi or a token budget comes to apply on one transport and not the other. Every distinction 0.4.2-0.4.6 established is preserved and tested on the new path: an ANSWER (even 503) is a verdict about the pod and belongs on the refusal path, nothing-came-back is PodUnreachable, a dead pooled socket is retried on a fresh one, and a timeout is never retried -- resending a page the pod is still working on is a second card doing the same work and a double bill. `ocr.pod_http_async` (default on, admin API + settings control) puts the forward back on the pod thread pool. That path is kept working deliberately, not as dead code: this is a new transport on the hottest path in the subsystem, and an operator who hits something odd should change one setting rather than wait for a release. Four existing test files had to be repaired, and the reason is worth keeping: they stubbed only the SYNC twin, so the moment the async default landed they stopped testing their own stubs and began dialling 10.0.0.1 for real -- 300 s hangs where there should have been failures. Their helpers now stub both twins via `_stub_page_post`, and the policy tests say `pod_http_async: False` out loud because they stub `pod_http.post` specifically. A test that quietly measures something else is worse than one that breaks, and this change produced four of them. 21 new transport tests. 3345 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A 512-page semaphore, a 48-wide admission gate on every pod, 31 A40s at `Running: 1` with `Waiting: 0`, and the orchestrator delivered 1.65 pages/s. Raising `ocr.max_concurrency` 128 -> 512 changed nothing. Nor did replacing the pods' 2-wide gate with 48. The reason both were inert is that neither was ever the limit: 32 threads / 19.4 s per page = 1.65 pages/s Two defects, and the second is why the first was invisible for every release this code has shipped in: * **The per-page forward ran on the DEFAULT executor.** `asyncio.to_thread` hands work to the loop's default pool -- 32 threads, shared with every other blocking call in the process -- while `codai/api/thread_pool.py` exists for exactly this call, documents it ("a per-page OCR forward to a pod (30-300 seconds each)"), and offers `run_on_pod_pool()`. Nothing called it. It was dead code next to the problem it was written for. * **Every sizing helper read a name that does not exist.** `thread_pool._configured_concurrency()` and `pod_http._pool_size()` both did `from codai.config import config_manager`; the shared manager lives in `codai.admin.routes`. The ImportError was swallowed by a bare except, so both answered 0 and took their FLOORS -- 32 threads, 64 connections -- on every install that has ever run, whatever the config said. `ocr.max_concurrency` has never once reached a thread pool. So the two knobs an operator would reach for were the two that could not help, and the number in the boot line ("default pool 32 workers") was the floor wearing the clothes of a configured value. Fixed: * the page forward and the liveness probe go to the pod pool; * sizing reads the holder that exists, tries both, and `sized_from_config()` reports which -- the boot line now says FLOORS out loud when it could not read the config, because "32 workers" means something entirely different when it is chosen than when it is a fallback nobody saw; * `_MAX_POD` 512 -> 2048: ~100 pages/s at ~20 s per page is ~2000 forwards outstanding, and each is a thread blocked on a socket. Past that an async client is the answer, not more stacks; * `configure()` grows the pod pool when `max_concurrency` changes, so raising it mid-run widens the threads and not just the semaphore. Growing only: shrinking would have to stranded threads mid-page. With max_concurrency 2048 the pools are now (256, 2048) rather than (32, 64). Credit where it is due: the client team measured 41-42 established sockets inside the container's network namespace and a 1.65 pages/s ceiling, and said the serialisation was on the orchestrator's OCR path. It was. The contrary reading here came from running `ss` on the HOST, outside the rootless netns, which can only see forwarded sockets -- a measurement of the wrong thing, reported with more confidence than it had earned. 11 new tests, including the one-liner that pins the original bug (`assert not hasattr(codai.config, "config_manager")`) and a static check that the forward is not back on `asyncio.to_thread`. 3324 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Measured on A4 at 200 DPI (3.4-3.9 Mpix), best of three: pdfium page.render() 21-46 ms PNG encode, PIL default (level 6) 86-353 ms 1846 KB PNG encode, level 1 (lossless) 123 ms 2117 KB JPEG q95 8.4 ms 596 KB the page's own embedded JPEG 0.1 ms 147 KB So most of what is called rasterisation is not rasterisation, and on the surya path it was paid TWICE per page: once in subprocess_engine to hand the page to the isolated-venv worker, once inside surya's own client to build the data URI (surya/inference/backends/openai_client.py:32, with "image/png" hard-coded into the URI it builds). Against roughly 5 s on the GPU per page -- 12 pages/min/pod at Running: 0-3 reqs, measured on two live pods -- that is CPU the pods need in order to hold 48 pages in flight. Two ways out, both shipped, both default: * A page that already IS an image is sent as the PDF holds it. A scan is usually one full-page DCTDecode object, and extract_embedded() takes it in 0.1 ms with no render, no encode and no resample. Every guard in there is a way those bytes would have been a WRONG page rather than a slower one: one image object and nothing else (a stamp or a text layer must not be dropped), DCTDecode alone (Flate is raw pixels, CCITT and JBIG2 are bitonal, JPX is JPEG 2000), grey or RGB only (a CMYK JPEG out of a PDF decodes inverted), /Rotate 0 and an axis-aligned positive matrix (otherwise the bytes are a sideways page, which reads as gibberish rather than as an error), 98 % coverage anchored at the origin, and an effective DPI within [min_dpi, max_dpi_factor x target]. * When a page must be encoded, it is encoded once, at zlib level 1. Lossless, 2.6x faster, and the bytes are carried forward so the request reuses them. The carrier is an attribute on the PIL image, recorded with the size it was true for. Any PIL transform returns a NEW object and so drops it, which is the design: a missed passthrough costs milliseconds, a wrong one shows the model a plausible page at the wrong resolution. chandra and olmOCR therefore scale BEFORE converting, since convert() was quietly invalidating every page. End to end, one page from PDF bytes to data URI: a page that must be rendered 610 ms -> 234 ms (jpeg: 68 ms) a scanned page 610 ms -> 6.9 ms, 1895 KB -> 147 KB New: ocr.page_image_format (auto | png | jpeg), page_image_jpeg_quality, page_image_min_dpi, page_image_max_dpi_factor, with the admin API and the settings form. `png` always re-encodes, which is what a controlled comparison needs; `jpeg` is lossy and stays opt-in, because whether it costs accuracy is a question about a corpus and not about a codec. The rasteriser worker loads page_image by path: it may not import codai.ocr (377 MB via litellm), and the guards must be the SAME ones the parent applies or a pool outage would change what the model is shown. ocr_worker.py may not import even that -- it runs in the engine's own venv -- so its copy of the carrier convention is tested against page_image's to keep the two from drifting. 42 new tests; 3306 passed with the 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pooled keep-alive connection that a pod closes while idle fails on its next use: pod OCR at https://.../v1/ocr is unreachable: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) _post_page_to_pod turned every transport failure into OcrError(503), 503 is in _POD_CANNOT_RUN_PIPELINE, so one dead TCP connection was read as "these pods cannot run the OCR pipeline" and set the whole pool aside for five minutes. It happened twelve times in four hours on a production run: the GPU-less orchestrator rasterised every page of every document for those windows and climbed to 40 GB at ~0.8 GB/min, while sixteen rented A40s sat idle and willing. An ANSWER may condemn an image; a socket may not. So: * PodUnreachable, a distinct error for "nothing came back", which never sets the pool aside. A pod that ANSWERS 404/501/503 still does. * One page is retried on a fresh connection (nothing was sent, so there is no side effect) and then on another pod — but a pod is dropped from the pool only after it fails to answer /healthz. Writing a pod off because an idle connection was reused has cost seven healthy cards before. * A timeout is NOT retried: the pod may be ninety seconds into a 130-second read, and sending the page again doubles the work on that card. * An active cooldown is no longer extended. It used to be, silently: every further refusal pushed the expiry out while only the first one printed, so a five-minute window could last twenty with nothing in the log. Two more ways to end up local, both of which were SILENT: * The gate compared the request's engine against the three VLM engine names. /v1/ocr has always also accepted the MODEL a caller knows in place of the engine driving it — resolve_engine maps the Surya-2 checkpoint to 'surya' and the catalogue advertises those ids as aliases — so a caller naming the model got no pod-side pipeline and no message. The switch has to be read the way a READ reads it, for the same reason it has to be read the way a LAUNCH reads it. * Serve mode not 'model', an empty model id, an engine with no pod pipeline, a standing refusal: each returned "" or None without a word. The log of the run that prompted this holds 8,278 successful pod reads and nothing at all explaining the thousands of documents in between. Every one of them now says which way the decision went, rate limited rather than said once, because a cooldown is minutes and a run is hours. And, since reading a page here means rasterising here, ocr.pod_pipeline: prefer (default) wait up to ocr.pod_pipeline_wait_s for a pod, then read it here — a pool with nothing free is usually one still booting. A timed-out wait is not paid again for five minutes, so the ceiling cannot become a per-page tax. auto read it here at once (0.4.1-0.4.5 behaviour). require never rasterise on this box: 503, saying why. Tests: 32 new (tests/test_ocr_pod_pipeline_policy.py) covering the alias, all four silent routes, the three policies and the transport/answer line. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.4.3 got placement working on a pod: it resolved the model and minted an attach lease, which it had never managed before. Then surya said: SURYA_INFERENCE_URL=http://127.0.0.1:8000/v1/ocr/attach/<lease>/v1 is not reachable at /health ...about an address the pod was answering perfectly well. A pod on the direct path serves HTTPS on its ONLY port, and an OCR engine there is a SUBPROCESS: it attaches by URL and goes through the socket like any other client. self_base() handed it `http://`, so it probed a plain request at a TLS listener. Fixed with verification ON end to end, not off: * the leaf certificate now carries DNS:coderai-pod, DNS:localhost, IP:127.0.0.1 and IP:::1. It had one SAN, `coderai-pod`, that the comment said was "never checked" -- true for the orchestrator, which reaches a pod by a bare IP with the hostname check off. A pod reaching ITSELF has no such excuse, so it uses a name the certificate actually holds. * boot.sh maps 127.0.0.1 coderai-pod and exports CODERAI_SELF_BASE=https://coderai-pod:8000; the plain-HTTP case (behind the RunPod proxy) exports the loopback http base explicitly instead of relying on a default. * the CA's PUBLIC certificate now travels with the leaf, appended to CERTIFI'S ROOTS into one bundle. That order is not cosmetic: a bundle that lost the public roots would break every HuggingFace download on the pod, which is worse than the bug. Exported only once built and non-empty, and it says so plainly when it cannot be. This covers chandra and olmOCR too -- in served mode they attach through the same gateway URL from the main venv, so one trust store serves all three. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
New `ocr.max_pages`. A document with more pages than the ceiling is refused with a 413 naming its page count and the limit, instead of being read. Checked off the page tree, before anything is rendered, split or sent -- so an over-long document costs milliseconds rather than an hour of time, memory, tokens and (where pages are read on rented cards) money, only to die in a client's timeout anyway. Length is the one cost knowable before any work, and the one that scales every other cost at once. Shipped OFF (0 = no ceiling). Page count is not a fault: an install that batch-reads book scans is as legitimate as one that reads two-page letters, and this product does not know which it is talking to. Settable from the OCR settings page. Refused LOUDLY and with both numbers. This subsystem already carries the trap that a degraded page answers 200; a skipped document that looked like an empty read would be the same mistake with a bigger blast radius. The message names the setting so an operator can find it, and says the document was NOT read so a caller can route it elsewhere rather than file an empty result. `max_pages` at exactly the ceiling passes -- "more than N" means N is fine. A nonsense value fails OPEN: refusing every document would be worse than the problem. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A rented pod answered, for a model it was serving perfectly well over its own loopback: model 'datalab-to/surya-ocr-2' is not served over HTTP by this install Not rasterisation: a PNG, which skips rasterising entirely, failed the same way in 0.26s. The pipeline on the pod needs a LOCAL placement candidate, where the pipeline on the orchestrator needs a POD one -- so the local path is the one that had never run. Two defects, and the second is why the first took all night. 1. `_model_entry_for` reads the catalogue from one place: the ConfigManager the admin module holds. A pod has no models.json -- its catalogue is seeded from CODERAI_SEED_MODELS into memory -- so any process or import order where that global is not that object sees an EMPTY catalogue. The seeded model then has no backend pin, resolves to no engine, yields no candidate, and the pod disowns a model it is serving. It now falls back to the seed list in the environment, which needs no global and no import order. A real catalogue entry still wins. 2. `_local_candidate` ended in `except Exception: return None`, and `_pod_candidate` too. Every decline now records a reason -- which of the catalogue entry, the backend pin or the enabled flag was missing, or the exception -- says it once, and clears it when the install starts working. And the reason travels IN THE ERROR, not only in a log. The machine that reads a pod's 503 is not the machine that can read its log; a pod's log is not reachable from the orchestrator at all. Blaming configuration for what may be a lookup, in the one sentence that does cross the network, is what made this invisible. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.4.1 sent pages to the pods, as it was meant to. The pods answered 503: model 'datalab-to/surya-ocr-2' is not served over HTTP by this install, so an OCR engine cannot attach to it _read_on_pods raised that, so the document failed -- while the local pipeline sat there perfectly able to read it. The method's own docstring already said "an optimisation that cannot run must cost latency, never an error"; only the "no pod at all" case implemented it. A refusal (404/501/503 from the pod's own /v1/ocr) is now a fallback, not a failure: * the rest of the document stops being sent one failure at a time; * the pool is set aside for 300s, so the next document does not pay a failed round trip per page, and a rebuilt image is picked up without restarting the orchestrator; * it is said ONCE, in the pod's own words, and says that this box will now rasterise so its memory is worth watching. A 400 still fails: falling back on any error would hide a broken page behind a slow one. This restores service. It does NOT make the pods run the pipeline -- that is the next commit, and the log line above is what will say when it works. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`runpod.image` is optional. Naming `engine: coderai-vllm-ocr` is how the image is chosen, and pod_plan resolves that at launch. But the OCR pipeline's "do these pods rasterise for themselves" gate read the RAW `image` field, which is empty in exactly that configuration, and concluded the pods could not run the pipeline. So a deployment renting A40s with the surya venv and pypdfium2 baked in ran the whole pipeline on a GPU-less orchestrator instead: 4 rasteriser processes, 8 surya workers, and ~120 documents each holding a 32-page bitmap window. 47 GB resident on a box with 1 GB free and its swap full. Nothing failed. The pods served the model, the pages came back, and the only symptom was memory. Two readings of one switch, applied to different inputs: * pod_image_for_entry() is now the single resolver, and pod_plan() uses it too, so neither side can drift from the other. * the gate SAYS which side of the line it landed on, once per answer, with the image that decided it and how to change it. Silence was the real cost here -- the local path is a complete, successful read, so an install could be configured for the pod-side pipeline, never once run it, and look exactly like it was working. A lookup that throws now reports itself rather than answering "no". Co-Authored-By:
-
- 15 Oct, 2026 23 commits
-
-
Stefy Lanza (nextime / spora ) authored
A version number, not a feature. The 0.3 line ended on a run of fixes that were all the same shape — a default sized for someone else's workload, a failure that could not be told apart from a success, a resource nobody owned after a restart — and the ones that mattered most were found by reading a production log rather than by a test going red: 0.3.99 a transient control-plane 503 read as a pod boot failure, the teardown swallowed, and a leaked GPU the reaper was structurally blind to (pod age came from the current run's uptime). 0.3.100 a growth line that named a trigger which had not fired. 0.3.101 ten connections per pod against a hundred and twenty-eight pages. 0.3.102 twenty threads for calls that block for minutes, so an OCR run starved the server of threads to answer questions about itself. 0.3.103 every restart abandoning the fleet it had already paid for. Nothing in this commit but the number. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Adoption existed, but only ON DEMAND: it runs inside `acquire()`, when a request needs a pod. Nothing reconciled the registry at boot. So after every restart this deployment's surviving pods sat unowned until either a request happened to want that pool or the reaper's 20-minute grace expired — and an hourly unattended upgrade could pay for a fleet twice, once for the pods it abandoned and again for the ones it rented beside them. Observed 2026-10-16. A forced upgrade at 00:01 orphaned three pods. At 00:04 the surya pool rented a NEW pod while a healthy surya pod from before the restart was still running and still in the registry. The three were reaped at 00:21 having billed $1.11 between them: "not tracked by any pool and its owner stopped reporting". THREE FIXES. 1. A POD THAT WAS BOOTING COULD NEVER BE ADOPTED. register_pod writes the url only once the pod is ready — `# url once ready` — and find_shared_pod skipped a url-less entry in SILENCE. These images take 6-14 minutes to boot, so a restart has a good chance of landing inside one, and the pod it lands on is unadoptable for the rest of its paid life. The pod id is enough to ask RunPod where it went, so an entry with no address now recovers one. 2. TWO OF THE THREE REFUSALS SAID NOTHING. No url and a failed health probe both `continue`d without a word, so "rented beside a pod it could have adopted" and "had nothing to adopt" produced identical logs. That is why tonight's case cannot be pinned down: the evidence was never written. Every refusal now names the pod and the reason. 3. NOTHING RECLAIMED AT STARTUP. `reclaim_registered_pods()` walks the registry when the scaler starts and takes this deployment's pods back into their pools — before `reap_orphans`, deliberately: a pod we are already paying for should rejoin its pool, not be terminated and rented again a minute later. It takes back EVERY pod of a pool, not one; the first version of this reclaimed one of sixteen and left fifteen billing, which is the same bug with extra steps. Bounded by how many the registry holds for that key, since adoption removes nothing by itself and the loop must end whatever it answers. A `pool:` key is left to the on-demand path: its shape comes from whichever model loads first, and guessing would be worse. Also: `scale_up_max_parallel_boots` is now editable in the admin model form (and survives the save). It decides how many GPUs start BILLING together when a pool grows — a floor of 16 fills in waves of 4 at the default — which is exactly the kind of thing an operator should not have to edit models.json for. 17 tests. Suite: 3183 passed, the same 13 known-stale failures. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The admin RunPod page showed: error: Unexpected token '<', "<!doctype "... is not valid JSON That '<' is nginx's HTML error page. The page was not failing to find data; its fetch never got an answer, the proxy gave up, and JSON.parse was handed HTML. Measured from inside the host, mid-run, with 64 pages in flight: /healthz 200 in 0.002s no thread needed /v1/models 200 in 0.006s no thread needed /metrics 200 in 0.18s no thread needed /v1/runpod/status no answer in 30s needs a thread `asyncio.to_thread` hands work to the loop's default executor, and a loop that was never told otherwise builds one with min(32, cpu_count + 4) — 20 threads on a 16-core host. Nothing in this codebase ever called set_default_executor. That number is for incidental background work measured in milliseconds; what we put on it is `pool.acquire()` waiting out a pod boot (MINUTES) and a per-page OCR forward (30-300s each), `ocr.max_concurrency` of them at once. The log says it plainly: "engine 'surya': 8 instance(s) ready, 8 lane(s) each — 64 pages at once". 64 blocking calls, 20 threads. Everything else waits behind a page. Nothing was broken. The server could not get a thread to answer on. codai/api/thread_pool.py now sizes two pools from the concurrency the install is CONFIGURED for rather than from its CPU count — these threads wait on other machines, so what matters is how many remote calls may be outstanding, not how many cores can run Python: * the DEFAULT executor, floor 32, for everything to_thread is used for today; * a separate POD executor, floor 64, for the calls that block for minutes, so that however many pages are in flight a control-plane request still finds a free thread. Both entry points install it (the front blocks on pool.acquire for whole pod boots, so it needs this at least as much as the engine). The pod pool copies the context the way to_thread does — the per-page options and the token trace are ContextVars, and run_in_executor does not copy by itself; a page losing its budget because it changed pools would be a nasty way to learn that. /v1/runpod/spend also took its snapshot ON the loop: pods_status() takes the pool locks and the ledger reads a file, so a lock held by a provisioning thread stalled every request rather than one. It runs in a thread now. A thread waiting on a socket costs a stack and nothing else. The asymmetry is the point: too small costs availability, too large costs memory we have. Also fixes a test of my own from an hour ago that asserted "tomorrow 02:00" is within 24 hours and went red at 00:22. The test was wrong about the clock, not the code about the booking. 15 tests, including the live failure reproduced: fill the pod pool and check the control plane still answers. Suite: 3166 passed, the same 13 known-stale failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
One shared session serves every pod, and `requests`' stock adapter keeps ten connections per host. `ocr.max_concurrency` defaults to 64 and is 128 on the install this came from, so from the ELEVENTH concurrent page onward urllib3 opened a connection, used it, found the pool full on release and threw it away: urllib3.connectionpool: Connection pool is full, discarding connection: 194.68.245.54. Connection pool size: 10 74 of those in five minutes — about fourteen a minute against one pod — while that run's health checks were timing out. Every discarded connection means the next page pays a full TLS handshake to a machine in another country: several round trips of latency each, and handshake CPU on the orchestrator at precisely the moment it is busiest. It does not CAUSE a slow run, but it multiplies one, and it is the part that is ours. The pool is now sized from the concurrency the install is configured for: `ocr.max_concurrency`, floored at 64 and capped at 256. Read at session creation rather than at import, because the config is not loaded when this module is imported — and because an operator raising max_concurrency should not have to know a connection pool exists. An idle connection costs a socket and nothing else, which is the cheap side of this trade. The adapter also carries the floor as its own default, so a `_PinnedAdapter` built anywhere cannot silently fall back to ten. Routing is deliberately unchanged: only https-to-a-bare-IP uses the pinned session, everything else still goes through `requests` untouched, so callers and tests that patch `requests.get` keep working. Not a fix for the slowness that exposed it. That host was running MariaDB at 534% CPU, four Postgres parallel workers, and a backup that had been streaming over SFTP for three hours, against coderai's own 8.8% of one core out of sixteen — the orchestrator was waiting for a CPU, not using one. 9 tests. Suite: 3151 passed, the same 13 known-stale failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pool with one healthy pod rented a second one second after the first became ready, and said: 'datalab-to/surya-ocr-2': growing to meet 1 pod(s) of demand — 1 up, 1 booting It needs one pod. It has one pod. It is renting another. Read plainly that is a double-rent bug, and it took a real investigation to establish that the fleet was behaving exactly as configured. `_should_grow` has two independent triggers: DEMAND (in-flight + waiting) / scale_up_inflight_per_pod, rounded up — a measured number, and the one that may rent several at once. SATURATION every serving pod is at its in-flight threshold, so the NEXT request has nowhere good to go. Anticipatory, single-flight. What happened was saturation: 16 pages landed on the one pod the moment it went ready, 16 at a threshold of 16 is exactly one pod's worth, so demand was already MET and that branch did not fire at all. But the caller computed `_pods_for_demand()` itself, after the fact, and printed it whichever branch had answered — so the line always claimed demand, and in the saturation case reported a number that contradicted the action it was describing. The decision now carries its own reason. `_grow_reason()` returns the sentence of the trigger that fired ('' for no), `_should_grow` is `bool()` of it — the bool contract is unchanged, callers and tests compare with `is True` — and acquire() prints what it was given instead of recomputing a figure of its own. Saturation now says so, and says what it is NOT: growing — every pod is at its 16-in-flight threshold (least loaded carries 16) — renting one more for the next burst, not for work already queued [1 up, 1 booting] Nothing about when a pod is rented changes; only what the log says about why. 10 tests, including the live line as a regression: a saturation rent must not contain the word "demand". Suite: 3142 passed, the same 13 known-stale failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A request for an embedding model rented a pod, which began pulling a 9.65 GB image that had been pushed twenty minutes earlier — a five-minute cold pull, and the ORDINARY case for the first pod on a newly tagged image. Two minutes in, RunPod's control plane answered HTTP 503 once. Three defects then fired in sequence, and the pod that was coming up fine became an untracked GPU billing at $1.09/hr: 21:36:58 pod created — booting 21:38:00 still no port after 62s (status=RUNNING) — image pull in progress 21:38:50 boot FAILED: RunPod API HTTP 503: Service Unavailable 21:38:53 POST /v1/embeddings 502 1. THE BOOT WAIT CONDEMNED THE POD FOR THE API'S FAILURE. Any exception out of the wait is read as "this pod failed to boot", and `get_pod` raised straight through the poll loop. A 503 from the control plane means the question went unanswered; it says nothing about the machine. RunpodTransient (a subclass, so every existing `except RunpodError` is unchanged) now marks 5xx/408/425/429 and transport errors, and the wait keeps polling to the deadline the caller set, saying once that the API is quiet and again when it comes back. A wait that never got a single answer raises transient too — "unknown, not known-bad" — rather than handing the teardown path a verdict nobody reached. 401 and 4xx stay final: those will say the same thing in five seconds. 2. THE TEARDOWN RAN THROUGH THE SAME DEAD API, INSIDE `except Exception: pass`. The attempt most likely to fail is the one that matters most, and its failure was dropped on the floor — no log line anywhere named the pod. terminate_pod now retries with backoff while the error is transient, and the boot handler prints what is still billing, at what rate, and how to stop it. A pod that could not be terminated KEEPS its registry entry: that entry is the only record it exists. `_give_back` had the mirror image of the bug — it unregistered unconditionally, throwing away the record of a live pod — and now forgets only what is actually gone. A boot that failed also ledgers the minutes it burned; no pool ever held that pod, so nothing else would. 3. THE REAPER COULD NEVER HAVE CLEANED UP AFTER EITHER. Pod age came from `runtime.uptimeInSeconds`, which is the CURRENT RUN's uptime and resets on every container restart. The leaked pod reported 234s while it was 11 minutes old, and its uptime reset every few minutes, so it could never cross the 1200s grace — the pod that most needs reaping, one nothing tracks that is failing over and over, was the one pod the reaper was structurally blind to. `lastStatusChange` carries the rental time ("Rented by User: Thu Oct 15 2026 19:36:58 GMT+0000") and was already in the response; list_pods now asks for it. Age is the larger of rental age and uptime — uptime cannot exceed the age, so the pair is still a true lower bound, and a pod whose stamp will not parse is no worse off than before. A future stamp is refused rather than making every pod ancient, and no age at all is still None: the reaper fails closed on that. Found because the image update was the trigger: a new tag is cold in every datacenter, so the first boot after every release spends minutes exposed to exactly this. The next pod proved the image itself was fine — 301s of pull, 22s of warm-up, 200 OK. 27 tests. Suite: 3132 passed, the same 13 known-stale failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The unattended policy has two states: wait for an idle moment, and — on a host that is never idle — book a forced upgrade for the next midnight. Neither is an operator saying "take it now". There was no way to say that at all. The only lever was the booking file: echo 1 > ~/.local/state/coderai/force-upgrade-at which works by accident (epoch 1 is in the past, so the next tick fires), is written nowhere, and is NOT self-clearing when the run finds nothing to upgrade. Observed on a production orchestrator: a `1` left from one forced release turned every hourly tick for the next day into "a FORCED upgrade is booked", silently disarming the idle check that protects live requests. Nobody asked for that; it was the residue of a one-off. So the three things an operator actually wants are options now: --force upgrade and restart on THIS run, idle or not, and leave no booking behind. It sets FORCED before the upgrade block, so it also lands an upgrade already sitting in the image behind deferrals — that host is precisely who types this. --force-at WHEN book one (anything date -d takes). No second timer: the hourly tick at or after WHEN takes it. --cancel-force drop a booking, whether an operator made it or the script armed it after MAX_DEFERRALS busy ticks. The arming message now names this instead of telling you to delete a file. The two booking edits return before the runner check and before anything touches a container engine, so they work on a host where the upgrade itself could not run, and they are written to the same log as the runs they change — a schedule change belongs in the history of the schedule. --help exists, and an unknown option is a usage error rather than a silent unattended run under a typo. Also drops one install's name from a sentence in the guide that did not need it. 13 tests. Suite: 3105 passed, the same 13 known-stale failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The suite has carried 25 failures for long enough that a real regression could hide in them — which is the failure mode that cost a full day this week. Twelve are fixed here. Every one was a test pinning behaviour that had been deliberately changed; no product bug was found. Two were actively harmful rather than merely stale: * test_build_hardware_summary faked `torch` and asserted 10240 MB of free VRAM. The summary moved off torch.cuda.mem_get_info on purpose — that path pinned a ~256 MiB CUDA context just to answer a hardware question — so the fake went unused and the test measured the REAL card of whatever machine ran it. It read 24111 here. A test that depends on the host's GPU is not a test; it now patches visible_gpu_memory(), which is what the code calls. * test_main_wiring_falls_back drove the whole of main() through a wall of stubs to reach one line, so every new config field or call surfaced as an AttributeError raised from INSIDE the code under test. Three of them, each reading exactly like a product bug. It now asserts the thing it exists for — an invalid broker config yields a disabled runtime rather than stopping the server — directly. The rest, by cause: * the broker protocol's accept field is `status: "ok"`. Six fixtures sent `accepted: true`, a field that appears nowhere in the client, which sends {"event":"pending","status":"pending"} itself and documents the server's heartbeat reply as status=ok. Registration now succeeds in the fakes. * `v1/…` endpoint lists were frozen copies that broke every time the product gained a route (eight by the time anyone looked) while saying nothing about whether the advertised routes exist. The studio-endpoints test now checks them against the app's real routes — the contract its name always claimed — comparing route SHAPES, since parameter names are local detail. * whisper: posting an existing id is an EDIT. The handler says so where it does it ("Otherwise editing an existing whisper-server model would either 409 or silently create a duplicate config"); the test demanded the 409 that comment describes removing. * whisper runtime fake: `is_running` always answered True and then expected start() to be called, and it defined `cleanup` but not `stop` — the unload path calls stop(), so it raised into an `except Exception: pass` and the server looked un-stopped with nothing saying why. * calling a FastAPI handler as a function leaves unpassed parameters as Form(...) sentinels: "unsupported operand type(s) for +: 'Form' and 'Form'". That test drives the real route now. * there is no GET /admin/api/settings any more, and the LongCat image builds its 3.10 venv in a python:3.10-slim stage rather than with a literal `python3.10 -m venv`. One real inconsistency surfaced on the way: the broker is advertised `v1/video/compose/{id}` where the app serves `{job_id}`. Shape-compatible, so nothing breaks; the advertised list is simply inaccurate. Not changed here, because what we advertise is a wire-visible contract. 13 remain: the broker handshake tests encode server-greets-first against a client that says "Client speaks first", plus streaming and studio shapes. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
THE SURYA BUG. `/v1/ocr/attach/<lease>/…` is an OCR engine attaching to THIS install's gateway. The lease is minted here and lives in this process's memory — and the remote gateway was FORWARDING those requests to the pod, which correctly refused a token it never issued: 404 {"error": {"message": "unknown OCR engine lease"}} surya's client swallows that and returns an empty generation, so every page came back unread with the GPU never asked to read one. Measured on a clean 2-page PDF that chandra (2212 chars) and olmOCR (1992) both read: surya returned 2 characters, and the pod's vLLM logged a single 6-token warm-up for the whole document. $88 of surya time in one day produced nothing. The gateway behind that prefix does its OWN forwarding, with its own pool and its own lease. That is the hop, and it has to start here. `/v1/ocr` itself stays routable, so pinning the attachment does not strand the subsystem. This also corrects an earlier diagnosis of mine: v0.3.87 treated this as a lease VOIDED BY A RESTART and shipped a re-resolve for it. No restart was involved — the lease was wrong from the first page, which is why surya has been broken continuously rather than only after restarts. That work was not wasted (the retry cap took four paid generations per page down to two, and the 502 plus `unread_pages` are what made this diagnosable at all) but it did not address the cause. Third instance today of one mistake: the gateway forwards anything under /v1/ that looks like a model request, including this install's own control plane. /v1/runpod rented a pod to deliver a teardown; /v1/loras was staying local by luck; now the OCR lease. All three are pinned. The structural answer — making control-plane routes opt IN to forwarding — is not in this commit. Also in here, from the fleet-hygiene pass: * accelerate added to every profile that carries transformers (audio, ocr, stt, tts, voice, tts-xtts.venv-TTS), with a test that fails if a new profile reintroduces the gap. Same latent fault that broke embeddings. * an embeddings pod is warmed on /v1/embeddings, not with a chat completion it refuses by design — that 404 was printed on every boot of a healthy pod and cost a wrong diagnosis. * the seed plan is announced once, not every fifteen seconds while a pod boots, so a boot log stays readable when it matters. * `pool:` is dropped from pod NAMES (ownership is the prefix, never the rest). * the reaper owns a pod by REGISTRY as well as by name, so renaming deployment_id no longer orphans pods that are already running and billing. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reported from production, minutes after v0.3.96 made an adapter a model of its own on the pod: 503 Model 'diogene-alpha' failed to load: Unknown backend: cpu Only the pool's OWN entry was seeded with its engine. Siblings got none, so the adapter arrived on the pod with no backend and fell through to the pod's default. Verified against the live catalogue: the seed genuinely had no `backend` key. This had always been wrong and could not bite before, because an adapter collapsed into its base on the pod and was never a model of its own to LOAD. Making it one — which was the right fix — is what exposed it. Every sibling now takes the POOL's engine, not its own. A pool is one pod running one image, so siblings cannot each have a different engine: seeding one would describe something the pod cannot satisfy, and the failure would land minutes later inside a rented machine. A test asserting the opposite was written and then corrected. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Five fixes, two of them reported from production in the same hour. EMPTY REPLIES from a chat naming `diogene-alpha`, while `qwen38` — the same weights on the same pod — answered normally. Parser selection reads the model NAME: if 'qwen' in model_lower: -> QwenParser `qwen38-awq` matches; `diogene-alpha` matches nothing and fell to the generic parser, so the model's <think>…</think> block was never separated from the answer. With a modest max_tokens the whole budget goes into reasoning that never closes, and the caller sees an empty reply from a 200 — the hardest kind of wrong to see. A LoRA is named for what it was trained to do; it is the same FAMILY as the weights it adapts, so a name matching nothing now resolves to the base's. Nothing is discarded: the thinking comes back as `reasoning` and `reasoning_content`, as it always did for the base. This also corrects a reading of my own: every temperature-0 comparison in this chain showed the adapter leaking its reasoning while the base answered cleanly, and I took that as evidence the adapter was not loading. It was the parser. /v1/chunk with X-Coderai-Placement: remote answered 502 "nothing configures 'embeddings' to run remotely" while a coderai-embeddings pod — which has the chunker in it — sat there serving bge-m3. A capability-only request names no model, so the per-model placement that moved everything else cannot place it. It now borrows the destination of a model OF THE SAME CAPABILITY that is already remote. Same capability only (an embeddings pod has no OCR deps), and only on an explicit `remote`: nothing should rent a GPU to run a regex split because nobody said otherwise. Registering an adapter is not serving it. vLLM takes its adapters at launch and the code that adds one to a live engine sits on the model-LOAD path, which does not run again for a model this instance already loaded. So a pod could be sent an adapter, register it, report it in /v1/loras and go on answering from the base. Registration now hands it to the running engine, swapping a stale one. A stale content id repairs itself. v0.3.94 fixed what new uploads are stored as; an entry already registered under a non-canonical id stayed broken with no symptom but base-model answers. Seeding and pushing resolve an id against the CONTENT first. And `it-legal` now sets min_chunk_chars: 300 (strict stays 0) — it is the OCR-aware template, so it meets short trailing sections more often, and a 56-character chunk is too small to carry meaning and long enough to match. A runt folds only into a chunk that can CARRY it, so a document of short sections stays several chunks instead of collapsing. Absorption may never cross conclusioni <-> dispositivo at any size: what a party asked for and what the court decided, and a recorded label is a weaker guarantee than refusing the merge. NOTE: this release MOVES CHUNK BOUNDARIES for anyone using `it-legal`. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/chunk already rode the embeddings capability, so it could be placed here, on a RunPod pod (the coderai-embeddings image) or on a cluster node. Verified end to end for the first time rather than inferred from the routing table: a capability image really serves the route, and the middleware really forwards it ("[remote-gateway] /v1/chunk -> … (capability 'embeddings')"). That testing found a real defect. A NAMED structure template was forwarded as a bare name whenever the name was a BUILT-IN, on the reasoning that every coderai has those. True of the name; not true of what it means. `it-legal` was widened to accept markdown headings in 0.3.87 and `it-legal-strict` did not exist before it — so a pod on an older image resolves the same name to different anchors, or 404s on one it has never heard of. A pod fleet is mixed-version by nature: one rented an hour ago runs whatever `:latest` meant then. Chunk boundaries that differ by which machine answered make a retrieval index unmeasurable, which is the one thing docs/chunking.md exists to prevent. Every named template is inlined now, built-ins included: a few hundred bytes on the request, and the boundaries become a property of THIS install's version. An unknown name is still forwarded untouched — resolution failing here must not swallow the request, and the remote's `unknown_template` is a clear error. Six tests pin the placement itself: local by default (a regex split must not rent a GPU), forwarded to a node, left here when only another capability is remote, the template inlined on the wire, and one request pinned local by header. Plus the note that a per-model remote does NOT move chunking, because a chunk request carries no model and only the capability map places it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The root cause under four releases of "the adapter is not applied". Measured on a live pod: catalogue says : sha256:b8eac366… (hash of the tar a client sent) this bundler's own : sha256:2e820c… (same content, its own packing) pod lookup b8eac366 : 404 pod upload returns : {"id":"sha256:2e820c…","existed":true} `save_lora_blob` stored an uploaded bundle under the hash of THE BYTES THAT ARRIVED. A tar of one adapter made by GNU tar, by this bundler, or by anything else hashes differently — mtimes, ownership, member order — so the id recorded in the catalogue was one nobody else could reproduce. Every later push re-bundled the unpacked directory, stored the canonical id, and the pod looked up the uploaded one and found nothing. It had the bytes the whole time, under another name, so it loaded the base model and answered under the adapter's name. Fluently, with nothing in the reply to say so. An uploaded bundle is now unpacked FIRST and stored under the id of this bundler's packing of it, which is what every side computes. /v1/loras/upload returns that id, so what a client registers is what a pod can resolve. This is also why it took four attempts: the engine flags, the seeding and the entry propagation were each genuinely broken and each fix was real, but underneath them the two sides were never going to agree on the id. _load_lora_runtime had two bare `return False` paths — no adapter spec, or an id the blob store cannot resolve — which is exactly why this hid for three releases while every other layer was examined. Both say what happened now, and the second says the thing that matters: the catalogue and the store disagree about this adapter's id. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Measured on a live pod. The pool gate from v0.3.91 worked — [lora] no adapters yet, but this engine serves a pool — starting with LoRA enabled so one can be added without a reload [vllm] launching: … --enable-lora --max-lora… — and the adapter was still dropped: pod seeded expecting : sha256:2e820c… (the local adapter directory) re-registered as : sha256:b8eac366… (an uploaded blob) orchestrator pushed : b8eac366… ✓ 159 MB The bytes arrived. The pod's CATALOGUE did not. It was seeded at boot with the first id, /v1/loras/register repointed the entry at the second, and nothing told the pod — so its engine looked for an id nobody holds, dropped the adapter, and served the base under its name. (The bundle hash is deterministic; the two ids differ because the adapter genuinely changed, not because hashing is unstable.) Two keys missing a version, the same shape as every other bug in this area: * _ensure_adapters sent the BYTES only. It sends bytes and entry now. * teach_model keyed "already taught" on (pod, model), so a re-register could never reach a pod it had spoken to once. Keyed on the adapter's source. /v1/loras listed only the TRAINING registry, so an adapter that arrived by upload+register was invisible there — a successful registration looked exactly like a failed one, which is how this was found. It lists both now, tagged `trained` or `registered`, with the adapter id, base and pool. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reported from production: 500 {"detail": "Generation error: the pod answered 500: Generation error: 400 Bad Request … \"auto\" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set"} Not a regression: `tools` has never worked on a vLLM-backed model here. The proxy forwards them correctly — the remote decides what it supports — and nothing ever launched vLLM able to take them. A request-time 400 from a server that started cleanly, for a capability the catalogue advertises, naming two flags the caller has no way to set. A tool call is TEXT the model emits in its own trained format, so the parser belongs to the model FAMILY rather than to the deployment, which is what makes a default possible at all. `tool_call_parser` defaults to "auto" and picks from the model name (qwen->hermes, llama-3->llama3_json, mistral, deepseek_v3, glm45, qwen3_coder…), longest match first so qwen3-coder is not swallowed by qwen. An explicit value wins, per install or per model in its `vllm` block — and that block now travels to a pod, so a pod serves a model the way this install does. "off" is honoured rather than overridden: a family guessed wrongly does not fail, it parses tool calls into nonsense, and silently producing wrong tool ARGUMENTS is worse than refusing the request. Both flags go together or neither: vLLM refuses to START with --enable-auto-tool-choice and no parser, and an engine that will not boot is worse than one that cannot do tool calls. An unrecognised family logs which setting to use, because otherwise the symptom lands on the client as a 400. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Re-registering an adapter updated the catalogue and the pod went on serving the OLD weights. Two caches, neither of which had a version in its key: * the orchestrator remembered "this pod already has this adapter" by NAME, so the new bytes were never sent; * and the engine was asked "do you serve this name?" — which a stale adapter answers yes to. It kept what it was launched with. So the update looked applied from every angle a client can see: the catalogue had the new blob, the name worked, the request succeeded. The only symptom was output identical to the previous version, which is exactly the symptom nobody can see from a 200. Both are keyed on the adapter's SOURCE now. When it changes, the bytes are pushed, the old adapter is unloaded from the running engine and the new one takes its name — no reload of the base model, nothing else on the pod disturbed. When the swap cannot be done in place the engine is restarted with the new adapter, and says which it is doing. An engine coderai holds no record for is never treated as stale: unloading an adapter on a guess is worse than serving the one that is there. Content-addressed versioning is what makes this safe to run on every retrain — bytes that have not changed are recognised as the same adapter and nothing reloads. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/loras/upload put the bytes somewhere; nothing made them answerable. The last step was editing models.json by hand, which is not a step a team retraining adapters can automate — so the answer to "how do we send you a QLoRA?" was "send it to us and we will register it", with a human in it. POST /v1/loras/register {name, base, adapter} closes that. It joins the BASE'S POOL by default, which is the entire economy of the thing: the base weights are already loaded on a rented card, an adapter is tens of megabytes of deltas, and vLLM applies them per request. It inherits the base's placement, card choice and budgets, and deliberately does NOT inherit min_pods or keep_warm — an adapter rides the base's pod, it does not ask for one of its own. Re-registering a name with a new blob is how a retrain ships. A name that already belongs to a real model is a 409 rather than a replacement, and an adapter whose bytes we do not hold is a 404 that names the upload endpoint: a pod sent a content id nobody holds serves the BASE instead, silently, which is the failure this whole area has been about. Identity is (weights, name) here too. An adapter and its base share a `path` by definition, so the lookup is by id-or-alias and never by path. /v1/loras is also pinned local in the gateway. An upload must land in THIS blob store — it is where a pod is sent its adapters from, and the content id this install hands out has to resolve here. It carries no `model` field, so it happened to stay local by accident; that is luck, not design, and the same shape of accident as /v1/runpod, which did not stay local and rented a pod to deliver a teardown. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
v0.3.88 made the pod FIND the adapter. The live pod then said: [lora] 'diogene-alpha': adapter sha256:2e820c… is not in this machine's blob store — serving without it [vllm] launching: … --served-model-name cyankiwi/Qwen3.8-27B-AWQ-INT4 qwen38-awq Found, and dropped. A seed names an adapter by CONTENT ID — computed here so the pod can be told about it before it exists — and the bytes travel separately. They were pushed on the first `acquire`, but the pool's WARM-UP request loads the model seconds after boot, long before any acquire. So vLLM started without the adapter, and `--enable-lora` cannot be turned on afterwards. Two changes, both gated on the model declaring a `pool`, because a pool is the explicit statement that this engine serves several names over one copy of the weights: * a pool engine launches with `--enable-lora` and the capacity flags even with no adapter resolvable yet. The flags that must be decided at LAUNCH are the capacity ones — the machinery, the slot size, how many adapters may be live in one batch — not the list. Opening the door costs a buffer allocation; keeping it shut costs a full reload of a 27B on a card billed by the hour, the first time an adapter arrives. * an adapter is then handed to the RUNNING engine (`/v1/load_lora_adapter`, vLLM >= 0.9 behind VLLM_ALLOW_RUNTIME_LORA_UPDATING, which the launch sets whenever it enabled LoRA) instead of restarting it. The restart stays as the fallback and is still reached on every path that cannot use the API — an engine started without `--enable-lora`, an older vLLM, an adapter whose rank exceeds the slot size. A pod rented an hour ago may be running an older image than the one we would launch today, so "it should be supported" is not something this can assume. What this buys: one rented card, one copy of the base model, and QLoRAs added or swapped per request by name with no reload. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
v0.3.86 made the pod register `diogene-alpha` as a model of its own, and it still answered with the base model's weights. The pod log is the whole story: [seed] registered qwen38-awq in text_models [seed] registered diogene-alpha in text_models [vllm] launching: … --model cyankiwi/Qwen3.8-27B-AWQ-INT4 --served-model-name cyankiwi/Qwen3.8-27B-AWQ-INT4 No --enable-lora. Both models were registered; the adapter was never collected. `_sibling_lora_specs` skips "myself" by comparing `entry["id"]` — and the only kind of entry a POD holds is a SEEDED one, which carries `alias` and no `id`. So the comparison was "" == "", every sibling matched, and all of them were excluded. Keyed on alias-or-id now, with identity by object for the one entry that really is the model itself. That fix alone is not enough: a seed names an adapter by CONTENT ID, computed here so the pod can be told about it before it exists, and the pod still has to receive the BYTES. Until it does, vLLM launches without the adapter. So the backend now pushes a model's local adapters to the pod before the first request rather than after a failure — vLLM registers adapters when it STARTS, and bytes that arrive late cost a full reload of the base model to take effect. Where an adapter still cannot be resolved, the pod does not serve that NAME at all: it is in neither --lora-modules nor --served-model-name, so the request 404s and the teach-and-retry path picks it up. A 404 is recoverable. The base model answering fluently under an adapter's name is not — nothing downstream can detect it, which is how this survived two releases that each looked fixed. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Seven items from a production field report. Three of them were one cause. /v1/runpod is NOT in the remote gateway's local list. It carries a `model` field like an inference request, so the gateway read it as one, resolved the model to its pod, and FORWARDED the control-plane call there. Asking for six pods back rented a seventh to ask on; the pod has no runpod configuration of its own, so it answered 404 {"detail": "'datalab-to/surya-ocr-2' has no runpod block"} which reads exactly like the orchestrator refusing a name it should know. The same call blocked for minutes while a pod booted to receive it. An operator could not turn off $3.54/hr from the API. Reproduced here while testing the fix: two teardown requests rented two A40s, which is the clearest possible statement of the bug. surya answered 200 with empty pages for a whole document. Three failures in a row, each of which had to be fixed: * its client catches the HTTP error itself and returns a generation with no tokens, so the lease check in recognize_image — which only ran in `except` — never ran, and the pod address was never re-resolved; * the repetition detector was handed that empty string, found no progress in it and re-ran the page at a rising temperature, four full paid generations per page, `tokens: [0,0,0,0]`; * and the page came back as read. We had already measured the ink and said there IS something on it; the judgement never reached the response, so a client treating an empty read as an unreadable scan consumed roughly a thousand documents from its queue and wrote them off. A page with ink and no text is now `unread`: it goes to the fallback engine like any other degraded page, it is listed in `unread_pages`, and a document whose every inked page came back empty is no longer a 200 (`ocr.unread_status`, 502 by default; a partially read document stays 200 with the list). `bge-m3` and `BAAI/bge-m3` each rented their own A40 — the pool key was whichever name the request carried. Keyed on the model ENTRY now, so every name for one set of weights lands on one pool. A client read timeout cannot bound an OCR call: it times each READ, so a response that dribbles never trips it and one stayed open nineteen minutes against timeout=420. `ocr.request_timeout_s` is the server-side stopwatch. Chunking: `it-legal` accepts a decorated heading (`**DISPOSITIVO**`) where the implementation it ports does not, which splits one more section and leaves a short trailing chunk. That was true and undocumented — the table printed the strict pattern. Documented now, with `it-legal-strict` named as the byte-identical one for re-chunking an existing index. `min_chunk_chars` folds a runt chunk into its predecessor, off by default because merging moves a boundary, and never silently: the absorbed section label is reported, since `conclusioni` is what a party asked for and `dispositivo` is what the court decided. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A shared pool is one pod serving several names over one copy of the weights — a base model and its adapters. Live, every request naming the adapter got 404 Client Error: Not Found for url: https://…/v1/chat/completions and kept getting it: the pool keeps the pod warm, so nothing replaced it. The pod, asked directly, listed only the base and reported {"loras": []}. Which of the two names worked depended on which model happened to rent the pod, so it looked intermittent rather than wrong. Five places decided "same model?" by comparing `path`, and a LoRA's path IS its base's path: * the orchestrator seeded the other pool members with no served name, so they arrived with no alias and were registered under their path; * the pod's boot seeder skipped the second entry as already known; * register_model_runtime did the same, which made teaching a running pod a no-op that answered "already known"; * the engine-side seed filter in main.py did it a third time; * and vLLM's service key included the served name, so had the adapter got through it would have started a SECOND copy of a 27B on the same card. Identity is now (weights, name) in all five, and one engine serves one set of weights under every name configured for it: the base under its own, each adapter through --lora-modules, which --served-model-name must not repeat. Adoption checks the image, the GPU class and the bearer token — none of which say anything about the catalogue. So a pod that cannot serve what the pool promises is now TAUGHT: the adapter bytes are pushed, the entry is registered, and the same request is retried on the same pod. Renting a second card to serve a name the running one can hold is the answer this avoids. Once per pod per model; a pod that still refuses is a real failure. An engine is also relaunched when it does not serve the name asked of it: vLLM registers adapters at LAUNCH, and a pod is taught after it boots. Only an ANSWERED probe that omits the name forces that — restarting a loaded 27B because one probe timed out would be worse than the doubt. Finally, raise_for_status() discarded the pod's body, which is the whole message: it names the models the pod actually has. The caller saw only the status line, which reads like a routing fault. The pod's own words now travel. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.3.84 made `_is_bge_m3` see `sparse_linear.pt` on a machine that had never downloaded the repo, so the multi-vector backend was finally selected on a pod. It then loaded the heads with the same cache-only lookup, and `from_pretrained` does not fetch them: it pulls the config and the weights the config NAMES, and bge-m3's sparse/colbert heads are plain tensors sitting beside the model. So they were still absent, and the request came back HTTP 400 "This model has no sparse head." which reads as a property of the MODEL rather than of the disk. `_fetch_repo_file` is the companion to `_resolve_repo_file` for files that have to be READ rather than merely detected: cache first, then a real hf_hub_download. Offline short-circuits, a local directory is the whole answer, and a failure returns None -- the dense path, which the caller already handles -- because a head that cannot be fetched must not turn into a failed load. Both halves of this were the same mistake made twice: a cache-only helper answering a question about existence on a machine whose cache is empty by design. A rented pod is that machine every single time. Needs a coderai-embeddings rebuild: this code runs on the pod. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`_is_bge_m3` keys off `sparse_linear.pt` -- the definitive signal for a multi-vector embedder, since plain dense models do not ship one. It asked `_resolve_repo_file`, which answers from the local disk and NEVER the network. Right for "give me the path", wrong for "does this exist", and the difference is invisible until the model has never been downloaded. On a rented pod it has never been downloaded. The pod is disposable and its cache starts empty, so the check ran before anything could satisfy it: on every pod, every time, bge-m3 loaded through the DENSE sentence-transformers path. The request for sparse was then refused with a 400 telling the caller to serve the model on the coderai-embeddings image -- which is the image it was already running on. Measured end to end on a real A40: 379 seconds, then that. `_repo_has_file` keeps the local answer first and, only when the file is not on disk, asks the Hub for the repo's file LIST: metadata, no download, cached once per process. Offline installs short-circuit on HF_HUB_OFFLINE, a local directory is taken as the whole answer (asking the Hub about a filesystem path would look up a stranger's model of the same name), and any failure answers False exactly as before -- a metadata lookup must never be able to fail a load. Needs a coderai-embeddings rebuild: this code runs on the pod. Co-Authored-By:
-