- 15 Oct, 2026 28 commits
-
-
Stefy Lanza (nextime / spora ) authoredThe admin RunPod page showed: error: Unexpected token '<', "<!doctype "... is not valid JSON That '<' is nginx's HTML error page. The page was not failing to find data; its fetch never got an answer, the proxy gave up, and JSON.parse was handed HTML. Measured from inside the host, mid-run, with 64 pages in flight: /healthz 200 in 0.002s no thread needed /v1/models 200 in 0.006s no thread needed /metrics 200 in 0.18s no thread needed /v1/runpod/status no answer in 30s needs a thread `asyncio.to_thread` hands work to the loop's default executor, and a loop that was never told otherwise builds one with min(32, cpu_count + 4) — 20 threads on a 16-core host. Nothing in this codebase ever called set_default_executor. That number is for incidental background work measured in milliseconds; what we put on it is `pool.acquire()` waiting out a pod boot (MINUTES) and a per-page OCR forward (30-300s each), `ocr.max_concurrency` of them at once. The log says it plainly: "engine 'surya': 8 instance(s) ready, 8 lane(s) each — 64 pages at once". 64 blocking calls, 20 threads. Everything else waits behind a page. Nothing was broken. The server could not get a thread to answer on. codai/api/thread_pool.py now sizes two pools from the concurrency the install is CONFIGURED for rather than from its CPU count — these threads wait on other machines, so what matters is how many remote calls may be outstanding, not how many cores can run Python: * the DEFAULT executor, floor 32, for everything to_thread is used for today; * a separate POD executor, floor 64, for the calls that block for minutes, so that however many pages are in flight a control-plane request still finds a free thread. Both entry points install it (the front blocks on pool.acquire for whole pod boots, so it needs this at least as much as the engine). The pod pool copies the context the way to_thread does — the per-page options and the token trace are ContextVars, and run_in_executor does not copy by itself; a page losing its budget because it changed pools would be a nasty way to learn that. /v1/runpod/spend also took its snapshot ON the loop: pods_status() takes the pool locks and the ledger reads a file, so a lock held by a provisioning thread stalled every request rather than one. It runs in a thread now. A thread waiting on a socket costs a stack and nothing else. The asymmetry is the point: too small costs availability, too large costs memory we have. Also fixes a test of my own from an hour ago that asserted "tomorrow 02:00" is within 24 hours and went red at 00:22. The test was wrong about the clock, not the code about the booking. 15 tests, including the live failure reproduced: fill the pod pool and check the control plane still answers. Suite: 3166 passed, the same 13 known-stale failures. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
-
Stefy Lanza (nextime / spora ) authored
One shared session serves every pod, and `requests`' stock adapter keeps ten connections per host. `ocr.max_concurrency` defaults to 64 and is 128 on the install this came from, so from the ELEVENTH concurrent page onward urllib3 opened a connection, used it, found the pool full on release and threw it away: urllib3.connectionpool: Connection pool is full, discarding connection: 194.68.245.54. Connection pool size: 10 74 of those in five minutes — about fourteen a minute against one pod — while that run's health checks were timing out. Every discarded connection means the next page pays a full TLS handshake to a machine in another country: several round trips of latency each, and handshake CPU on the orchestrator at precisely the moment it is busiest. It does not CAUSE a slow run, but it multiplies one, and it is the part that is ours. The pool is now sized from the concurrency the install is configured for: `ocr.max_concurrency`, floored at 64 and capped at 256. Read at session creation rather than at import, because the config is not loaded when this module is imported — and because an operator raising max_concurrency should not have to know a connection pool exists. An idle connection costs a socket and nothing else, which is the cheap side of this trade. The adapter also carries the floor as its own default, so a `_PinnedAdapter` built anywhere cannot silently fall back to ten. Routing is deliberately unchanged: only https-to-a-bare-IP uses the pinned session, everything else still goes through `requests` untouched, so callers and tests that patch `requests.get` keep working. Not a fix for the slowness that exposed it. That host was running MariaDB at 534% CPU, four Postgres parallel workers, and a backup that had been streaming over SFTP for three hours, against coderai's own 8.8% of one core out of sixteen — the orchestrator was waiting for a CPU, not using one. 9 tests. Suite: 3151 passed, the same 13 known-stale failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pool with one healthy pod rented a second one second after the first became ready, and said: 'datalab-to/surya-ocr-2': growing to meet 1 pod(s) of demand — 1 up, 1 booting It needs one pod. It has one pod. It is renting another. Read plainly that is a double-rent bug, and it took a real investigation to establish that the fleet was behaving exactly as configured. `_should_grow` has two independent triggers: DEMAND (in-flight + waiting) / scale_up_inflight_per_pod, rounded up — a measured number, and the one that may rent several at once. SATURATION every serving pod is at its in-flight threshold, so the NEXT request has nowhere good to go. Anticipatory, single-flight. What happened was saturation: 16 pages landed on the one pod the moment it went ready, 16 at a threshold of 16 is exactly one pod's worth, so demand was already MET and that branch did not fire at all. But the caller computed `_pods_for_demand()` itself, after the fact, and printed it whichever branch had answered — so the line always claimed demand, and in the saturation case reported a number that contradicted the action it was describing. The decision now carries its own reason. `_grow_reason()` returns the sentence of the trigger that fired ('' for no), `_should_grow` is `bool()` of it — the bool contract is unchanged, callers and tests compare with `is True` — and acquire() prints what it was given instead of recomputing a figure of its own. Saturation now says so, and says what it is NOT: growing — every pod is at its 16-in-flight threshold (least loaded carries 16) — renting one more for the next burst, not for work already queued [1 up, 1 booting] Nothing about when a pod is rented changes; only what the log says about why. 10 tests, including the live line as a regression: a saturation rent must not contain the word "demand". Suite: 3142 passed, the same 13 known-stale failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A request for an embedding model rented a pod, which began pulling a 9.65 GB image that had been pushed twenty minutes earlier — a five-minute cold pull, and the ORDINARY case for the first pod on a newly tagged image. Two minutes in, RunPod's control plane answered HTTP 503 once. Three defects then fired in sequence, and the pod that was coming up fine became an untracked GPU billing at $1.09/hr: 21:36:58 pod created — booting 21:38:00 still no port after 62s (status=RUNNING) — image pull in progress 21:38:50 boot FAILED: RunPod API HTTP 503: Service Unavailable 21:38:53 POST /v1/embeddings 502 1. THE BOOT WAIT CONDEMNED THE POD FOR THE API'S FAILURE. Any exception out of the wait is read as "this pod failed to boot", and `get_pod` raised straight through the poll loop. A 503 from the control plane means the question went unanswered; it says nothing about the machine. RunpodTransient (a subclass, so every existing `except RunpodError` is unchanged) now marks 5xx/408/425/429 and transport errors, and the wait keeps polling to the deadline the caller set, saying once that the API is quiet and again when it comes back. A wait that never got a single answer raises transient too — "unknown, not known-bad" — rather than handing the teardown path a verdict nobody reached. 401 and 4xx stay final: those will say the same thing in five seconds. 2. THE TEARDOWN RAN THROUGH THE SAME DEAD API, INSIDE `except Exception: pass`. The attempt most likely to fail is the one that matters most, and its failure was dropped on the floor — no log line anywhere named the pod. terminate_pod now retries with backoff while the error is transient, and the boot handler prints what is still billing, at what rate, and how to stop it. A pod that could not be terminated KEEPS its registry entry: that entry is the only record it exists. `_give_back` had the mirror image of the bug — it unregistered unconditionally, throwing away the record of a live pod — and now forgets only what is actually gone. A boot that failed also ledgers the minutes it burned; no pool ever held that pod, so nothing else would. 3. THE REAPER COULD NEVER HAVE CLEANED UP AFTER EITHER. Pod age came from `runtime.uptimeInSeconds`, which is the CURRENT RUN's uptime and resets on every container restart. The leaked pod reported 234s while it was 11 minutes old, and its uptime reset every few minutes, so it could never cross the 1200s grace — the pod that most needs reaping, one nothing tracks that is failing over and over, was the one pod the reaper was structurally blind to. `lastStatusChange` carries the rental time ("Rented by User: Thu Oct 15 2026 19:36:58 GMT+0000") and was already in the response; list_pods now asks for it. Age is the larger of rental age and uptime — uptime cannot exceed the age, so the pair is still a true lower bound, and a pod whose stamp will not parse is no worse off than before. A future stamp is refused rather than making every pod ancient, and no age at all is still None: the reaper fails closed on that. Found because the image update was the trigger: a new tag is cold in every datacenter, so the first boot after every release spends minutes exposed to exactly this. The next pod proved the image itself was fine — 301s of pull, 22s of warm-up, 200 OK. 27 tests. Suite: 3132 passed, the same 13 known-stale failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The unattended policy has two states: wait for an idle moment, and — on a host that is never idle — book a forced upgrade for the next midnight. Neither is an operator saying "take it now". There was no way to say that at all. The only lever was the booking file: echo 1 > ~/.local/state/coderai/force-upgrade-at which works by accident (epoch 1 is in the past, so the next tick fires), is written nowhere, and is NOT self-clearing when the run finds nothing to upgrade. Observed on a production orchestrator: a `1` left from one forced release turned every hourly tick for the next day into "a FORCED upgrade is booked", silently disarming the idle check that protects live requests. Nobody asked for that; it was the residue of a one-off. So the three things an operator actually wants are options now: --force upgrade and restart on THIS run, idle or not, and leave no booking behind. It sets FORCED before the upgrade block, so it also lands an upgrade already sitting in the image behind deferrals — that host is precisely who types this. --force-at WHEN book one (anything date -d takes). No second timer: the hourly tick at or after WHEN takes it. --cancel-force drop a booking, whether an operator made it or the script armed it after MAX_DEFERRALS busy ticks. The arming message now names this instead of telling you to delete a file. The two booking edits return before the runner check and before anything touches a container engine, so they work on a host where the upgrade itself could not run, and they are written to the same log as the runs they change — a schedule change belongs in the history of the schedule. --help exists, and an unknown option is a usage error rather than a silent unattended run under a typo. Also drops one install's name from a sentence in the guide that did not need it. 13 tests. Suite: 3105 passed, the same 13 known-stale failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The suite has carried 25 failures for long enough that a real regression could hide in them — which is the failure mode that cost a full day this week. Twelve are fixed here. Every one was a test pinning behaviour that had been deliberately changed; no product bug was found. Two were actively harmful rather than merely stale: * test_build_hardware_summary faked `torch` and asserted 10240 MB of free VRAM. The summary moved off torch.cuda.mem_get_info on purpose — that path pinned a ~256 MiB CUDA context just to answer a hardware question — so the fake went unused and the test measured the REAL card of whatever machine ran it. It read 24111 here. A test that depends on the host's GPU is not a test; it now patches visible_gpu_memory(), which is what the code calls. * test_main_wiring_falls_back drove the whole of main() through a wall of stubs to reach one line, so every new config field or call surfaced as an AttributeError raised from INSIDE the code under test. Three of them, each reading exactly like a product bug. It now asserts the thing it exists for — an invalid broker config yields a disabled runtime rather than stopping the server — directly. The rest, by cause: * the broker protocol's accept field is `status: "ok"`. Six fixtures sent `accepted: true`, a field that appears nowhere in the client, which sends {"event":"pending","status":"pending"} itself and documents the server's heartbeat reply as status=ok. Registration now succeeds in the fakes. * `v1/…` endpoint lists were frozen copies that broke every time the product gained a route (eight by the time anyone looked) while saying nothing about whether the advertised routes exist. The studio-endpoints test now checks them against the app's real routes — the contract its name always claimed — comparing route SHAPES, since parameter names are local detail. * whisper: posting an existing id is an EDIT. The handler says so where it does it ("Otherwise editing an existing whisper-server model would either 409 or silently create a duplicate config"); the test demanded the 409 that comment describes removing. * whisper runtime fake: `is_running` always answered True and then expected start() to be called, and it defined `cleanup` but not `stop` — the unload path calls stop(), so it raised into an `except Exception: pass` and the server looked un-stopped with nothing saying why. * calling a FastAPI handler as a function leaves unpassed parameters as Form(...) sentinels: "unsupported operand type(s) for +: 'Form' and 'Form'". That test drives the real route now. * there is no GET /admin/api/settings any more, and the LongCat image builds its 3.10 venv in a python:3.10-slim stage rather than with a literal `python3.10 -m venv`. One real inconsistency surfaced on the way: the broker is advertised `v1/video/compose/{id}` where the app serves `{job_id}`. Shape-compatible, so nothing breaks; the advertised list is simply inaccurate. Not changed here, because what we advertise is a wire-visible contract. 13 remain: the broker handshake tests encode server-greets-first against a client that says "Client speaks first", plus streaming and studio shapes. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
THE SURYA BUG. `/v1/ocr/attach/<lease>/…` is an OCR engine attaching to THIS install's gateway. The lease is minted here and lives in this process's memory — and the remote gateway was FORWARDING those requests to the pod, which correctly refused a token it never issued: 404 {"error": {"message": "unknown OCR engine lease"}} surya's client swallows that and returns an empty generation, so every page came back unread with the GPU never asked to read one. Measured on a clean 2-page PDF that chandra (2212 chars) and olmOCR (1992) both read: surya returned 2 characters, and the pod's vLLM logged a single 6-token warm-up for the whole document. $88 of surya time in one day produced nothing. The gateway behind that prefix does its OWN forwarding, with its own pool and its own lease. That is the hop, and it has to start here. `/v1/ocr` itself stays routable, so pinning the attachment does not strand the subsystem. This also corrects an earlier diagnosis of mine: v0.3.87 treated this as a lease VOIDED BY A RESTART and shipped a re-resolve for it. No restart was involved — the lease was wrong from the first page, which is why surya has been broken continuously rather than only after restarts. That work was not wasted (the retry cap took four paid generations per page down to two, and the 502 plus `unread_pages` are what made this diagnosable at all) but it did not address the cause. Third instance today of one mistake: the gateway forwards anything under /v1/ that looks like a model request, including this install's own control plane. /v1/runpod rented a pod to deliver a teardown; /v1/loras was staying local by luck; now the OCR lease. All three are pinned. The structural answer — making control-plane routes opt IN to forwarding — is not in this commit. Also in here, from the fleet-hygiene pass: * accelerate added to every profile that carries transformers (audio, ocr, stt, tts, voice, tts-xtts.venv-TTS), with a test that fails if a new profile reintroduces the gap. Same latent fault that broke embeddings. * an embeddings pod is warmed on /v1/embeddings, not with a chat completion it refuses by design — that 404 was printed on every boot of a healthy pod and cost a wrong diagnosis. * the seed plan is announced once, not every fifteen seconds while a pod boots, so a boot log stays readable when it matters. * `pool:` is dropped from pod NAMES (ownership is the prefix, never the rest). * the reaper owns a pod by REGISTRY as well as by name, so renaming deployment_id no longer orphans pods that are already running and billing. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reported from production, minutes after v0.3.96 made an adapter a model of its own on the pod: 503 Model 'diogene-alpha' failed to load: Unknown backend: cpu Only the pool's OWN entry was seeded with its engine. Siblings got none, so the adapter arrived on the pod with no backend and fell through to the pod's default. Verified against the live catalogue: the seed genuinely had no `backend` key. This had always been wrong and could not bite before, because an adapter collapsed into its base on the pod and was never a model of its own to LOAD. Making it one — which was the right fix — is what exposed it. Every sibling now takes the POOL's engine, not its own. A pool is one pod running one image, so siblings cannot each have a different engine: seeding one would describe something the pod cannot satisfy, and the failure would land minutes later inside a rented machine. A test asserting the opposite was written and then corrected. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Five fixes, two of them reported from production in the same hour. EMPTY REPLIES from a chat naming `diogene-alpha`, while `qwen38` — the same weights on the same pod — answered normally. Parser selection reads the model NAME: if 'qwen' in model_lower: -> QwenParser `qwen38-awq` matches; `diogene-alpha` matches nothing and fell to the generic parser, so the model's <think>…</think> block was never separated from the answer. With a modest max_tokens the whole budget goes into reasoning that never closes, and the caller sees an empty reply from a 200 — the hardest kind of wrong to see. A LoRA is named for what it was trained to do; it is the same FAMILY as the weights it adapts, so a name matching nothing now resolves to the base's. Nothing is discarded: the thinking comes back as `reasoning` and `reasoning_content`, as it always did for the base. This also corrects a reading of my own: every temperature-0 comparison in this chain showed the adapter leaking its reasoning while the base answered cleanly, and I took that as evidence the adapter was not loading. It was the parser. /v1/chunk with X-Coderai-Placement: remote answered 502 "nothing configures 'embeddings' to run remotely" while a coderai-embeddings pod — which has the chunker in it — sat there serving bge-m3. A capability-only request names no model, so the per-model placement that moved everything else cannot place it. It now borrows the destination of a model OF THE SAME CAPABILITY that is already remote. Same capability only (an embeddings pod has no OCR deps), and only on an explicit `remote`: nothing should rent a GPU to run a regex split because nobody said otherwise. Registering an adapter is not serving it. vLLM takes its adapters at launch and the code that adds one to a live engine sits on the model-LOAD path, which does not run again for a model this instance already loaded. So a pod could be sent an adapter, register it, report it in /v1/loras and go on answering from the base. Registration now hands it to the running engine, swapping a stale one. A stale content id repairs itself. v0.3.94 fixed what new uploads are stored as; an entry already registered under a non-canonical id stayed broken with no symptom but base-model answers. Seeding and pushing resolve an id against the CONTENT first. And `it-legal` now sets min_chunk_chars: 300 (strict stays 0) — it is the OCR-aware template, so it meets short trailing sections more often, and a 56-character chunk is too small to carry meaning and long enough to match. A runt folds only into a chunk that can CARRY it, so a document of short sections stays several chunks instead of collapsing. Absorption may never cross conclusioni <-> dispositivo at any size: what a party asked for and what the court decided, and a recorded label is a weaker guarantee than refusing the merge. NOTE: this release MOVES CHUNK BOUNDARIES for anyone using `it-legal`. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/chunk already rode the embeddings capability, so it could be placed here, on a RunPod pod (the coderai-embeddings image) or on a cluster node. Verified end to end for the first time rather than inferred from the routing table: a capability image really serves the route, and the middleware really forwards it ("[remote-gateway] /v1/chunk -> … (capability 'embeddings')"). That testing found a real defect. A NAMED structure template was forwarded as a bare name whenever the name was a BUILT-IN, on the reasoning that every coderai has those. True of the name; not true of what it means. `it-legal` was widened to accept markdown headings in 0.3.87 and `it-legal-strict` did not exist before it — so a pod on an older image resolves the same name to different anchors, or 404s on one it has never heard of. A pod fleet is mixed-version by nature: one rented an hour ago runs whatever `:latest` meant then. Chunk boundaries that differ by which machine answered make a retrieval index unmeasurable, which is the one thing docs/chunking.md exists to prevent. Every named template is inlined now, built-ins included: a few hundred bytes on the request, and the boundaries become a property of THIS install's version. An unknown name is still forwarded untouched — resolution failing here must not swallow the request, and the remote's `unknown_template` is a clear error. Six tests pin the placement itself: local by default (a regex split must not rent a GPU), forwarded to a node, left here when only another capability is remote, the template inlined on the wire, and one request pinned local by header. Plus the note that a per-model remote does NOT move chunking, because a chunk request carries no model and only the capability map places it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The root cause under four releases of "the adapter is not applied". Measured on a live pod: catalogue says : sha256:b8eac366… (hash of the tar a client sent) this bundler's own : sha256:2e820c… (same content, its own packing) pod lookup b8eac366 : 404 pod upload returns : {"id":"sha256:2e820c…","existed":true} `save_lora_blob` stored an uploaded bundle under the hash of THE BYTES THAT ARRIVED. A tar of one adapter made by GNU tar, by this bundler, or by anything else hashes differently — mtimes, ownership, member order — so the id recorded in the catalogue was one nobody else could reproduce. Every later push re-bundled the unpacked directory, stored the canonical id, and the pod looked up the uploaded one and found nothing. It had the bytes the whole time, under another name, so it loaded the base model and answered under the adapter's name. Fluently, with nothing in the reply to say so. An uploaded bundle is now unpacked FIRST and stored under the id of this bundler's packing of it, which is what every side computes. /v1/loras/upload returns that id, so what a client registers is what a pod can resolve. This is also why it took four attempts: the engine flags, the seeding and the entry propagation were each genuinely broken and each fix was real, but underneath them the two sides were never going to agree on the id. _load_lora_runtime had two bare `return False` paths — no adapter spec, or an id the blob store cannot resolve — which is exactly why this hid for three releases while every other layer was examined. Both say what happened now, and the second says the thing that matters: the catalogue and the store disagree about this adapter's id. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Measured on a live pod. The pool gate from v0.3.91 worked — [lora] no adapters yet, but this engine serves a pool — starting with LoRA enabled so one can be added without a reload [vllm] launching: … --enable-lora --max-lora… — and the adapter was still dropped: pod seeded expecting : sha256:2e820c… (the local adapter directory) re-registered as : sha256:b8eac366… (an uploaded blob) orchestrator pushed : b8eac366… ✓ 159 MB The bytes arrived. The pod's CATALOGUE did not. It was seeded at boot with the first id, /v1/loras/register repointed the entry at the second, and nothing told the pod — so its engine looked for an id nobody holds, dropped the adapter, and served the base under its name. (The bundle hash is deterministic; the two ids differ because the adapter genuinely changed, not because hashing is unstable.) Two keys missing a version, the same shape as every other bug in this area: * _ensure_adapters sent the BYTES only. It sends bytes and entry now. * teach_model keyed "already taught" on (pod, model), so a re-register could never reach a pod it had spoken to once. Keyed on the adapter's source. /v1/loras listed only the TRAINING registry, so an adapter that arrived by upload+register was invisible there — a successful registration looked exactly like a failed one, which is how this was found. It lists both now, tagged `trained` or `registered`, with the adapter id, base and pool. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reported from production: 500 {"detail": "Generation error: the pod answered 500: Generation error: 400 Bad Request … \"auto\" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set"} Not a regression: `tools` has never worked on a vLLM-backed model here. The proxy forwards them correctly — the remote decides what it supports — and nothing ever launched vLLM able to take them. A request-time 400 from a server that started cleanly, for a capability the catalogue advertises, naming two flags the caller has no way to set. A tool call is TEXT the model emits in its own trained format, so the parser belongs to the model FAMILY rather than to the deployment, which is what makes a default possible at all. `tool_call_parser` defaults to "auto" and picks from the model name (qwen->hermes, llama-3->llama3_json, mistral, deepseek_v3, glm45, qwen3_coder…), longest match first so qwen3-coder is not swallowed by qwen. An explicit value wins, per install or per model in its `vllm` block — and that block now travels to a pod, so a pod serves a model the way this install does. "off" is honoured rather than overridden: a family guessed wrongly does not fail, it parses tool calls into nonsense, and silently producing wrong tool ARGUMENTS is worse than refusing the request. Both flags go together or neither: vLLM refuses to START with --enable-auto-tool-choice and no parser, and an engine that will not boot is worse than one that cannot do tool calls. An unrecognised family logs which setting to use, because otherwise the symptom lands on the client as a 400. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Re-registering an adapter updated the catalogue and the pod went on serving the OLD weights. Two caches, neither of which had a version in its key: * the orchestrator remembered "this pod already has this adapter" by NAME, so the new bytes were never sent; * and the engine was asked "do you serve this name?" — which a stale adapter answers yes to. It kept what it was launched with. So the update looked applied from every angle a client can see: the catalogue had the new blob, the name worked, the request succeeded. The only symptom was output identical to the previous version, which is exactly the symptom nobody can see from a 200. Both are keyed on the adapter's SOURCE now. When it changes, the bytes are pushed, the old adapter is unloaded from the running engine and the new one takes its name — no reload of the base model, nothing else on the pod disturbed. When the swap cannot be done in place the engine is restarted with the new adapter, and says which it is doing. An engine coderai holds no record for is never treated as stale: unloading an adapter on a guess is worse than serving the one that is there. Content-addressed versioning is what makes this safe to run on every retrain — bytes that have not changed are recognised as the same adapter and nothing reloads. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/loras/upload put the bytes somewhere; nothing made them answerable. The last step was editing models.json by hand, which is not a step a team retraining adapters can automate — so the answer to "how do we send you a QLoRA?" was "send it to us and we will register it", with a human in it. POST /v1/loras/register {name, base, adapter} closes that. It joins the BASE'S POOL by default, which is the entire economy of the thing: the base weights are already loaded on a rented card, an adapter is tens of megabytes of deltas, and vLLM applies them per request. It inherits the base's placement, card choice and budgets, and deliberately does NOT inherit min_pods or keep_warm — an adapter rides the base's pod, it does not ask for one of its own. Re-registering a name with a new blob is how a retrain ships. A name that already belongs to a real model is a 409 rather than a replacement, and an adapter whose bytes we do not hold is a 404 that names the upload endpoint: a pod sent a content id nobody holds serves the BASE instead, silently, which is the failure this whole area has been about. Identity is (weights, name) here too. An adapter and its base share a `path` by definition, so the lookup is by id-or-alias and never by path. /v1/loras is also pinned local in the gateway. An upload must land in THIS blob store — it is where a pod is sent its adapters from, and the content id this install hands out has to resolve here. It carries no `model` field, so it happened to stay local by accident; that is luck, not design, and the same shape of accident as /v1/runpod, which did not stay local and rented a pod to deliver a teardown. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
v0.3.88 made the pod FIND the adapter. The live pod then said: [lora] 'diogene-alpha': adapter sha256:2e820c… is not in this machine's blob store — serving without it [vllm] launching: … --served-model-name cyankiwi/Qwen3.8-27B-AWQ-INT4 qwen38-awq Found, and dropped. A seed names an adapter by CONTENT ID — computed here so the pod can be told about it before it exists — and the bytes travel separately. They were pushed on the first `acquire`, but the pool's WARM-UP request loads the model seconds after boot, long before any acquire. So vLLM started without the adapter, and `--enable-lora` cannot be turned on afterwards. Two changes, both gated on the model declaring a `pool`, because a pool is the explicit statement that this engine serves several names over one copy of the weights: * a pool engine launches with `--enable-lora` and the capacity flags even with no adapter resolvable yet. The flags that must be decided at LAUNCH are the capacity ones — the machinery, the slot size, how many adapters may be live in one batch — not the list. Opening the door costs a buffer allocation; keeping it shut costs a full reload of a 27B on a card billed by the hour, the first time an adapter arrives. * an adapter is then handed to the RUNNING engine (`/v1/load_lora_adapter`, vLLM >= 0.9 behind VLLM_ALLOW_RUNTIME_LORA_UPDATING, which the launch sets whenever it enabled LoRA) instead of restarting it. The restart stays as the fallback and is still reached on every path that cannot use the API — an engine started without `--enable-lora`, an older vLLM, an adapter whose rank exceeds the slot size. A pod rented an hour ago may be running an older image than the one we would launch today, so "it should be supported" is not something this can assume. What this buys: one rented card, one copy of the base model, and QLoRAs added or swapped per request by name with no reload. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
v0.3.86 made the pod register `diogene-alpha` as a model of its own, and it still answered with the base model's weights. The pod log is the whole story: [seed] registered qwen38-awq in text_models [seed] registered diogene-alpha in text_models [vllm] launching: … --model cyankiwi/Qwen3.8-27B-AWQ-INT4 --served-model-name cyankiwi/Qwen3.8-27B-AWQ-INT4 No --enable-lora. Both models were registered; the adapter was never collected. `_sibling_lora_specs` skips "myself" by comparing `entry["id"]` — and the only kind of entry a POD holds is a SEEDED one, which carries `alias` and no `id`. So the comparison was "" == "", every sibling matched, and all of them were excluded. Keyed on alias-or-id now, with identity by object for the one entry that really is the model itself. That fix alone is not enough: a seed names an adapter by CONTENT ID, computed here so the pod can be told about it before it exists, and the pod still has to receive the BYTES. Until it does, vLLM launches without the adapter. So the backend now pushes a model's local adapters to the pod before the first request rather than after a failure — vLLM registers adapters when it STARTS, and bytes that arrive late cost a full reload of the base model to take effect. Where an adapter still cannot be resolved, the pod does not serve that NAME at all: it is in neither --lora-modules nor --served-model-name, so the request 404s and the teach-and-retry path picks it up. A 404 is recoverable. The base model answering fluently under an adapter's name is not — nothing downstream can detect it, which is how this survived two releases that each looked fixed. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Seven items from a production field report. Three of them were one cause. /v1/runpod is NOT in the remote gateway's local list. It carries a `model` field like an inference request, so the gateway read it as one, resolved the model to its pod, and FORWARDED the control-plane call there. Asking for six pods back rented a seventh to ask on; the pod has no runpod configuration of its own, so it answered 404 {"detail": "'datalab-to/surya-ocr-2' has no runpod block"} which reads exactly like the orchestrator refusing a name it should know. The same call blocked for minutes while a pod booted to receive it. An operator could not turn off $3.54/hr from the API. Reproduced here while testing the fix: two teardown requests rented two A40s, which is the clearest possible statement of the bug. surya answered 200 with empty pages for a whole document. Three failures in a row, each of which had to be fixed: * its client catches the HTTP error itself and returns a generation with no tokens, so the lease check in recognize_image — which only ran in `except` — never ran, and the pod address was never re-resolved; * the repetition detector was handed that empty string, found no progress in it and re-ran the page at a rising temperature, four full paid generations per page, `tokens: [0,0,0,0]`; * and the page came back as read. We had already measured the ink and said there IS something on it; the judgement never reached the response, so a client treating an empty read as an unreadable scan consumed roughly a thousand documents from its queue and wrote them off. A page with ink and no text is now `unread`: it goes to the fallback engine like any other degraded page, it is listed in `unread_pages`, and a document whose every inked page came back empty is no longer a 200 (`ocr.unread_status`, 502 by default; a partially read document stays 200 with the list). `bge-m3` and `BAAI/bge-m3` each rented their own A40 — the pool key was whichever name the request carried. Keyed on the model ENTRY now, so every name for one set of weights lands on one pool. A client read timeout cannot bound an OCR call: it times each READ, so a response that dribbles never trips it and one stayed open nineteen minutes against timeout=420. `ocr.request_timeout_s` is the server-side stopwatch. Chunking: `it-legal` accepts a decorated heading (`**DISPOSITIVO**`) where the implementation it ports does not, which splits one more section and leaves a short trailing chunk. That was true and undocumented — the table printed the strict pattern. Documented now, with `it-legal-strict` named as the byte-identical one for re-chunking an existing index. `min_chunk_chars` folds a runt chunk into its predecessor, off by default because merging moves a boundary, and never silently: the absorbed section label is reported, since `conclusioni` is what a party asked for and `dispositivo` is what the court decided. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A shared pool is one pod serving several names over one copy of the weights — a base model and its adapters. Live, every request naming the adapter got 404 Client Error: Not Found for url: https://…/v1/chat/completions and kept getting it: the pool keeps the pod warm, so nothing replaced it. The pod, asked directly, listed only the base and reported {"loras": []}. Which of the two names worked depended on which model happened to rent the pod, so it looked intermittent rather than wrong. Five places decided "same model?" by comparing `path`, and a LoRA's path IS its base's path: * the orchestrator seeded the other pool members with no served name, so they arrived with no alias and were registered under their path; * the pod's boot seeder skipped the second entry as already known; * register_model_runtime did the same, which made teaching a running pod a no-op that answered "already known"; * the engine-side seed filter in main.py did it a third time; * and vLLM's service key included the served name, so had the adapter got through it would have started a SECOND copy of a 27B on the same card. Identity is now (weights, name) in all five, and one engine serves one set of weights under every name configured for it: the base under its own, each adapter through --lora-modules, which --served-model-name must not repeat. Adoption checks the image, the GPU class and the bearer token — none of which say anything about the catalogue. So a pod that cannot serve what the pool promises is now TAUGHT: the adapter bytes are pushed, the entry is registered, and the same request is retried on the same pod. Renting a second card to serve a name the running one can hold is the answer this avoids. Once per pod per model; a pod that still refuses is a real failure. An engine is also relaunched when it does not serve the name asked of it: vLLM registers adapters at LAUNCH, and a pod is taught after it boots. Only an ANSWERED probe that omits the name forces that — restarting a loaded 27B because one probe timed out would be worse than the doubt. Finally, raise_for_status() discarded the pod's body, which is the whole message: it names the models the pod actually has. The caller saw only the status line, which reads like a routing fault. The pod's own words now travel. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
0.3.84 made `_is_bge_m3` see `sparse_linear.pt` on a machine that had never downloaded the repo, so the multi-vector backend was finally selected on a pod. It then loaded the heads with the same cache-only lookup, and `from_pretrained` does not fetch them: it pulls the config and the weights the config NAMES, and bge-m3's sparse/colbert heads are plain tensors sitting beside the model. So they were still absent, and the request came back HTTP 400 "This model has no sparse head." which reads as a property of the MODEL rather than of the disk. `_fetch_repo_file` is the companion to `_resolve_repo_file` for files that have to be READ rather than merely detected: cache first, then a real hf_hub_download. Offline short-circuits, a local directory is the whole answer, and a failure returns None -- the dense path, which the caller already handles -- because a head that cannot be fetched must not turn into a failed load. Both halves of this were the same mistake made twice: a cache-only helper answering a question about existence on a machine whose cache is empty by design. A rented pod is that machine every single time. Needs a coderai-embeddings rebuild: this code runs on the pod. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`_is_bge_m3` keys off `sparse_linear.pt` -- the definitive signal for a multi-vector embedder, since plain dense models do not ship one. It asked `_resolve_repo_file`, which answers from the local disk and NEVER the network. Right for "give me the path", wrong for "does this exist", and the difference is invisible until the model has never been downloaded. On a rented pod it has never been downloaded. The pod is disposable and its cache starts empty, so the check ran before anything could satisfy it: on every pod, every time, bge-m3 loaded through the DENSE sentence-transformers path. The request for sparse was then refused with a 400 telling the caller to serve the model on the coderai-embeddings image -- which is the image it was already running on. Measured end to end on a real A40: 379 seconds, then that. `_repo_has_file` keeps the local answer first and, only when the file is not on disk, asks the Hub for the repo's file LIST: metadata, no download, cached once per process. Offline installs short-circuit on HF_HUB_OFFLINE, a local directory is taken as the whole answer (asking the Hub about a filesystem path would look up a stranger's model of the same name), and any failure answers False exactly as before -- a metadata lookup must never be able to fail a load. Needs a coderai-embeddings rebuild: this code runs on the pod. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
In models.json a model's kind is the dict KEY it sits under. Only the ENTRY travels to a pod, and the pod's seeder reads section = e.get("model_type") or "text_models" A models.json entry carries no `model_type` -- so every capability pod registered every seeded model as a TEXT model, whatever it was. Found on a real A40 immediately after the gateway fix finally let a request reach a pod. It booted in 303s, served /v1/models, listed BAAI/bge-m3, and answered: Model 'BAAI/bge-m3' is a text model and cannot be used for embedding generation. Use the text endpoint instead. Which is the expensive part: nothing looks broken. The routing is right, the image is right, the catalogue is right, and the error blames the caller for calling the wrong endpoint. Both seed builders now carry the section -- `capability_seed_entries` already had it and dropped it at `out.append(entry)`, and `seed_model_env` gets it from the new `entry_model_type()` (by identity, then by id, "" on any failure). Stamped on a COPY, so nothing writes a field into models.json that this install never had, and an entry that already declares its own `model_type` is not second-guessed. Text pods were unaffected, which is why this survived: a vLLM pod serves text, and OCR engines are picked by name rather than registered at all. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The pod booted healthy, answered /v1/models, advertised bge-m3 -- and failed every embedding request with Model 'BAAI/bge-m3' failed to load: Using a `device_map`, `tp_plan`, `torch.device` context manager or setting `torch.set_default_device(device)` requires `accelerate` sentence-transformers loads under a device_map context and transformers refuses that without accelerate. It is in requirements.txt and in the text, images, video and stt-crisper profiles; it was never in this one. What makes it worth a comment in the file rather than just a line: the failure does not look like a missing dependency from outside. The image is healthy, the catalogue is right, and the error names the MODEL. On a real A40 that cost 303 seconds of boot before anyone could find out, and the next error after it ("'BAAI/bge-m3' is a text model") sends the reader somewhere else entirely. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
THE /v1/embeddings bug, root-caused on a live 0.3.80 instead of guessed at. `RemoteGatewayMiddleware` short-circuits on `any_remote_configured()` BEFORE it calls `resolve_target`. That gate knew about capability endpoints, capability pods and `service_url` -- and nothing about a per-model `backend`. So an install whose remote models are all per-model RunPod pods answered "nothing is remote", every /v1 request was handed straight to the local app, and a GPU-less orchestrator loaded the model itself. Measured on production: POST /v1/embeddings for bge-m3 spent 312 seconds downloading from HuggingFace and returned 500, while /v1/models/test/state?model=bge-m3 answered `"placement":"pod"` in the same process. Both were right. `model_placement` is asked directly by every page that reports where a model runs; the gate is asked by the code that actually routes, and the two disagreed in silence. Two earlier fixes aimed at the same report and missed it: v0.3.77 made `model_capability()` infer "embeddings" for an entry in the embedding section, and the config was given an explicit image and capability. Both were necessary and neither was reached. OCR is why this survived a release: the OCR manager calls `placement.choose` itself rather than going through this middleware, so the pod-side pipeline worked and made the gateway look fine. The gate now makes the same two tests `model_placement` makes -- a `backend` WITH its block, for runpod and for host. A leftover block whose backend was flipped off still does not count, which is how a model's pod is meant to be disabled. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three reports from the same run, all of them about giving pods back. SCALE BLOCKED THE WHOLE SERVER. The endpoint called into the pool straight from the coroutine, so while it walked dozens of (card, region) candidates -- a network round trip each -- nothing else was served. /v1/runpod/status stopped answering during exactly the operation someone would poll it about, which makes a slow rental look identical to an outage. It now runs off the event loop and the reply is bounded: after `wait_s` (default 20) it answers `pending` with a request id, and the SCALING CONTINUES -- shielded, because cancelling it would leave a half-made lease and possibly a pod. GET /v1/runpod/scale/{id} has the result for a caller that gave up. A BOOT COULD NOT BE CANCELLED. `_booting` is an integer, not pod objects -- a pod being created does not exist here yet -- so `shed_now` could not see one, count one or stop one. A client that asked for a pod, got four on the way and released them was told `kept:0, needed:1, floor:1` and paid every boot out in full. A shed now bumps a cancel epoch; an ANTICIPATORY boot (warm floor, background growth) checks it during the model load and again just before it joins the fleet, and hands its pod straight back. A boot a request is blocked on is never cancellable, whatever arrives meanwhile: the request is what the pod is for. Booting pods also count as capacity in the surplus arithmetic, and the shed reports how many it stopped. A POD COULD NOT BE RELEASED unless some pool floor could be lowered to lose it. `'pool:digesta-qwen38' has no runpod block` came from code whose own comment says the caller cannot look that key up -- so a live pool is now addressable by the key /v1/runpod/status reports. And nothing anywhere could terminate ONE pod by id: not an adopted pod, not one left by a crash, not one whose pool was reconfigured underneath it. POST /v1/runpod/release does, admin-scoped (leasing is bounded and expires; terminating a named pod is neither), attributing the spend before it drops the pod so the money does not land nowhere. And the over-provisioning: `ensure_ready` incremented `_booting` by ONE and then started `want` boots, so with min_pods 3 three cards were on the way while the 15s scaler, `_should_grow` and the parallel-boot cap all believed there was one -- each topping up against a deficit already being filled. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
From last night's run: ~97% of a 1,000-document batch came out truncated, with no error anywhere. The engine answered 200 every time. Its decode had fallen into a loop, run to the token cap emitting the same fragment, and the resumer correctly refused to continue a reply that was already repeating itself -- so the page was flagged, returned, and indexed. `fallback_engines` never fired, because it only fires on an exception. The same pages read cleanly on another engine (38,055 characters against a truncated stub), so the recovery was there the whole time and nothing reached for it. `ocr.retry_degraded` (default "auto") re-reads ONLY the degraded pages on another engine and keeps the replacement only if it is not degraded in turn. Bounded by construction: a clean document costs nothing, a ruined one costs at most a second pass over its ruined pages, and `retry_degraded_max_pages` caps even that. The page keeps its own index and carries `degraded_retry` saying who re-read it and why; the response reports `recovered_pages`. Degraded means looping or truncated -- the two the engine itself attests. NOT a short page, a blank page or an unreadable one: those have their own meaning and re-reading them spends a card to be told the same thing. Which leaves the other half of the report: olmOCR returned 4,046 characters where chandra read 30,983 on the same page, ending cleanly well short of the cap, flagging nothing. The only evidence an early stop leaves is the company it keeps, so `retry_short_pages_below` compares a page with the DOCUMENT'S OWN median rather than a character count -- a threshold in characters would need re-tuning for every language, DPI and layout. Default 0, OFF, and deliberately: this signal is inferred where the others are attested, and a genuine title or signature page looks identical. A short page is also only replaced by a LONGER one. Off on pods: only the orchestrator can reach another engine's pods, so a pod would re-read on the engine that just produced the flag. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ran both side by side on the same documents instead of reading the spec, and the spec's prose turned out to allow several readings the port took differently. Seven divergences in reference extraction, one in sectioning: * `artt. 163, 164 e 167 c.p.c.` extracted nothing -- the code had to sit immediately after the number, so an enumeration lost the whole citation. It now resolves to its first article, which is what the indexes hold. * `art. 2055-bis c.c.` extracted nothing (only a space was tolerated before the suffix); the hyphen is now kept in the key, a space still squeezed out. * `art. 2043 cod. civ.` extracted nothing -- the code spelled out was in the normalisation table but not in the pattern. * `articolo 2043 c.c.` and `art. 2043 della c.c.` extracted a reference the original does not. Looking like an improvement does not make it one: it puts keys in the field that the corpus it will be matched against lacks. * `legge 241 del 1990` extracted nothing; `l 241/1990` without the dot extracted one. Both inverted. * `art. 360` with no code came back as `art.360`, and `d.p.r. 115/02` as nothing -- the two already reported. And the off-by-one, which was none of the things it looked like. Section bodies were not stripped, so merging two adjacent same-label sections (a filing opening its conclusions twice) joined a body ending in a blank line to the next, turning a line break into a paragraph break -- packed as `\n\n` rather than riding inside one piece. Exactly one character, on exactly the documents that merge. Empty sections were also kept, spending a `parent` id and shifting every id after it. Verified by diffing chunk-for-chunk over 63 generated documents covering CRLF, repeated headers, enumerations, two-digit years and degenerate input: zero divergences in text, section, parent and norm_refs. The one deliberate difference is `**FATTO**`, which the original does not match and `it-legal` does. `it-legal-strict` is added for re-chunking a corpus indexed with the original, where a boundary that moves makes the new index incomparable with the old. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two divergences from the spec, both reported from a real corpus: * `art. 360` with no code came back as `art.360`. Every procedural code has an art. 360, so an unqualified number is not a citation -- it files three different articles under one exact-match key, which is the one thing this field exists to prevent. The code suffix is now required. * `d.p.r. 115/02` extracted as nothing, because the year was pinned to four digits. Two to four now, kept verbatim: expanding `02` to `2002` would split one act across two keys depending on the source's style. The spec's prose allowed both readings; it now says which. Co-Authored-By:
-
- 14 Oct, 2026 12 commits
-
-
Stefy Lanza (nextime / spora ) authored
Switching a model's `runpod.engine` from `coderai-vllm` to `coderai` -- the correct setting for a capability pod -- silently removed its capability. `_CODERAI_SERVER_ENGINES["coderai-vllm"]` had been supplying the classification; plain `coderai` supplies none, and `model_capability()` returned "". Both halves of the failure follow from that: * the front could not classify the request, so the model was loaded LOCALLY on a GPU-less orchestrator; * default_capability_image("") is "", so a blank image field raises "set `image` on the runpod block" at provisioning time. It presented as "tries to load on the wrong machine", which is a long way from "this entry has no capability". The cause: model_capability() receives a BARE DICT, and _model_entry_for() iterates md.values() -- it matches inside `embedding_models` and discards the section key. Nothing had ever told it that an entry in `embedding_models` is an embeddings model. Now it asks: by object identity first (the dict the loader holds), falling back to the id so a copied or rebuilt entry still resolves, and returning False on any failure because a capability GUESS is never worth an exception in a placement path. An explicit `capability` field still wins, which is what that field is for. Only `embedding_models` is inferred, deliberately. The other sections work today, and remapping them all to fix one would change placement for every model kind -- a blast radius this does not need. A test asserts the inference does not reach into text_models, image_models, video_models or tts_models. Found by a user report ("broken -- front loads bge-m3 locally on the GPU-less host; not in /v1/models") after I made exactly that config change. I had already noted this gap when the capability came back empty during the switch, said it was worth raising rather than working around, and then worked around it in the config anyway -- which is how it reached production. 11 tests. Suite at the 25 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
RETRIEVAL CHUNKING. A new endpoint that splits a document into the pieces an index stores, with the document's structure decided per request by a named or inline template. NOT embedding windowing: `embed_long_input` makes ONE input fit a model window and pools it into ONE vector; this produces MANY chunks, each indexable, each with its section, parent and references. It never pools and never drops text. Pure CPU -- token counts are characters / chars_per_token, so no tokenizer is loaded and no model, GPU or pod is involved. Every image serves it, light-core ones included, and a test asserts no torch/transformers import creeps in. The packing loop is reproduced verbatim, including the parts that look improvable: paragraphs joined with "\n\n" (hence the `+ 2`), and an overlap that is a RAW CHARACTER TAIL which may begin mid-word. Changing any of it moves every existing chunk boundary, and then a re-chunked index cannot be compared with the one it replaces. `conclusioni` is its own label and is never folded into `dispositivo`: in a ruling the dispositivo is what the judge decided, in a filing the conclusioni are what a party ASKED FOR, and merging them would let a demand surface as a holding. TWO DEVIATIONS FROM THE SPEC, both recorded in the source where someone comparing against the original will look: * The ceiling is target + overlap + 2 (3682), not the 3680 the prose states -- which agrees with the spec's own "observed max is 3682" two lines later. Asserting 3680 would fail a correct port. * The verbatim patterns cannot match `**FATTO**`, yet the fidelity list asks for that case. The line-anchored alternatives are widened at BOTH ends (`^[\s*#>_]*fatto[\s*#>_]*$`), which is how the patterns that already coped spell it. Purely additive: every line that matched still matches, so no existing match moves -- but a document whose headers were emphasised used to fall back to flat chunking and now gets sectioned, so ITS chunks change and it needs re-chunking. FORWARDABLE, with the hazard that comes with it. /v1/chunk rides the EMBEDDINGS capability: chunks are on their way to be embedded, that image is full-core so it carries the chunker, and a capability of its own would be one nothing advertises -- or would rent a GPU to run a regex split. It was first classified as deliberately local on the grounds that it is cheap; that was the wrong lens, since the orchestrator is the box that was OOM-killed today, so WHERE the CPU is matters more than how much there is. But a template registered HERE does not exist on the remote: forwarded by name it returns 404, or worse matches a DIFFERENT template with the same name and returns different boundaries silently. So a named non-builtin template is resolved and INLINED before the request leaves. The hook never raises -- a body it cannot parse is forwarded untouched -- because a rewrite that can throw would take down every forwarded request, not just a chunk one. THE OCR PIPELINE NOW COVERS ALL THREE VLM ENGINES, not just surya. Leaving chandra and olmOCR on the orchestrator was an asymmetry rather than a design: surya needed a 1.6 GB client venv baked in, while those two speak only HTTP in served mode (PIL + requests, already in the main venv), so for them the pod side costs nothing beyond the rasteriser. Two bugs fixed on the way, both of which would have produced plausible WRONG text rather than an error: the dispatch hardcoded `engine: "surya"` in the form it posted, and the pod env was surya-shaped throughout -- a chandra pod was told to run surya against chandra's model id. Per-engine tuning and per-engine licence acceptance now travel. AND THE LOG THAT CAUSED A MISDIAGNOSIS. One pod request walks a candidate list until a (card, region) pair has capacity, and each attempt printed "provisioning pod for ...". 312 attempts for 10 pods therefore read as runaway provisioning and was reported as such. It now says "trying candidate N/M for ONE pod", so the shape of the list is visible instead of inferred. What these tests CANNOT do is prove byte-identity with the implementation this ports -- that needs fixtures from a live corpus. They prove the documented behaviour on each path; identity is what decides whether a re-chunk is safe and it is still outstanding. Suite at the 25 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
`_read_on_pods` returned None in silence when no pod could serve, and the caller then read the document locally. So an install configured for the pod-side pipeline could fall back on EVERY request and look exactly like a success: the OCR works, the text is right, and the operator concludes the new path is working when it has never run once. A test that cannot distinguish the two paths proves nothing about either. Both paths now announce themselves in one line -- the fallback naming the CAUSE ("placement answered 'local' — nothing rented, or the pool could not serve") rather than only the effect, and the success saying the pages were rasterised and read on the pods rather than here. Found while answering "can I launch the new OCR to test?", which is exactly when it mattered: the answer was yes, and the test would have been uninterpretable. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two gaps in 0.3.72/73, both of which left chunking looking enabled while doing nothing useful. THE THRESHOLD WAS A NUMBER SOMEONE TYPED. max_len = int(_emb_cfg.get('n_ctx') or _raw.get('n_ctx') or 8192) Set n_ctx above what the model accepts and the windows are built too big; the tokenizer then truncates each window at its real limit anyway and chunking silently stops protecting anything -- the same invisible loss this code exists to remove, reached through a config typo instead of a missing feature. The window is now min(configured, tokenizer.model_max_length), with one trap handled: HuggingFace sets model_max_length to a sentinel near 1e30 when the checkpoint declares none, and believing that means never chunking at all, so a value that large is read as "unknown" and the caller's number stands. Same clamp on the sentence-transformers path, taking the smaller of max_seq_length and the tokenizer's own limit. THE MULTI-VECTOR PATH COLLECTED THE COUNTS AND DROPPED THEM. It builds its own response, so the dense path's report did not cover it -- chunking was invisible on exactly the path that needs it most, which is the bug this feature was written to fix. It now reports "chunked": {"windows": [...], "pool": ..., "token_limit": N} and logs the limit with the merges applied. `token_limit` is deliberate: it is what makes a WRONG n_ctx visible rather than merely ineffective. 46 tests. Suite at the 25 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
bge-m3's sparse and colbert come from linear heads (sparse_linear.pt / colbert_linear.pt) applied to the last hidden state -- a transformers forward pass, not an embedding server's output. Two consequences that were both silent: A DENSE-ONLY BACKEND NOW SAYS SO. `embedding_types` was simply ignored outside the native path: a caller asking for sparse got a normal-looking response with only `embedding` in it, its hybrid retrieval then ran on half the signal, and the result was merely a worse score -- which reads as a model problem, not a configuration one. Now a 400 that names the cause, where it DOES work (coderai-embeddings), and why the vLLM image cannot: that image is LIGHT core with no torch in its main venv BY DESIGN, so _load_bge_m3 could not even import. Dense-only requests are untouched. MULTI-VECTOR LONG INPUTS ARE WINDOWED. 8192 tokens is ~6000 words, so this bites on whole documents rather than passages -- and when it bit, the tokenizer's `truncation=True` kept the opening and said nothing. Each type is merged the way the encoder already merges WITHIN a window, which is what makes these choices defensible rather than invented: dense mean, RE-NORMALIZED. It is an L2-normalized CLS vector and a mean of unit vectors is not one; cosine similarity on a non-unit vector is quietly wrong. sparse MAX per token id -- the encoder max-pools per token id already (scatter_reduce amax), and a term's weight should be its strongest occurrence, not an average diluted by the pages that never mention it. A test guards against someone later tidying max into mean. colbert CONCATENATED: max-sim scores each query token against ALL document token vectors, so every content token of every window belongs. `embed_long_input=truncate` keeps the old behaviour explicitly, and `first` keeps only the opening window, for an operator who would rather have that than a merge. Also removes three customer references I had put in shipped profile comments, two of which went out in 0.3.69. A capability must be scoped and justified by what it does, never by one install's needs, and a customer name does not belong in a released image. The measurements keep their force without the attribution: "hybrid retrieval wants dense and sparse from one pass", "a production fleet", "a GPU-less orchestrator". 39 tests. Suite at the 25 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Only the GGUF path ever chunked. Every other backend passed `truncation=True` and returned a confident UNIT vector of roughly the first page of a long document, with nothing in the response to say so: truncation=True, max_length=1800 # GME-Qwen2-VL, a hardcoded ceiling truncation=True, max_length=max_length # bge-m3 native model.encode(texts, ...) # sentence-transformers, its own cap Not an error -- a wrong answer that looks right, which is the failure mode this codebase keeps paying for. A caller had no way to tell an embedding of a document from an embedding of its first page. Now a long input is windowed to the model's real limit, every window of every input goes to the backend in ONE batch (a 40-page document must not become 40 round trips), and the chunk vectors are mean-pooled AND RE-NORMALIZED: mean pooling unit vectors does not give a unit vector, and cosine similarity on a non-unit vector is quietly wrong, so the renormalization is part of the contract rather than a tidy-up. Tunable per-model -> env -> default, the convention the admission gate already uses: embed_long_input chunk | truncate | error (default chunk) embed_chunk_pool mean | first (default mean) `truncate` stays reachable and is now DELIBERATE: for a short-text encoder a 40-page input is usually a caller bug, and an operator may prefer the old behaviour to a vector pooled from forty windows. `error` says so out loud. AND IT IS REPORTED: resp.chunked = {"windows": [...], "pool": ...} whenever any input needed more than one. Replacing a silent truncation with a silent pooling would have been the same bug with a better answer. The GGUF path's hard-won guard now covers every backend: each window is RE-ENCODED and shrunk until it verifiably fits, because a detokenize->retokenize round trip is not token-count-stable (specials and BOS are re-added) and a window cut at exactly the limit can re-inflate past it -- which is how that path first tripped llama.cpp's GGML_ASSERT. Wired: sentence-transformers, and GME-Qwen2-VL (windowed BEFORE the chat prompt is built around the text, since the processor's own truncation would cut the template's closing tokens as readily as the body). GGUF already chunked. NOT wired, on purpose: the CLIP family (clip, geoclip, vpr) takes 77 tokens BY DESIGN -- those encoders embed captions, not documents, and pooling forty windows of a contract into a CLIP vector would be meaningless. A test asserts this so "chunk everywhere" is not later read as "chunk where it makes no sense". NOT wired, needs a decision: bge-m3's native multi-vector path returns dense PLUS sparse and colbert. Dense mean-pools fine, but sparse would need max-merging per token id across windows and colbert would concatenate per-token vectors. Those are retrieval semantics, not a mechanical change, and improvising them would degrade search in a way that looks like a model problem. 26 tests. Suite at the 25 pre-existing failures. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The timer's own comment promised "a restart only happens on a real upgrade, and only when the fleet is idle (or after MAX_DEFERRALS hours of never being idle)". The second half could not happen: guard_idle || exit 0 # a busy host exits HERE ...upgrade... N=$(cat "$DEFERRALS" ...) # the counter only ever incremented HERE So on a busy host the script left before the image was even pulled, the counter the force depends on never advanced, and MAX_DEFERRALS was unreachable. Measured on the Digesta orchestrator 2026-10-14: 24 skips in one day, `deferrals` still 0, two versions behind, and 0.3.68 landed only because the host happened to be idle at 17:01. Now a busy skip COUNTS, and at MAX_DEFERRALS the script books a forced upgrade for the next midnight. The 00:00 tick of the existing hourly timer fires it, so there is no second unit to install and nothing stranded by a reboot. Midnight because it is the cheapest hour to interrupt a business-hours workload and a time an operator can predict without reading the script. Four things that each had a way to go wrong: * booked ONCE, not re-booked on every busy tick -- otherwise the deadline walks forward forever and never arrives; * a corrupt deadline file is discarded, not obeyed: garbage must not read as "the time has passed" and restart the service, nor sit there blocking a fresh booking; * a forced run does not defer at the restart step either -- deferring there would put it back where the booking was made; * MAX_DEFERRALS=0 keeps its old meaning (wait for an idle moment, however long, never force), so an install that did not opt in is unchanged. The booking is announced in the config banner of every later run and cancelled by deleting one file, because a forced restart that nobody saw coming is worse than a late upgrade. 16 tests, and the ones that matter are behavioural rather than source-matching: they extract the real arm_midnight_force and force_due from the script, run them under bash against a temp state dir, and assert the timestamp really is local midnight and that force_due flips at the right moment. A test that restates the logic only proves it agrees with itself. Suite at the 25 pre-existing failures. NOT installed on Aruba: that copy is byte-identical to this one and runs hourly, so overwriting it mid-run would be editing a running script. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An OCR VLM needs a model server AND the surya pipeline that drives it. Which machine runs the second one is now chosen by the pod IMAGE, and by nothing else: coderai-vllm the pod serves the model; the ORCHESTRATOR rasterises every page and sends a ~1.06 MB base64 PNG per A4 page at 250 DPI -- and again for every resumption round. Unchanged. coderai-vllm-ocr the pod carries surya's client venv and pypdfium2, so it is sent a ONE-PAGE PDF (~474 KB, 1.7 ms to split here against ~105 ms to rasterise) and reads it on the card with its own vLLM over loopback. A separate `pod_ocr_pipeline` flag was built first and removed. Two switches that can disagree is the invisible mismatch that cost three rounds of blaming a pod image for orchestrator bugs; choosing the image IS choosing the mode, it already lives in models.json and on the model page, and an image that cannot run a pipeline can no longer be asked to. PAGE-LEVEL FAN-OUT IS UNCHANGED. One page is one request in both modes -- `placement.choose` picks an upstream per page -- so every pod in the pool shares the work and none owns a document. Sending a whole PDF to one pod would turn an 8-way parallel read into a serial one, which is why the split is per page. THE TUNING TRAVELS, not just the wiring. surya_dpi, the page budget, the continuation count, the repeat-detector thresholds, langs, the page window, the resident cap. Every one was set in response to a measured failure, and a pod reading pages at surya's own defaults would be indistinguishable from the "pre-fix pod image" that was wrongly blamed three times. A test cross-checks every CODERAI_OCR_* the plan sends against what config.py actually reads: a variable sent and never read is a setting that silently does not apply, which is the exact shape of two bugs found earlier today. FALLBACK, and what it refuses to hide: * no pod rented -> the WHOLE document is read locally, not half each way (a document read by two pipelines is the hardest kind of result to explain); * a pod answers 500 -> raised, not quietly re-read here; a broken fleet must stay visible; * the image lacks the pipeline -> the orchestrator does what it always did. The image, built and verified (14306 MB, /healthz OK): venvs/surya 1.6 GB torch 2.14.1+cpu cuda=None + ultralytics 8.4.177 venvs/vllm 11 GB main venv: pypdfium2, and no torch at all (light core intact) A CUDA torch in the surya venv would add ~4.2 GB AND put surya's small local line detector on the card serving the model; the venv check asserts torch.version.cuda is None so a resolve cannot quietly undo it. 35 tests. Two flaws of my own that they caught: four passed alone and failed in the suite because they patched sys.modules while `from codai.models import placement` resolves the name as an attribute of the already-imported package -- now patched on the real module, verified in both orders; and _plan_env used a hand-rolled stub that tested the stub, now built from a real RunpodModelConfig. Suite at the 25 pre-existing failures. NOT YET EXERCISED AGAINST A REAL POD: the orchestrator path has only run against fakes, and the image is local -- not pushed or signed. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
THE WARM FLOOR. A declared floor of 8 took ~40 minutes to exist. Measured on the Digesta orchestrator, one pod per scaler pass at ~5 minutes each: 15:15:27 warm floor 8, 2 pod(s) healthy — renting one 15:21:12 warm floor 8, 3 pod(s) healthy — renting one 15:32:41 warm floor 8, 4 pod(s) healthy — renting one Two causes. First, `maintain()` rented exactly one pod per pass, gated on `not self._booting`. A floor is a DECLARATION, not a prediction -- min_pods, keep_warm or a client lease already said how many should be warm -- so the deficit is filled at once, bounded by the model's existing `scale_up_max_parallel_boots` (no new knob; it already means "how many pods may be booting at once"). Second, `_provision_one` is "Blocking (minutes)" and this called it INLINE from the scaler loop, so one warm boot stalled that loop -- including the registry heartbeat that lives in it and runs every 60s. A stalled heartbeat is exactly what makes a sibling treat our booting pod as abandoned, which 0.3.68 made load-bearing. Warm boots now run on their own threads. Adoption loops too (a restart orphans the whole fleet; eleven pods at one per tick took six minutes), but BOUNDED two ways: `_adopt_shared_pod()` returning True is a report, not a guarantee -- it can succeed without the pool gaining a healthy pod -- and the first `while ... and adopt()` form span forever the first time that happened, taking the scaler thread and the heartbeat with it. At most `deficit` attempts, and stop when an adoption does not raise the count. A pass that adopted rents nothing: renting beside a fresh adoption is how one job pays for two cards. THE IMAGE. coderai-vllm-ocr existed to pin vLLM 0.20.1, the version surya spawns, because a page of crisp Italian came back from 0.27.1 with one U+FFFD per accented letter. That was the wrong conclusion from a real measurement: surya-ocr-2's tokenizer is BYTE-level, and the fleet has been reading accented Italian on 0.27.1 ever since. Two images serving the same weights on two vLLMs is an invisible difference, and it cost three rounds of blaming the pod image. So the pins are now IDENTICAL and the check READS the fleet's pin from disk rather than restating a version -- bumping profiles/vllm cannot silently leave this image behind. What distinguishes the image instead is the OCR PIPELINE on the pod: the surya client venv beside the model, plus pypdfium2 in the main venv, so a page is rasterised and read on the card rather than crossing the network as a ~1 MB base64 PNG (and four more times over for a page resumed four times). CPU TORCH, and it is the whole image-size story. Measured in a resolved surya venv: nvidia 3196 MB + torch 1175 MB + triton 896 MB = ~5.3 GB of CUDA stack, for a venv whose only local torch workload is surya's small line detector. On the pod the GPU is already committed to serving the model, so giving the detector the card would take VRAM and SMs from the thing that matters; on a GPU-less orchestrator it can never execute at all (cuda available: False today while its surya venv carries cu130). ~280 MB instead of ~5.3 GB, and the venv check asserts torch.version.cuda is None so a resolve cannot quietly add 5 GB back. THE DETECTOR rides that same CPU torch, which is why it costs ~30 MB rather than a third torch. detect.py tries ultralytics in-process first (every existing install is unchanged) and falls back to codai/ocr/workers/detect_worker.py in whichever venv has it -- surya's first. A detector that cannot be reached looks like a MISSING detector (the ordinary 503), never a broken install. No weights are baked: detect_model_path defaults to "" and a released image must contain no models, configs, keys or secrets. Tests: 17 warm-floor (+4 for the spin), 14 detector. Four older tests encoded the decisions this reverses -- two asserted the 0.20.1 pin (replaced by one requiring the two pin files to be identical), one asserted the warm floor "must still be single-flight", one counted a source string globally and could not tell intent-held-spelling-changed from a regression (now per path). Suite at the 25 pre-existing failures. NOT BUILT and NOT USABLE YET: nothing has resolved surya-ocr + pillow<11 + torch+cpu + ultralytics together, and the orchestrator cannot drive a pod-side pipeline until the dual mode lands. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Observed live: six pods billing on the RunPod account, none in coderai's interface, and "another engine is booting a pod for 'datalab-to/surya-ocr-2' -- waiting for it" every fifteen seconds for seven minutes. Nothing was booting. A deploy had restarted coderai mid-boot, leaving that pod's URL-less registry entry with no owner. Three fixes, three different mistakes: 1. sibling_is_provisioning() judged the entry's liveness against REAP_GRACE_SECONDS (1200s). That constant answers how long the REAPER waits before killing an unowned pod -- a different question. The registry has a 60s heartbeat, so a dead owner is detectable in seconds. Now PROVISION_STALE_SECONDS (180s = three missed beats). 2. The sibling check ran BEFORE _adopt_shared_pod(), so the recovery path for orphaned pods sat behind the marker that orphaning creates. Five running, registered, billing pods were never adopted. Adoption now runs first: a pod already serving this pool beats waiting for a boot and beats renting. 3. A pod registered at CREATION recorded no $/hr -- the rate was passed only at the ready-registration 80 lines later -- so a boot interrupted before ready could only be reaped with "the registry records no rate for it". The price is known at creation. Also, hourly_usd is now carried over on re-registration like image and gpu already were: an adoption that did not know the rate turned a billing pod free. 14 tests, including the exact live entry (url-less, 910s stale) as a case. test_runpod_client_scale's adopt-before-rent assertion was pinned to the old source spelling; updated to assert the intent, which is now stronger. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.3.66 serialised every pdfium call behind a lock, because pdfium permits one caller at a time and calling it concurrently corrupts its state into "PDFium: Data format error" on valid documents. Correct, and it made rasterisation single-threaded for the whole install -- a ceiling of about one core. pdfium IS safe one-document-per-process, so `ocr.raster_workers` (0 = auto at half the cores capped at 4, 1 = in-process under the lock) hands documents to worker processes. Measured on 12 concurrent 47-page PDFs at 150 DPI: 21.9s -> 10.3s, 2.13x. Not 4x -- the parent is single-threaded receiving the pixels and rebuilding the images, and ~4.5 GB crosses the pipes. The ceiling is the parent. Three things this had to get right, each of which cost a bug first: - A worker is borrowed per CHUNK, not per document. The caller OCRs each window between pulls, which takes far longer than rasterising it, so holding a worker across that would idle it for most of its tenure and cap the install at `raster_workers` concurrent documents -- worse than the lock it replaces. - A pool that cannot start must cost nothing. The first version waited 300s for a worker that could never arrive (missing script), stalling every document for five minutes: the optimisation becoming an outage, which is the exact failure its test was written for. Creation failure is now remembered rather than retried per document, and a busy pool falls back in-process after 30s. - Workers are executed, never imported. `import codai.ocr` costs 377 MB and 1.5s via codai.models.manager -> litellm; paying that per worker, in the subsystem whose other half exists to bound memory, would be self-defeating. The worker imports pypdfium2 and PIL only, ~25 MB. This is why it is not a ProcessPoolExecutor: spawn pickles a function by qualified name and re-imports the package to find it. Also fixes a cost 0.3.65 introduced and nothing but a stopwatch would surface: pricing a document called `pdf[i].get_size()` per page, which LOADS each page object -- 593 ms for a 47-page document, paid UNDER THE LOCK, so every document in the install queued behind every other being priced. `pdf.get_page_size(i)` reads the same numbers from the page tree by index. page_estimate: ~600 ms -> 9.6 ms. Every failure path degrades to a correct in-process render: no pool, a dead worker mid-document, a missing script, an unusable module, an abandoned document. Correctness never depends on the optimisation working. 87 tests across the budget and pool suites. Full suite at the 25 pre-existing failures. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
0.3.65 moved PDF rendering into asyncio.to_thread so a long document would stop stalling the event loop. That was correct in intent and broke OCR within the hour: until then `load_pages` ran ON the loop, which serialised every rasterisation BY ACCIDENT. Removing the accident let documents into pdfium concurrently, and pdfium permits exactly one caller at a time. Concurrent use corrupts its state and reports failed to rasterise PDF (pypdfium2): PDFium: Data format error on documents that are perfectly valid -- so the failure accuses the input, and the operator goes looking at their files. Three valid Diogene PDFs that poppler parsed cleanly were rejected this way. Measured: 1 document OK, 8 concurrent = 8 failures, 24 = 24 failures. After the fix, 48 concurrent 47-page PDFs all return 47 pages. Every pdfium call -- document open, page size probe, page render -- now runs under one lock, held per CALL rather than per document so documents still interleave page by page instead of queueing behind each other's full length. Rendering still happens off the event loop, so this is strictly better than 0.3.64, which serialised AND blocked the loop. Cost, stated plainly: rasterisation is single-threaded, a ceiling of about one core. That is what 0.3.64 effectively was, so it is not a regression against it. Rasterising in a process pool (pdfium is safe one-document-per-process) is the scalability fix and is deliberately not bolted onto this hotfix. Two process lessons, in the tests so they outlive the memory of this: - this path is only ever exercised concurrently, so testing it sequentially tests the one case that cannot fail. I shipped it tested sequentially. - the error STRING predated the change, and I used that to argue the behaviour did too. "This message is not new" is not evidence of "this is not new". Co-Authored-By:
-