- Oct 15, 2026
-
-
Stefy Lanza (nextime / spora ) authored
The admin RunPod page showed: error: Unexpected token '<', "<!doctype "... is not valid JSON That '<' is nginx's HTML error page. The page was not failing to find data; its fetch never got an answer, the proxy gave up, and JSON.parse was handed HTML. Measured from inside the host, mid-run, with 64 pages in flight: /healthz 200 in 0.002s no thread needed /v1/models 200 in 0.006s no thread needed /metrics 200 in 0.18s no thread needed /v1/runpod/status no answer in 30s needs a thread `asyncio.to_thread` hands work to the loop's default executor, and a loop that was never told otherwise builds one with min(32, cpu_count + 4) — 20 threads on a 16-core host. Nothing in this codebase ever called set_default_executor. That number is for incidental background work measured in milliseconds; what we put on it is `pool.acquire()` waiting out a pod boot (MINUTES) and a per-page OCR forward (30-300s each), `ocr.max_concurrency` of them at once. The log says it plainly: "engine 'surya': 8 instance(s) ready, 8 lane(s) each — 64 pages at once". 64 blocking calls, 20 threads. Everything else waits behind a page. Nothing was broken. The server could not get a thread to answer on. codai/api/thread_pool.py now sizes two pools from the concurrency the install is CONFIGURED for rather than from its CPU count — these threads wait on other machines, so what matters is how many remote calls may be outstanding, not how many cores can run Python: * the DEFAULT executor, floor 32, for everything to_thread is used for today; * a separate POD executor, floor 64, for the calls that block for minutes, so that however many pages are in flight a control-plane request still finds a free thread. Both entry points install it (the front blocks on pool.acquire for whole pod boots, so it needs this at least as much as the engine). The pod pool copies the context the way to_thread does — the per-page options and the token trace are ContextVars, and run_in_executor does not copy by itself; a page losing its budget because it changed pools would be a nasty way to learn that. /v1/runpod/spend also took its snapshot ON the loop: pods_status() takes the pool locks and the ledger reads a file, so a lock held by a provisioning thread stalled every request rather than one. It runs in a thread now. A thread waiting on a socket costs a stack and nothing else. The asymmetry is the point: too small costs availability, too large costs memory we have. Also fixes a test of my own from an hour ago that asserted "tomorrow 02:00" is within 24 hours and went red at 00:22. The test was wrong about the clock, not the code about the booking. 15 tests, including the live failure reproduced: fill the pod pool and check the control plane still answers. Suite: 3166 passed, the same 13 known-stale failures. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
-
Stefy Lanza (nextime / spora ) authored
THE SURYA BUG. `/v1/ocr/attach/<lease>/…` is an OCR engine attaching to THIS install's gateway. The lease is minted here and lives in this process's memory — and the remote gateway was FORWARDING those requests to the pod, which correctly refused a token it never issued: 404 {"error": {"message": "unknown OCR engine lease"}} surya's client swallows that and returns an empty generation, so every page came back unread with the GPU never asked to read one. Measured on a clean 2-page PDF that chandra (2212 chars) and olmOCR (1992) both read: surya returned 2 characters, and the pod's vLLM logged a single 6-token warm-up for the whole document. $88 of surya time in one day produced nothing. The gateway behind that prefix does its OWN forwarding, with its own pool and its own lease. That is the hop, and it has to start here. `/v1/ocr` itself stays routable, so pinning the attachment does not strand the subsystem. This also corrects an earlier diagnosis of mine: v0.3.87 treated this as a lease VOIDED BY A RESTART and shipped a re-resolve for it. No restart was involved — the lease was wrong from the first page, which is why surya has been broken continuously rather than only after restarts. That work was not wasted (the retry cap took four paid generations per page down to two, and the 502 plus `unread_pages` are what made this diagnosable at all) but it did not address the cause. Third instance today of one mistake: the gateway forwards anything under /v1/ that looks like a model request, including this install's own control plane. /v1/runpod rented a pod to deliver a teardown; /v1/loras was staying local by luck; now the OCR lease. All three are pinned. The structural answer — making control-plane routes opt IN to forwarding — is not in this commit. Also in here, from the fleet-hygiene pass: * accelerate added to every profile that carries transformers (audio, ocr, stt, tts, voice, tts-xtts.venv-TTS), with a test that fails if a new profile reintroduces the gap. Same latent fault that broke embeddings. * an embeddings pod is warmed on /v1/embeddings, not with a chat completion it refuses by design — that 404 was printed on every boot of a healthy pod and cost a wrong diagnosis. * the seed plan is announced once, not every fifteen seconds while a pod boots, so a boot log stays readable when it matters. * `pool:` is dropped from pod NAMES (ownership is the prefix, never the rest). * the reaper owns a pod by REGISTRY as well as by name, so renaming deployment_id no longer orphans pods that are already running and billing. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Five fixes, two of them reported from production in the same hour. EMPTY REPLIES from a chat naming `diogene-alpha`, while `qwen38` — the same weights on the same pod — answered normally. Parser selection reads the model NAME: if 'qwen' in model_lower: -> QwenParser `qwen38-awq` matches; `diogene-alpha` matches nothing and fell to the generic parser, so the model's <think>…</think> block was never separated from the answer. With a modest max_tokens the whole budget goes into reasoning that never closes, and the caller sees an empty reply from a 200 — the hardest kind of wrong to see. A LoRA is named for what it was trained to do; it is the same FAMILY as the weights it adapts, so a name matching nothing now resolves to the base's. Nothing is discarded: the thinking comes back as `reasoning` and `reasoning_content`, as it always did for the base. This also corrects a reading of my own: every temperature-0 comparison in this chain showed the adapter leaking its reasoning while the base answered cleanly, and I took that as evidence the adapter was not loading. It was the parser. /v1/chunk with X-Coderai-Placement: remote answered 502 "nothing configures 'embeddings' to run remotely" while a coderai-embeddings pod — which has the chunker in it — sat there serving bge-m3. A capability-only request names no model, so the per-model placement that moved everything else cannot place it. It now borrows the destination of a model OF THE SAME CAPABILITY that is already remote. Same capability only (an embeddings pod has no OCR deps), and only on an explicit `remote`: nothing should rent a GPU to run a regex split because nobody said otherwise. Registering an adapter is not serving it. vLLM takes its adapters at launch and the code that adds one to a live engine sits on the model-LOAD path, which does not run again for a model this instance already loaded. So a pod could be sent an adapter, register it, report it in /v1/loras and go on answering from the base. Registration now hands it to the running engine, swapping a stale one. A stale content id repairs itself. v0.3.94 fixed what new uploads are stored as; an entry already registered under a non-canonical id stayed broken with no symptom but base-model answers. Seeding and pushing resolve an id against the CONTENT first. And `it-legal` now sets min_chunk_chars: 300 (strict stays 0) — it is the OCR-aware template, so it meets short trailing sections more often, and a 56-character chunk is too small to carry meaning and long enough to match. A runt folds only into a chunk that can CARRY it, so a document of short sections stays several chunks instead of collapsing. Absorption may never cross conclusioni <-> dispositivo at any size: what a party asked for and what the court decided, and a recorded label is a weaker guarantee than refusing the merge. NOTE: this release MOVES CHUNK BOUNDARIES for anyone using `it-legal`. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/chunk already rode the embeddings capability, so it could be placed here, on a RunPod pod (the coderai-embeddings image) or on a cluster node. Verified end to end for the first time rather than inferred from the routing table: a capability image really serves the route, and the middleware really forwards it ("[remote-gateway] /v1/chunk -> … (capability 'embeddings')"). That testing found a real defect. A NAMED structure template was forwarded as a bare name whenever the name was a BUILT-IN, on the reasoning that every coderai has those. True of the name; not true of what it means. `it-legal` was widened to accept markdown headings in 0.3.87 and `it-legal-strict` did not exist before it — so a pod on an older image resolves the same name to different anchors, or 404s on one it has never heard of. A pod fleet is mixed-version by nature: one rented an hour ago runs whatever `:latest` meant then. Chunk boundaries that differ by which machine answered make a retrieval index unmeasurable, which is the one thing docs/chunking.md exists to prevent. Every named template is inlined now, built-ins included: a few hundred bytes on the request, and the boundaries become a property of THIS install's version. An unknown name is still forwarded untouched — resolution failing here must not swallow the request, and the remote's `unknown_template` is a clear error. Six tests pin the placement itself: local by default (a regex split must not rent a GPU), forwarded to a node, left here when only another capability is remote, the template inlined on the wire, and one request pinned local by header. Plus the note that a per-model remote does NOT move chunking, because a chunk request carries no model and only the capability map places it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Measured on a live pod. The pool gate from v0.3.91 worked — [lora] no adapters yet, but this engine serves a pool — starting with LoRA enabled so one can be added without a reload [vllm] launching: … --enable-lora --max-lora… — and the adapter was still dropped: pod seeded expecting : sha256:2e820c… (the local adapter directory) re-registered as : sha256:b8eac366… (an uploaded blob) orchestrator pushed : b8eac366… ✓ 159 MB The bytes arrived. The pod's CATALOGUE did not. It was seeded at boot with the first id, /v1/loras/register repointed the entry at the second, and nothing told the pod — so its engine looked for an id nobody holds, dropped the adapter, and served the base under its name. (The bundle hash is deterministic; the two ids differ because the adapter genuinely changed, not because hashing is unstable.) Two keys missing a version, the same shape as every other bug in this area: * _ensure_adapters sent the BYTES only. It sends bytes and entry now. * teach_model keyed "already taught" on (pod, model), so a re-register could never reach a pod it had spoken to once. Keyed on the adapter's source. /v1/loras listed only the TRAINING registry, so an adapter that arrived by upload+register was invisible there — a successful registration looked exactly like a failed one, which is how this was found. It lists both now, tagged `trained` or `registered`, with the adapter id, base and pool. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/v1/loras/upload put the bytes somewhere; nothing made them answerable. The last step was editing models.json by hand, which is not a step a team retraining adapters can automate — so the answer to "how do we send you a QLoRA?" was "send it to us and we will register it", with a human in it. POST /v1/loras/register {name, base, adapter} closes that. It joins the BASE'S POOL by default, which is the entire economy of the thing: the base weights are already loaded on a rented card, an adapter is tens of megabytes of deltas, and vLLM applies them per request. It inherits the base's placement, card choice and budgets, and deliberately does NOT inherit min_pods or keep_warm — an adapter rides the base's pod, it does not ask for one of its own. Re-registering a name with a new blob is how a retrain ships. A name that already belongs to a real model is a 409 rather than a replacement, and an adapter whose bytes we do not hold is a 404 that names the upload endpoint: a pod sent a content id nobody holds serves the BASE instead, silently, which is the failure this whole area has been about. Identity is (weights, name) here too. An adapter and its base share a `path` by definition, so the lookup is by id-or-alias and never by path. /v1/loras is also pinned local in the gateway. An upload must land in THIS blob store — it is where a pod is sent its adapters from, and the content id this install hands out has to resolve here. It carries no `model` field, so it happened to stay local by accident; that is luck, not design, and the same shape of accident as /v1/runpod, which did not stay local and rented a pod to deliver a teardown. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Seven items from a production field report. Three of them were one cause. /v1/runpod is NOT in the remote gateway's local list. It carries a `model` field like an inference request, so the gateway read it as one, resolved the model to its pod, and FORWARDED the control-plane call there. Asking for six pods back rented a seventh to ask on; the pod has no runpod configuration of its own, so it answered 404 {"detail": "'datalab-to/surya-ocr-2' has no runpod block"} which reads exactly like the orchestrator refusing a name it should know. The same call blocked for minutes while a pod booted to receive it. An operator could not turn off $3.54/hr from the API. Reproduced here while testing the fix: two teardown requests rented two A40s, which is the clearest possible statement of the bug. surya answered 200 with empty pages for a whole document. Three failures in a row, each of which had to be fixed: * its client catches the HTTP error itself and returns a generation with no tokens, so the lease check in recognize_image — which only ran in `except` — never ran, and the pod address was never re-resolved; * the repetition detector was handed that empty string, found no progress in it and re-ran the page at a rising temperature, four full paid generations per page, `tokens: [0,0,0,0]`; * and the page came back as read. We had already measured the ink and said there IS something on it; the judgement never reached the response, so a client treating an empty read as an unreadable scan consumed roughly a thousand documents from its queue and wrote them off. A page with ink and no text is now `unread`: it goes to the fallback engine like any other degraded page, it is listed in `unread_pages`, and a document whose every inked page came back empty is no longer a 200 (`ocr.unread_status`, 502 by default; a partially read document stays 200 with the list). `bge-m3` and `BAAI/bge-m3` each rented their own A40 — the pool key was whichever name the request carried. Keyed on the model ENTRY now, so every name for one set of weights lands on one pool. A client read timeout cannot bound an OCR call: it times each READ, so a response that dribbles never trips it and one stayed open nineteen minutes against timeout=420. `ocr.request_timeout_s` is the server-side stopwatch. Chunking: `it-legal` accepts a decorated heading (`**DISPOSITIVO**`) where the implementation it ports does not, which splits one more section and leaves a short trailing chunk. That was true and undocumented — the table printed the strict pattern. Documented now, with `it-legal-strict` named as the byte-identical one for re-chunking an existing index. `min_chunk_chars` folds a runt chunk into its predecessor, off by default because merging moves a boundary, and never silently: the absorbed section label is reported, since `conclusioni` is what a party asked for and `dispositivo` is what the court decided. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A shared pool is one pod serving several names over one copy of the weights — a base model and its adapters. Live, every request naming the adapter got 404 Client Error: Not Found for url: https://…/v1/chat/completions and kept getting it: the pool keeps the pod warm, so nothing replaced it. The pod, asked directly, listed only the base and reported {"loras": []}. Which of the two names worked depended on which model happened to rent the pod, so it looked intermittent rather than wrong. Five places decided "same model?" by comparing `path`, and a LoRA's path IS its base's path: * the orchestrator seeded the other pool members with no served name, so they arrived with no alias and were registered under their path; * the pod's boot seeder skipped the second entry as already known; * register_model_runtime did the same, which made teaching a running pod a no-op that answered "already known"; * the engine-side seed filter in main.py did it a third time; * and vLLM's service key included the served name, so had the adapter got through it would have started a SECOND copy of a 27B on the same card. Identity is now (weights, name) in all five, and one engine serves one set of weights under every name configured for it: the base under its own, each adapter through --lora-modules, which --served-model-name must not repeat. Adoption checks the image, the GPU class and the bearer token — none of which say anything about the catalogue. So a pod that cannot serve what the pool promises is now TAUGHT: the adapter bytes are pushed, the entry is registered, and the same request is retried on the same pod. Renting a second card to serve a name the running one can hold is the answer this avoids. Once per pod per model; a pod that still refuses is a real failure. An engine is also relaunched when it does not serve the name asked of it: vLLM registers adapters at LAUNCH, and a pod is taught after it boots. Only an ANSWERED probe that omits the name forces that — restarting a loaded 27B because one probe timed out would be worse than the doubt. Finally, raise_for_status() discarded the pod's body, which is the whole message: it names the models the pod actually has. The caller saw only the status line, which reads like a routing fault. The pod's own words now travel. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
THE /v1/embeddings bug, root-caused on a live 0.3.80 instead of guessed at. `RemoteGatewayMiddleware` short-circuits on `any_remote_configured()` BEFORE it calls `resolve_target`. That gate knew about capability endpoints, capability pods and `service_url` -- and nothing about a per-model `backend`. So an install whose remote models are all per-model RunPod pods answered "nothing is remote", every /v1 request was handed straight to the local app, and a GPU-less orchestrator loaded the model itself. Measured on production: POST /v1/embeddings for bge-m3 spent 312 seconds downloading from HuggingFace and returned 500, while /v1/models/test/state?model=bge-m3 answered `"placement":"pod"` in the same process. Both were right. `model_placement` is asked directly by every page that reports where a model runs; the gate is asked by the code that actually routes, and the two disagreed in silence. Two earlier fixes aimed at the same report and missed it: v0.3.77 made `model_capability()` infer "embeddings" for an entry in the embedding section, and the config was given an explicit image and capability. Both were necessary and neither was reached. OCR is why this survived a release: the OCR manager calls `placement.choose` itself rather than going through this middleware, so the pod-side pipeline worked and made the gateway look fine. The gate now makes the same two tests `model_placement` makes -- a `backend` WITH its block, for runpod and for host. A leftover block whose backend was flipped off still does not count, which is how a model's pod is meant to be disabled. Co-Authored-By:
-
- Oct 14, 2026
-
-
Stefy Lanza (nextime / spora ) authored
RETRIEVAL CHUNKING. A new endpoint that splits a document into the pieces an index stores, with the document's structure decided per request by a named or inline template. NOT embedding windowing: `embed_long_input` makes ONE input fit a model window and pools it into ONE vector; this produces MANY chunks, each indexable, each with its section, parent and references. It never pools and never drops text. Pure CPU -- token counts are characters / chars_per_token, so no tokenizer is loaded and no model, GPU or pod is involved. Every image serves it, light-core ones included, and a test asserts no torch/transformers import creeps in. The packing loop is reproduced verbatim, including the parts that look improvable: paragraphs joined with "\n\n" (hence the `+ 2`), and an overlap that is a RAW CHARACTER TAIL which may begin mid-word. Changing any of it moves every existing chunk boundary, and then a re-chunked index cannot be compared with the one it replaces. `conclusioni` is its own label and is never folded into `dispositivo`: in a ruling the dispositivo is what the judge decided, in a filing the conclusioni are what a party ASKED FOR, and merging them would let a demand surface as a holding. TWO DEVIATIONS FROM THE SPEC, both recorded in the source where someone comparing against the original will look: * The ceiling is target + overlap + 2 (3682), not the 3680 the prose states -- which agrees with the spec's own "observed max is 3682" two lines later. Asserting 3680 would fail a correct port. * The verbatim patterns cannot match `**FATTO**`, yet the fidelity list asks for that case. The line-anchored alternatives are widened at BOTH ends (`^[\s*#>_]*fatto[\s*#>_]*$`), which is how the patterns that already coped spell it. Purely additive: every line that matched still matches, so no existing match moves -- but a document whose headers were emphasised used to fall back to flat chunking and now gets sectioned, so ITS chunks change and it needs re-chunking. FORWARDABLE, with the hazard that comes with it. /v1/chunk rides the EMBEDDINGS capability: chunks are on their way to be embedded, that image is full-core so it carries the chunker, and a capability of its own would be one nothing advertises -- or would rent a GPU to run a regex split. It was first classified as deliberately local on the grounds that it is cheap; that was the wrong lens, since the orchestrator is the box that was OOM-killed today, so WHERE the CPU is matters more than how much there is. But a template registered HERE does not exist on the remote: forwarded by name it returns 404, or worse matches a DIFFERENT template with the same name and returns different boundaries silently. So a named non-builtin template is resolved and INLINED before the request leaves. The hook never raises -- a body it cannot parse is forwarded untouched -- because a rewrite that can throw would take down every forwarded request, not just a chunk one. THE OCR PIPELINE NOW COVERS ALL THREE VLM ENGINES, not just surya. Leaving chandra and olmOCR on the orchestrator was an asymmetry rather than a design: surya needed a 1.6 GB client venv baked in, while those two speak only HTTP in served mode (PIL + requests, already in the main venv), so for them the pod side costs nothing beyond the rasteriser. Two bugs fixed on the way, both of which would have produced plausible WRONG text rather than an error: the dispatch hardcoded `engine: "surya"` in the form it posted, and the pod env was surya-shaped throughout -- a chandra pod was told to run surya against chandra's model id. Per-engine tuning and per-engine licence acceptance now travel. AND THE LOG THAT CAUSED A MISDIAGNOSIS. One pod request walks a candidate list until a (card, region) pair has capacity, and each attempt printed "provisioning pod for ...". 312 attempts for 10 pods therefore read as runaway provisioning and was reported as such. It now says "trying candidate N/M for ONE pod", so the shape of the list is visible instead of inferred. What these tests CANNOT do is prove byte-identity with the implementation this ports -- that needs fixtures from a live corpus. They prove the documented behaviour on each path; identity is what decides whether a re-chunk is safe and it is still outstanding. Suite at the 25 pre-existing failures. Co-Authored-By:
-
- Oct 12, 2026
-
-
Stefy Lanza (nextime / spora ) authored
`teach_model` called `seed_model_env(entry, "", value)` — no served name — so the pod registered the entry under its `path`. For an ordinary model that is merely untidy. For an ADAPTER it is fatal: a LoRA's path IS its base, so diogene-alpha registered as cyankiwi/Qwen3.8-27B-AWQ-INT4 and collapsed into the base. Measured on the live pod before the fix: taught https://…:22143 about 'diogene-alpha' (cyankiwi/Qwen3.8-27B-AWQ-INT4) catalogue: […, 'qwen38-awq'] diogene-alpha listed: False HTTP 404: Model 'diogene-alpha' is not available. Taught successfully, still 404 — and the version of this where the base answers under the adapter's name would have been silent. With the name carried, the same pod registered it and served it. Verified by repetition rather than by eye: each name is byte-identical across two runs at temperature 0 and the two differ from each other, so they are backed by different weights. diogene-alpha sha 77f7db98ab19 ×2 (reasons, then answers) qwen38-awq sha 4767ec4fcf00 ×2 (answers directly) One pod, one copy of the weights, the adapter selected per request by name. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A QLoRA trained for a client sat on disk for ten days while a second service rented its own A40 every morning, pushed the adapter over SSH and proxied to it. The path that should have carried it was written but had never been run end to end, and five things were in the way. * `ensure_text_loras` DROPPED a PEFT directory — "publish it to HuggingFace" — which is the shape every trainer writes. It now travels as one deterministic tar (sorted names, fixed uid/gid/mtime, so the content id is stable) and is unpacked by the blob store, because vLLM loads an adapter from a directory: adapter_config.json carries the rank and target modules, and a bare .safetensors is not loadable. * the adapter was re-read and re-hashed ON EVERY REQUEST. A Target is rebuilt per request, so `lora_entry` was set again each time; the bytes were never re-sent (the store is content-addressed) but 159 MB was read and hashed, and the pod asked, only to learn it already had it. Cached per (pod, path, size+mtime), so a retrained adapter at the same path is still sent again. * `_lora_args` passed a CONTENT ID straight through, producing `--lora-modules name=sha256:…`, which vLLM cannot open. On a pod that is the only form the adapter has. * an engine launched for a base model carried none of its adapters. vLLM registers them at launch and one vLLM serves one base, so the pod a shared pool rented for the base alone could never serve an adapter of it — the whole point of sharing the pool. Every model naming the same base now contributes its adapter under its own model id, which is the name requests ask for, so two versions can be A/B'd on one GPU with no reload. * `--max-loras` was never set. At its default of 1, requests naming different adapters cannot share a batch and serialise instead: the cost lands on latency, not on an error. Extracting a bundle is a write from network bytes, so member names are checked for escapes before anything is unpacked, and tar's data filter is used as well. Co-Authored-By:
-
- Sep 30, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Motus request 4. The most common reel format is a person talking to camera, and only this server can build it: composition synthesises the narration, so it holds the exact audio of each scene, and the lip-sync engines and identity-locked video generation are here too. - codai/compose/presenter.py: per scene — a face (a character profile's front reference, or a portrait), something to move in (a still, the caller's clip, or a short identity-locked shot generated through the ordinary video endpoint so its model config/acceleration/keyframe identity all apply), then an engine driven by THAT scene's narration, held to the scene's exact length and the canvas size. wav2lip and sadtalker are known by name; anything else installed under its own name can be selected by that name, so a newer engine needs no API change. engine=auto prefers a video-driven engine when there is a clip and a portrait engine otherwise. - Layouts in render.py: full (replaces the scene's visuals — the scene may then have none), pip (a masked circle/rounded/rect bubble with an optional border, inside the caption safe area) and split (stacked halves). With a presenter a scene's visuals are joined without an internal transition and composited as one clip; without one nothing changes. - Bottom captions move above a bottom bubble: ASS takes a per-event MarginV, so only the scenes that need it move (captions.py carries margin_v through). - Failure never loses a render: no engine, no face, a crash or a failed composite falls back to the scene's visuals (a gradient when there are none) and warns, naming the scene. New stage "presenter"; result.scenes[i].presenter says which engine and layout, or null. - Lip-sync is the only GPU step in a composition, so it takes a lease from the central scheduler instead of fighting a generation for the card. - POST /v1/video/talking-head (+ /{id}, /{id}/cancel) is the same pipeline as a one-scene full-screen composition, for a character-page preview, capped at 15 s; GET /v1/video/presenter/engines tells a client what is installed so it can grey the feature out instead of meeting a 501. Both stay local in the capability gateway (a video pod has the shims but not this install's characters). - 400 for an unknown character / invalid layout, position, size, shape, motion, or a pip scene with no visuals; 501 with what to install when no engine is present. - docs/video-compose.md, README; tests/test_compose.py 32 -> 43 (engines stubbed, compositing verified on real pixels). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
CoderAI could make every piece of a short video and nothing turned the pieces into a reel — assembly needs ffmpeg and two facts only the server has (how long the narration it just synthesised is, and when each word falls), so a detached client cannot do it. Requested by the Motus project over the HTTP API. - codai/api/compose.py: scenes -> one uploadable video. Scene length = narration + padding (min_duration floor), visuals split it evenly; async job with stage/progress/message/warnings, cancel (the ffmpeg child is killed), a queue that reports its position, and a blocking form for tests. Narration, music and word timings come from this server's own endpoints, called on a loop this module owns — never the request's loop, which may be closed before the render needs it. 400 names the offending scene/field; 501 when a sub-feature has no model (music.generate, timing=stt, TTS for narration) or ffmpeg is missing. - codai/compose/render.py: one normalised clip per visual (so a failed visual becomes a gradient and a warning instead of a lost render), concat or xfade that preserves the total duration, narration padded per scene and loudness-normalised, music looped/faded/ducked with sidechaincompress, burned captions + overlays with the final H.264 High/AAC/+faststart encode, thumbnail. - codai/compose/captions.py: word timings from STT aligned onto the CALLER's words (a mis-heard word never reaches the screen) or estimated; line breaking; ASS with six presets inside a 10%/6% safe area; SRT/VTT with the same text. - codai/compose/media.py: resolve a URL / /v1/files path / data URI / base64 (never a server path), probe with ffprobe, resolve fonts through fontconfig. - codai/api/uploads.py: POST /v1/files/upload, content-addressed by sha256 with GET /v1/files/blob/{hash} so a client can skip a re-upload; type decided by content, not by the client's claim. - compose and uploads are pinned local in the capability gateway (a pod has ffmpeg but not this install's models or files); both advertised in the capability document; caption fonts (Montserrat/Inter/Noto CJK) added to the three image builds. - docs/video-compose.md; tests/test_compose.py (30, real ffmpeg, no model/GPU). Co-Authored-By:
-
- Sep 19, 2026
-
-
Stefy Lanza (nextime / spora ) authored
host pools: keep the single-host wording and _run signature; start/stop commands never inherit across machines Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang multi-node, pools, compat-filtered model form Several coderai installs as one: - Cluster nodes (cluster.nodes): another install is an engine of this front — polled on its /cluster/state, assigned models, pinned by name, routed by capability and load; entries pushed to /cluster/reload-config and localized there; Load/Unload reach it through token-guarded twins. Live add/remove. - llama.cpp RPC: codai/backends/ggml_rpc.py registers rpc-server endpoints through ctypes and gives every load an explicit device list (local cards, then the model's rpc_servers); the split and the VRAM pool see remote memory. cluster.rpc_servers runs this machine's rpc-server processes; packaging/build-rpc-server.sh builds it from the vendored llama.cpp; build.sh and the images build llama-cpp-python with GGML_RPC. - vLLM on Ray (pipeline parallel across nodes) and SGLang --nnodes: per-model vllm/kt blocks over the global config, node start/stop commands, ray head and worker mode in boot.sh. - Pools: several URLs per capability remote and several machines per host model, with failover. - Models page: engine/backend selects narrowed live to compatible pairs, impossible pairs refused; Settings → Cluster; /admin/cluster page. Co-Authored-By:
-
- Sep 18, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Two verification runs failed with a 404 issued the same second the pod was declared ready — tts on one card, ocr on another — and both passed unchanged on a rerun: RunPod's proxy answers 404 until its route settles, and a catalogue can lag its listing by a moment. A 404/502/503 from a base seen less than a minute ago is retried three times over ~20 s before it is returned. A base up for longer gets no such doubt. Co-Authored-By:
-
- Sep 17, 2026
-
-
Stefy Lanza (nextime / spora ) authored
direct_tcp traded the proxy's 100 s cutoff for plain HTTP. Now a coderai pod on that path brings its own TLS: a per-install CA (openssl, kept beside the pod registry) signs a short-lived leaf per pod, sent as env like the token; boot.sh and the venv-on-volume bootstrap start uvicorn on it. This side verifies https-to-a-bare-IP against that CA only — pod_http, a requests shim every pod-facing call now goes through. No hostname check: the IP is unknown at issue time, and a foreign cert fails the chain before a name is compared. Proven end to end in a test with a real https server. vLLM/llama.cpp images stay plain on this path. Co-Authored-By:
-
- Sep 16, 2026
-
-
Stefy Lanza (nextime / spora ) authored
gpu_count was passed to RunPod's gpuCount and then ignored: the price ceiling checked one card, min_vram_gb one card's memory, and the engine loaded on GPU 0 while the rest sat idle. Now the pod is ranked as a whole (price x N, VRAM x N, '2x A100'), vLLM gets --tensor-parallel-size N and llama.cpp --split-mode layer. Exposed on the model page, clamped 1-8. Requests forwarded to a pod or host never appeared on the Tasks page, so a running remote job was invisible from the admin. _forward now registers a 'remote' task whose message names the pod it landed on. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod was launched with docTR as its default and still answered 'OCR engine surya is not enabled' — because the request carried engine=surya, and the OCR route honours the request's own field over the configured default. Setting the pod's env changed the default; it did not change what the request asked for. A pod target for OCR now rewrites the request's engine field on the way out, to the engine the pod will actually serve (pod_ocr_engine mirrors the launch env). Multipart is edited on the bytes with every other part untouched and Content-Length corrected; verified through FastAPI's own parser with the real probe body, file intact. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The capability images are ordinary containers; nothing about them needs RunPod. The same image can run on a box you own, a server rented by the month, a second GPU in the next room — and coderai should use it the way it uses a pod: per model, with an endpoint and a token, and nothing else. always on: {"backend": "host", "host": {"url": ..., "api_key": ...}} on demand: add start_cmd (+ optional stop_cmd, boot_timeout_s, idle_timeout_s) start_cmd is whatever starts the thing — docker run, ssh box docker start, a systemd unit. coderai runs it, waits for /healthz, uses the host, and runs stop_cmd after idle_timeout_s with nothing in flight. It only ever stops a host it started itself; one that was already up is never touched. Deliberately absent: GPU search, price ranking, budgets, spot, capacity fallback, the orphan reaper. Those exist because a cloud rents an anonymous card by the second and can lose it. A host is a named machine you are responsible for — the machinery would be wrong, not just unnecessary. Verified for real: a capability container started by command from a cold state, healthy in 6s, used, and stopped by the idle timer. Also fixed on the way: a plain service_url carried no token, so a hand-run capability image behind one answered 401 to everything. service_token travels now. Neither host nor service_* is seeded to a pod: they say where a model runs from HERE, and would make a pod try to forward to itself. Co-Authored-By:
-
- Sep 15, 2026
-
-
Stefy Lanza (nextime / spora ) authored
/v1/ocr names its engine in a multipart field called `engine`, not `model`. The gateway only ever looked for `model`, so an OCR request carried no model as far as it was concerned: per-model placement resolved correctly, the test reported 'target: its own pod', and then the gateway refused with 'nothing configures ocr to run remotely'. Two layers disagreeing about the same request. _MODEL_FIELD maps the endpoints whose field is named something else, and the extractor takes the field name for JSON, multipart and urlencoded alike. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Testing an image model by setting remotes.endpoints["images"] moves every request of that kind. An embeddings remote left on from an earlier test did exactly that on a live box: production embedding traffic went to a rented pod while the local GPU sat idle, and the operator noticed because the card was quiet. tools/runpod_model_test.py writes a runpod block on the single model entry instead — per-model placement already beats the capability map, which is what lets two models of the same kind sit in different places. Writing the config is also not enough. The front pushes a reload and the engine takes it only when idle, so a harness that writes and immediately tests is testing the PREVIOUS configuration. That produced a whole pass of false "nothing configures this to run remotely" failures, and could just as easily have reported a remote pass for a request served locally. GET /v1/models/test/state answers for the process that serves the request; the harness polls it until the placement it wrote is visible and refuses to test if it never is. The new endpoint is in _SKIP_PATHS: forwarded, it would report a pod's view of a config the pod does not have. Co-Authored-By:
-
- Sep 14, 2026
-
-
Stefy Lanza (nextime / spora ) authored
A pod knew only what it was seeded with at boot, so any request naming another model got "not available" for the rest of that pod's life. That makes a pod a fixed catalogue rather than an extension of the local system, which is the whole point of it. When a remote refuses a model it does not know, the gateway now teaches it and retries once: resolve the model to something the remote can fetch (HuggingFace id, URL, or an upload when it exists nowhere else), POST it to the new /v1/models/register, and replay the request. Registration is in memory — a pod is disposable — and idempotent, so a model already loaded is never disturbed. Taught once per pod, tracked by (base, model). The retry is bounded to one attempt and only fires on a 400/404 whose body actually says the model is unknown: a 500 or an out-of-memory must never turn into a retry loop against a rented GPU. A model that resolves to no fetchable source is refused with that reason rather than registered as something the pod could never load. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Placement now has many moving parts (image, weights, adapters, auth, budgets, staging) and most of them fail minutes into a pod boot, in a log nobody is watching. POST /v1/models/test runs the smallest real request for the model's kind and reports what happened, with a Test run button on every model page. It dispatches THROUGH THE FRONT, so it exercises the same routing, auth, pod provisioning and staging a real request would — a test that simulated the path would prove nothing about the path. `where` forces placement for that one request via a new x-coderai-placement header the gateway honours: "local" ignores any remote configuration, "remote" refuses to fall back to local. Without that you cannot verify a pod at all: the request would be served wherever the config decided, and a green result would mean nothing. Kinds with no cheap synthetic input — video (minutes of GPU and real money), faceswap, STT, OCR — report a reachability check and say plainly that no generation was run, instead of implying more than was proven. Also fixed test pollution I had introduced: the gateway fixture replaced module-level functions without restoring them, and one test assigned _model_entry_for unscoped. Both leaked into later tests, which passed alone and failed in the suite — the failure mode that teaches you to distrust the suite. Verified stable across repeated runs. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A text LoRA that exists only on this machine had nowhere to go: vLLM and llama.cpp resolve adapters at launch and have no endpoint to upload one to, so the previous commit could only report and skip it. Such a model is now routed to a coderai TEXT pod — the one pod image that can be sent an adapter — and the transfer is arranged around the fact that a pod is created before anything can be sent to it: 1. the pod is told the adapter's sha256 id, computed HERE, so both sides agree on the name before the pod exists; 2. the bytes go to the pod's content-addressed blob store before the first request, once per pod (a repeat costs one existence check); 3. the pod resolves sha256:<hex> back to a file — text_loras.local_path now understands the content-addressed forms — and applies it. A published adapter still goes to vLLM, which fetches it itself: cheaper and faster, so publishing an adapter you use often remains the better path, and the log says so. A PEFT adapter DIRECTORY is still refused with a reason — the blob store holds single files. packaging/runpod/profiles/text.txt adds the text profile (transformers, peft, bitsandbytes for QLoRA bases). It has to be built and pushed before use; the other nine are published already. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A pod cannot see this machine's disk, so "use my local models on RunPod" needed a way for the pod to GET them. Three, now, picked by `source`: auto — an explicit hf_repo, a path that already is a repo id, or a repo id recovered from the HuggingFace cache path (models--Owner--Repo). Most of a normal catalogue resolves with nothing typed: weights downloaded through coderai live in that cache, and the directory name still says where from. hf — download hf_repo from HuggingFace. url — download model_url directly, which is how most one-off GGUFs are had. upload — send the weights from here. Each server gets only what it can use, refused at configuration time rather than after five minutes of pod boot: vLLM is launched `--model` and takes a repo id only; llama.cpp takes `-hf` or downloads a URL with `-mu`; upload works only on a coderai pod, because vLLM and llama.cpp need the file before their container starts while a coderai pod loads models on demand. Upload is a real option, not a token one — on symmetric gigabit a 30 GB model is a few minutes — but it is the cold cost of every NEW pod, so it is never chosen automatically. /v1/models/upload streams to disk in chunks (never into memory), extracts a tar for multi-file models with every member checked against the destination, and registers the entry in memory only. An existence check by name and size makes a repeat free and refuses a half-copy. keep_warm keeps one pod running so nothing waits for a cold boot. OFF by default and deliberately: a warm pod bills every hour of every day, including the ones nobody uses it. Warm pods start at startup, since a pool created on first use would still pay the boot it exists to avoid. All of it is in the UI: weights source with the right field shown per choice, keep-warm, and an "always run this model locally" pin. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
PER-MODEL PLACEMENT FOR ANY KIND. Sending a model to RunPod used to work only for text: resolve_pod_engine looked at whether the path ended in .gguf, and RunpodBackend implements chat only. A diffusion model pinned to runpod would have been handed to vLLM, which cannot serve it. Now the model's OWN KIND picks the pod: an image/video/TTS/STT/embedding model gets that capability's published coderai image, a GGUF gets llama.cpp, an HF repo gets vLLM. The gateway routes by model BEFORE capability, so two models of the same kind can differ — one video model pinned local with `placement: "local"` while another runs on its own pod with its own budget, which capability-level routing could never express. A pod rented for one model starts with an EMPTY catalogue and would refuse every request for it, so pod_plan seeds CODERAI_SEED_MODELS with that model's entry and the pod registers it at startup (in memory only — a pod is disposable, and writing it would surprise anyone setting the variable on a real install). Only fields that describe the model travel: never local paths, which mean nothing there, and never this deployment's placement settings, which would have the pod renting pods of its own. ORCHESTRATION STAYS LOCAL. /v1/pipelines, /v1/characters and /v1/environments are no longer gatewayable capabilities. They are a sequence of calls to other endpoints, and they already dispatch each step through the front as its own /v1 request — so every step is placed by its own model's config, and a chain can mix local and remote freely. Forwarding the chain would have moved the logic to a pod and placed every step by that pod's catalogue instead. LORAS FOLLOW THE REQUEST. A LoRA is a file on this machine; a pod has never seen it. The blob store is content-addressed, so each adapter is hashed, checked against the remote's /v1/loras/blob/<hash>, uploaded only when missing, and the reference rewritten to that id — repeat requests upload nothing. QLoRA needs nothing extra: the quantisation is in how the base model loads, the adapter is the same safetensors file. An HF id or URL is left alone for the remote to fetch. Also: a vLLM pod now refuses a local model path up front, with the same reasoning as the GGUF guard — otherwise it boots for minutes and dies — and passes `quantization` through, since a local load_in_4bit does not carry over to vLLM. Co-Authored-By:
-
- Sep 13, 2026
-
-
Stefy Lanza (nextime / spora ) authored
Three things, one of which is a security fix. POD AUTH. backends/runpod.py said "the pod's vLLM server needs no auth" and sent no credential. A RunPod proxy URL is reachable by anyone who learns it, so every pod we have ever started was an open GPU on the public internet. Pods are now launched locked to a bearer token, and blank does NOT mean open: the pool generates one per pool unless `allow_open_pod` is set explicitly. Each server takes it its own way — `--api-key` for vLLM and llama.cpp, CODERAI_API_TOKEN for a coderai pod, which required teaching BearerAuthMiddleware to accept an env token since a pod has no auth.json to seed. The health probe sends it too: a locked pod answers 401 on /v1/models, which would otherwise look exactly like "never became ready". All three callers (always-remote backend, capability gateway, front spill) now carry it. SCALING. scale_up_inflight_per_pod was parsed and stored but never used: the pool grew only when it had NO healthy pod, so with max_pods=3 fifty concurrent requests all piled onto pod #1. It now grows when the least-loaded pod reaches the threshold, in the BACKGROUND — the request that proved the pool needs to grow is served by an existing pod rather than waiting minutes for a cold one. `max_inflight_per_pod` (0 = let the remote's batching absorb it) makes requests wait for a slot instead of piling on, for engines that degrade under concurrency rather than queueing internally. CACHE. `sticky_sessions` sends a conversation back to the pod that already served it, so the remote's prefix/KV cache still holds its context; round-robin would re-prefill every turn from scratch. Bounded to the newest 4096 conversations. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Audited all 66 /v1 endpoints against the capability map. One was unmapped: /v1/audio/progress, so polling an audio job would be answered locally while the job ran on the remote — always empty. Mapped to audio_gen, and a test now asserts the whole set, so a newly added endpoint cannot quietly stay local. The bigger hole was results. With the default response_format "url" a render answers with a URL built from the REMOTE's base (http://pod-xyz:8000/v1/files/out.png ): unreachable from the client, and a 404 if fetched here because the file is on the pod. The gateway now rewrites those URLs to this instance and remembers which remote produced each file, so GET /v1/files/<name> follows back to it. Only JSON bodies are buffered for the rewrite; everything else still streams untouched. The map is a bounded routing hint (newest 4096), not a store: a pod reaped for idleness takes its outputs with it, so fetch what you need or ask for b64_json. Documented. Admin, /internal, /chat, /login and /healthz stay local by design — they administer or identify THIS instance. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod pool never cared that it was serving one LLM — it provisions, health-checks, scales and reaps whatever image it is given. Two things were hardcoded: the image and the /v1/models readiness probe. Both are now per-pool, via pod_plan(): auto vLLM, or llama.cpp for a GGUF -> /v1/models coderai your image (required) -> /healthz custom your image + docker_args -> health_path So a capability can now be a pool instead of a fixed URL. Configure `remotes.pods: {"images": {...}}` (the same fields a model's runpod block takes) and the gateway borrows a pod per request and hands it back; `"endpoints": {"images": "runpod"}` is the shorthand. Everything that already guards LLM pods now guards these: $/hr ceilings, rolling spend limits, the global budget, min/max autoscaling, idle teardown, the stale -pod reaper, and the stats page (as `capability:<name>`). That closes the gap this work left open: an idle image or video pod is exactly the kind you forget about, and until now nothing would reap it. A pod that cannot be provisioned — RunPod disabled, budget hit, no GPU under the ceiling — returns 502 rather than falling back to running the model locally. Silent local fallback is a failure mode you only notice by watching your own VRAM vanish. 22 tests, including pod acquire/release (the handle is returned even on the happy path, so the reaper can see it), the runpod shorthand, the provisioning-failure 502, and every pod_plan branch. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Images, video, embeddings, rerank, OCR, TTS, STT, voice cloning, audio generation, stems, 3D and pipelines all load their model in-process, so none of them had a service to point elsewhere. Writing thirteen services would be the long way round; another coderai already exposes all of them, so it can be the service. RemoteGatewayMiddleware sits in front of the routes and, when a request is addressed to something configured as remote, replays it verbatim to that endpoint and streams the answer back — JSON and multipart alike, since the body is forwarded as raw bytes. Routing is per model (`service_url` on the models.json entry) first, then per capability (`remotes.endpoints` in config.json, or CODERAI_REMOTE_<CAP>_URL). Placement matters and is deliberate: it is registered first, so it ends up innermost — auth, rate limiting and request logging all run before it, and a request still has to be authorised here before being forwarded. Chat and completions are excluded: those go through RemoteOpenAIBackend, which keeps usage accounting, tool parsing and the manager's bookkeeping. When nothing is configured remote it returns on its first branch and never buffers a body (cached 5s so it doesn't walk the model list per request). Bodies over `max_body_mb` are served locally rather than held in RAM. A non-forwarded request replays its buffered body downstream untouched — tested, including multipart uploads that still parse locally. docs/remote-execution.md documents all four remotization mechanisms together. Co-Authored-By:
-
