- 05 Oct, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authoredA pod asked for surya or olmOCR was downgraded to docTR UNCONDITIONALLY. The reasoning was sound — both are VLMs needing a model server, locally they ride coderai's own vLLM engine, and a pod image will not get one (that venv alone is ~10.5 GB against a ~12 GB ceiling, and it would be a second torch beside the one docTR and Surya already need) — but the conclusion was too broad. In their SERVED modes these engines only speak HTTP, and PIL and requests are already in the image. So a pod can run them whenever the operator has pointed them at a reachable endpoint, including at a vLLM pod, which is the composition to prefer over a fat image. - _ocr_external_server(engine) decides it: olmocr_serve="server" + olmocr_server_url, or surya_serve="llamacpp" + surya_server_url. The local-only modes ("vllm", "model") still downgrade, and the log now names the setting to fix instead of only the symptom. - The mode, URL and model ids are seeded to the pod. config.py gained the env overrides for surya_serve/surya_server_url and fallback_engines; only olmOCR's existed. - fallback_engines is pinned to the engines the image actually carries: "auto" would hand a failed page to an engine with no server on that pod, which the fallback added in 0.2.28 made newly reachable. - Engine selection is alias-aware (`alias or served or path`, the convention used throughout the manager). An entry aliased "olmocr" over an HF repo path was read as an unknown engine and silently downgraded. Aliases are how one model carries several configs and how several models share an endpoint, so this path has to honour them. - The licence decision now survives a downgrade: `wanted` was reassigned to "doctr" before the `wanted == "surya"` check, making that branch's own documented intent ("asking for surya by name IS the decision") unreachable. olmOCR needs no pip dependency anywhere — it imports only PIL and requests — which is why nothing flagged its absence from the profiles; ocr.txt now records that and why a pod has no vLLM. 19 new tests; 278 pass across the OCR, front-gate, vLLM-share, cluster, remote-execution, placement and routing suites. Co-Authored-By:Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
Stefy Lanza (nextime / spora ) authored
Escalation alone only ratchets upward, so an opening bid that is too generous was permanent: 0.60 of a 24 GB card for an engine that needs 0.25 is 8 GB no other model could have. A successful boot states how much of the budget was left for the cache, and `footprint = budget - cache` is the only direct measurement of an engine's real cost there is — so it is now read back and used. - _pump_logs captures "Available KV cache memory: N GiB" as it STREAMS. By the time the server answers its health check that line has scrolled out of the 80-line tail behind vLLM's route listing, so it cannot be recovered afterwards. - On a boot with clear slack (> 1 GiB beyond the ~2 GiB target), a smaller share is computed from the measured footprint and recorded for the next start. - The learned store is now a record per key, {"ok": booted, "failed": too small}, and the downward half may never propose a share at or below "failed" + a step. That floor is what stops it oscillating between a budget that works and one that does not. - Over-provisioning now costs one boot instead of the life of the process, and the flat float format the previous version wrote still loads. 6 more tests (shrinking, the tight-fit no-op, the anti-oscillation floor, a missing cache figure, the old format, and the footprint round-trip): 16 in the file, 255 across the OCR/gate/cluster/remote-execution/placement files together. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The 0.60 shipped in 0.2.26 was measured from one model on one card, which is the same mistake as the 0.35 before it: a footprint is not knowable up front. It depends on the model, the card's size, the context and the multimodal limits vLLM profiles for — and the 0.2.26 boot proved the point, since capping max_num_batched_tokens moved the footprint only 11.13 -> 10.88 GiB (all of it CUDA graphs, 0.30 -> 0.05) while the ~9.4 GiB peak stayed put: vLLM sizes surya-ocr-2's encoder cache at 114,688 tokens and profiles it with one VIDEO item at maximum feature size. So the share is now an opening bid and the launcher measures the rest: - ensure_service escalates on the KV-exhaustion signature (_KV_EXHAUSTED: "No available memory for the cache blocks", a negative "Available KV cache memory", or vLLM's free-memory complaint), +0.12 per attempt to at most 4 attempts, capped at the 0.92 ceiling, asking the evictor for the larger prelaunch figure before each retry. - An unrelated failure (missing model, broken venv) is NOT retried — escalating past it would burn minutes of boot per attempt for nothing. - What booted is persisted to ~/.coderai/vllm_gmu_learned.json keyed by model|card-total|ctx|max_num_batched_tokens, so later starts skip rediscovery, a different model/card/limit set discovers its own, and a configured share LOWER than what is known to work is ignored rather than reintroducing a dead budget. - A read-only config dir costs rediscovery, not a failure. The multimodal limits are deliberately NOT capped by default: doing so would stop that instance serving video at all, and the point is that any request type keeps working. docs/ocr.md records the measurement and how to opt in via vllm.extra_args. 10 new tests covering escalation, the step, the ceiling, non-retryable failures, persistence, scoping and an unwritable store. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Live measurement on the 24 GB 3090 settled what the 0.35 share could never do. Two boots of surya-ocr-2: a 9.57 GiB budget reported `Available KV cache memory: -1.54 GiB`, a 14.31 GiB budget reported `+3.18 GiB` — a fixed footprint of 11.11 and 11.13 GiB, i.e. ~11.1 GiB (weights 1.37 + CUDA graphs 0.30 + ~9.4 GiB activation peak) before a single KV block. 0.35 of this card is 8.4 GiB: under the floor, so it could not start no matter what else was resident, and the 06:41 boot that finally succeeded did so only because 6.1 GB held by other models inflated the old add-back to an absolute 0.604. Those same two data points show vLLM charges only its OWN allocations against total × gmu — if other residents counted, they would imply activation peaks of 8.3 and 3.6 GiB rather than one consistent 11.1. So: - effective_gmu no longer adds back what others hold. It made the budget depend on unrelated residents and asked for share*total for this instance while `used` stayed held by someone else — at 0.60 that would have been 20+ GiB of a 24 GB card. The evict-first pass (prelaunch_free_gb) is what clears the card, which is the honest mechanism; the log line now states the share, its GB and what else is resident. - vlm_gpu_memory_utilization default 0.35 -> 0.60 (14.4 GiB, ~3 GiB of KV cache). - New ocr.vlm_max_num_batched_tokens (4096), vlm_max_num_seqs (16) and vlm_max_model_len (0 = inherit), plumbed through ensure_service/_launch_cmd via base.vlm_serve_limits(). This is the better lever: the 9.4 GiB peak is profiled for the batch size, and a VLM OCR engine otherwise inherits the LLM backend's ctx=18432. Chunked prefill is on, so capping tokens per batch shrinks the peak without shrinking the usable context — lower it enough and OCR fits beside an LLM instead of needing 60% of the card. 9 new tests (the caps, the launch command, the footprint floor); the share tests that asserted the add-back now assert its absence, with the reason. Co-Authored-By:
-
- 04 Oct, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authored
/v1/ocr bypassed the gate entirely because it is not in the router's _INFERENCE_PATHS — and it should not be, since that predicate drives the per-model queue, pins, spillover and keepalive, none of which fit OCR. But OCR is GPU work: an OCR request could start a surya vLLM boot while an LLM was mid-generation on the same card, and vice versa. The gate is what turns that contention into turns. - is_gpu_inference_path() = is_inference_path() + /v1/ocr and /v1/ocr/batch, and that is what the swap gate keys on now. is_inference_path() is unchanged, so no other front machinery starts applying to OCR. - All OCR shares ONE gate owner ("ocr"): the OCR manager holds one set of pools and evicts between them itself, so paddle and surya are the same tenant for the purpose of who owns the card. It also means the front need not read the `engine` field out of a multipart body carrying a 100 MB PDF. - _computed_off_this_gpu(): the gate AND the local wait queue are skipped when the work is computed elsewhere — a cluster node (engine.remote), a RunPod pod (model backend), or an external API endpoint (every enabled OCR engine pointing off-box). Such a request holds no local VRAM, so a turn on this card's gate would add latency and swaps for nothing. Applied at all four queue sites (direct, keepalive, both brokered) and both gate sites. 15 new tests, the front proxy's first: path classification, the owner key, each off-box case, and that an OCR tenant and a model tenant alternate without starving each other. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Eviction now does the same clean swap the model manager does for a busy model (release_idle_vram waits for a request boundary rather than evicting mid-request). OcrManager._release_vram tore pools down immediately, which lost whatever page was being recognised — and with it the caller's document. Taking turns on one card means the handover waits for the handover point. - _Pool tracks in-flight pages (threading-guarded: incremented on the event loop, read from the manager's eviction thread) and drain_and_release() waits for them to finish before tearing down, bounded by ocr.evict_drain_timeout_s (default 60 s) so a wedged engine cannot block a model load forever. - A rebuild waits out an in-progress drain, instead of re-evicting the model that just took its turn and loading into VRAM the drain is about to free. - Evicted stays temporary: the next OCR request rebuilds and asks for room again, so OCR and models can alternate as many times as the traffic asks for. 4 new tests, including the two that matter: an in-flight page survives an eviction that arrives mid-recognition, and three consecutive OCR/model turns each rebuild and re-reserve. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
surya-on-vLLM was failing every ~20 minutes in production with "Available KV cache memory: -2.81 GiB", and the cause was the 0.35 share itself. vLLM's --gpu-memory-utilization is a fraction of the card's TOTAL memory and every other process' allocation counts against it, so with 3.6 GB of resident bge-m3 on a 24 GB 3090 a 0.35 budget left surya about 5 GB — less than its weights (1.37) plus activation peak at max_model_len=18432 plus CUDA graphs (0.30). Each attempt burned ~3 minutes of boot and profiling, then died, and the next OCR request started over. - vlm_gpu_memory_utilization is now the engine's OWN share and is translated at launch against what else holds the card (effective_gmu): 0.35 with 3.6 GB resident becomes 0.50, i.e. a 12 GB budget of which 8.4 GB is the engine's. Capped at 0.92 so the driver keeps its room. - The evict-first demand (prelaunch_free_gb) now covers the share plus that headroom, so a genuinely crowded card gets models evicted rather than the share clamped. - stop_service_for reports what the instance actually reserved, recorded at launch, instead of the caller's share — under-reporting sent the evictor hunting for VRAM that was already free. - Stopping the managed vLLM is no longer gated on the engine's current serve mode: a mode changed while a service was up leaked its share of the card until a restart. - ocr.fallback_engines (default "auto"): an engine that cannot serve hands the document to the other enabled engines instead of returning 503, fewest-ways-to-fail first. The response's "engine" field names whoever actually read the pages. This also covers a pool evicted mid-document — the failed pages are re-read rather than failing. 21 new tests; the two older vLLM-worker tests now patch the package attribute as well as sys.modules, without which the fake worker silently did not apply. Co-Authored-By:
-
- 02 Oct, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
`docker import` cannot express "empty list", so a base committed by the in-image upgrader (run_oci.sh passes --change "CMD []") came back as null in the flattened image and the equality check refused to retag — over a difference `docker run` cannot see. Normalise empty Entrypoint/Cmd to absent on both sides. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
New engine `olmocr` — allenai/olmOCR-2 (Apache-2.0), a document VLM for the pages the detector-based engines read badly: old scans, maths, multi-column, dense tables. Equations as LaTeX, tables as HTML, reading order sorted; no bounding boxes, so paddle/surya stay the choice when something has to point at the page. It is served three ways (`ocr.olmocr_serve`): "model" (default) routes through coderai's OWN model manager, so a GGUF+mmproj or HF entry in models.json gets VRAM accounting, eviction and the thermal governor — and it works without CUDA; "vllm" gives it its own vLLM instance; "server" attaches to an OpenAI endpoint already running it. olmOCR's trained no-anchoring-v4 prompt is sent verbatim, the page is rendered at 1288 px, the YAML front matter lands in the new `OcrPage.meta` (omitted for engines that report none), and a page the model calls rotated is turned and asked once more. Four fixes the production Surya failures exposed — 169 consecutive failed boots between 08:00 and 09:35 today, every one against the same 11.34 of 23.56 GiB free against a 14.13 GiB demand, each costing the caller a 35 s hang: - The pool evicted for an engine AFTER loading it. For an engine that boots a server claiming a fixed share of the card, the load is what fails. Engines now declare `prelaunch_vram_gb()` and the pool frees that much first. - That reservation is the WHOLE POOL's, asked for once. Per instance it would have asked the manager to free 24 x 8.4 GB on this install's 24-instance surya pool, and an engine that reserved up front was then evicted for a second time, after its server had already taken the memory. - A failed build was retried on every single request. `ocr.build_retry_cooldown_s` (60 s) now fails fast with the reason the boot gave. - `ocr.vlm_gpu_memory_utilization` (default 0.35): the OCR VLM's own share of the card instead of the one `vllm.gpu_memory_utilization` asks for on behalf of an LLM. 0.35 of 24 GB fits surya-2 / olmOCR-2 and leaves the resident engines alone; 0.6 was more than the card had left. Per-instance footprints corrected to match: a surya worker in a served mode speaks only HTTP (0.05 GB, not 1 GB x 24), and an olmocr instance is one in-flight page, holding nothing of its own. Eviction in the other direction is unchanged and tested: the model manager's external releaser tears down the pools AND stops both VLM engines' vLLM services, and the pools rebuild on the next request. 18 new tests (no model, no GPU, no network). Docs: docs/ocr.md. Co-Authored-By:
-
- 30 Sep, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authored
Motus request 4. The most common reel format is a person talking to camera, and only this server can build it: composition synthesises the narration, so it holds the exact audio of each scene, and the lip-sync engines and identity-locked video generation are here too. - codai/compose/presenter.py: per scene — a face (a character profile's front reference, or a portrait), something to move in (a still, the caller's clip, or a short identity-locked shot generated through the ordinary video endpoint so its model config/acceleration/keyframe identity all apply), then an engine driven by THAT scene's narration, held to the scene's exact length and the canvas size. wav2lip and sadtalker are known by name; anything else installed under its own name can be selected by that name, so a newer engine needs no API change. engine=auto prefers a video-driven engine when there is a clip and a portrait engine otherwise. - Layouts in render.py: full (replaces the scene's visuals — the scene may then have none), pip (a masked circle/rounded/rect bubble with an optional border, inside the caption safe area) and split (stacked halves). With a presenter a scene's visuals are joined without an internal transition and composited as one clip; without one nothing changes. - Bottom captions move above a bottom bubble: ASS takes a per-event MarginV, so only the scenes that need it move (captions.py carries margin_v through). - Failure never loses a render: no engine, no face, a crash or a failed composite falls back to the scene's visuals (a gradient when there are none) and warns, naming the scene. New stage "presenter"; result.scenes[i].presenter says which engine and layout, or null. - Lip-sync is the only GPU step in a composition, so it takes a lease from the central scheduler instead of fighting a generation for the card. - POST /v1/video/talking-head (+ /{id}, /{id}/cancel) is the same pipeline as a one-scene full-screen composition, for a character-page preview, capped at 15 s; GET /v1/video/presenter/engines tells a client what is installed so it can grey the feature out instead of meeting a 501. Both stay local in the capability gateway (a video pod has the shims but not this install's characters). - 400 for an unknown character / invalid layout, position, size, shape, motion, or a pip scene with no visuals; 501 with what to install when no engine is present. - docs/video-compose.md, README; tests/test_compose.py 32 -> 43 (engines stubbed, compositing verified on real pixels). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Motus request 3. The character and voice extractors decoded media with their own helpers, which took base64, data: URIs and http(s) URLs but not the /v1/files/... paths compose accepts — so a client had to inline every source clip as base64 (a third larger) even when the server already held the file it had uploaded minutes earlier. - codai/compose/media.py: one resolver for every accepted reference — blob_path() for a sha256: upload id, local_path() for that or a /v1/files path (traversal-checked), fetch_bytes() for the endpoints that want bytes; a file we hold is read from disk, never fetched over the loopback. - characters._decode_source and voice_clone._decode_b64_or_url now delegate to it, which also gives POST /v1/characters and PATCH add_images the same references (they only base64-decoded before), and _decode_audio takes the stored extension from the CONTENT rather than the caller's claim (a data:audio/mpeg URI holding a wav was written as .mpeg). - uploads.find_blob() reuses media.blob_path so there is one lookup, not two. - A bare server path stays refused; a missing blob is a 400 naming the problem. - docs/video-compose.md lists the endpoints that share the resolver; tests/test_compose.py +2 (every reference form, and a character saved from an upload id). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`coderai-docker --upgrade` was unusable: git.nexlab.net's certificate expired on 2026-09-15, the clone failed, and the script gave up — although the same production branch sits on the public GitHub mirror. Nobody running the published image has a stake in that host, so an unreachable default is now a detour, not a dead end. - packaging/linux/launcher/coderai-upgrade: clone_from() per remote; a connection /TLS/DNS failure on the DEFAULT remote retries the mirror and logs both the reason and the source it ended up using. A repo the operator named (--upgrade-repo / CODERAI_UPGRADE_REPO) or an SSH-key clone is never silently replaced, and a ref that does not exist stays an error — papering over that with the mirror would install the wrong code. - Two bugs found while testing it, both silent: * `if ! cmd; then rc=$?` captured the status of the negation (0 on failure), so the fallback branch could never run; * reqs_content() ran under `set -o pipefail` with a `[[ -f ]] && grep` group, so a tree without packaging/common/requirements-oci.txt (or with no deps at all) aborted the upgrade one line after "upgrading x -> y", with no message. - tests/test_upgrade_launcher.py: the real script against local git repos — no network — for the healthy default, the fallback, the explicit repo, a missing ref and an up-to-date install. The certificate itself is on the host serving git.nexlab.net (94.23.108.54, not lisa) and still needs renewing there. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
CoderAI could make every piece of a short video and nothing turned the pieces into a reel — assembly needs ffmpeg and two facts only the server has (how long the narration it just synthesised is, and when each word falls), so a detached client cannot do it. Requested by the Motus project over the HTTP API. - codai/api/compose.py: scenes -> one uploadable video. Scene length = narration + padding (min_duration floor), visuals split it evenly; async job with stage/progress/message/warnings, cancel (the ffmpeg child is killed), a queue that reports its position, and a blocking form for tests. Narration, music and word timings come from this server's own endpoints, called on a loop this module owns — never the request's loop, which may be closed before the render needs it. 400 names the offending scene/field; 501 when a sub-feature has no model (music.generate, timing=stt, TTS for narration) or ffmpeg is missing. - codai/compose/render.py: one normalised clip per visual (so a failed visual becomes a gradient and a warning instead of a lost render), concat or xfade that preserves the total duration, narration padded per scene and loudness-normalised, music looped/faded/ducked with sidechaincompress, burned captions + overlays with the final H.264 High/AAC/+faststart encode, thumbnail. - codai/compose/captions.py: word timings from STT aligned onto the CALLER's words (a mis-heard word never reaches the screen) or estimated; line breaking; ASS with six presets inside a 10%/6% safe area; SRT/VTT with the same text. - codai/compose/media.py: resolve a URL / /v1/files path / data URI / base64 (never a server path), probe with ffprobe, resolve fonts through fontconfig. - codai/api/uploads.py: POST /v1/files/upload, content-addressed by sha256 with GET /v1/files/blob/{hash} so a client can skip a re-upload; type decided by content, not by the client's claim. - compose and uploads are pinned local in the capability gateway (a pod has ffmpeg but not this install's models or files); both advertised in the capability document; caption fonts (Montserrat/Inter/Noto CJK) added to the three image builds. - docs/video-compose.md; tests/test_compose.py (30, real ffmpeg, no model/GPU). Co-Authored-By:
-
- 20 Sep, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
cluster: mDNS discovery with a shared token, prefix-cache routing, Prometheus /metrics + per-key usage, node logs and re-push on return, GGUF row split, cosign-signed images - codai/cluster/discovery.py: _coderai._tcp announce + browse (zeroconf), HMAC fingerprint of cluster.token as membership; members auto-join as nodes, listed nodes with a blank key get the shared token; nodes and engines accept the cluster token (front /cluster/*, engine BearerAuth); env overrides for capability images (CODERAI_CLUSTER_TOKEN/DISCOVERY/NODE_NAME/ADVERTISE_HOST); coderai-docker --host-network - frontproxy/affinity.py + router prefer=: a conversation goes back to the engine/node that served it when the model is resident in several places; engine_for_model now picks the least busy holder - frontproxy/metrics.py: /metrics (requests per key/model/kind/status/engine, engines/nodes up, VRAM, in-flight, RPC, discovery, RunPod spend), usage tables on the Cluster page, server.metrics_public - node logs: /cluster/logs on the node, /admin/api/cluster/logs on the head, log viewer on the Cluster page; a node that comes back gets its assigned models pushed again - per-model split_mode=row (llama.cpp tensor parallel over local + RPC devices); vLLM TP may span nodes (documented) - packaging/sign-images.sh + packaging/cosign.pub: all 21 GHCR images signed - docs/cluster.md §§ discovery, 7 prefix routing, 9 operating, 10 accelerators evaluation; tests/test_cluster_ops.py (11) Co-Authored-By:
-
- 19 Sep, 2026 17 commits
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
never ship the local video_editor config (it carried an API token) or .claude settings; scrub them in the image overlay Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packages: install-from-packages guide, Windows installer (Docker Desktop + WSL2 launcher, Inno Setup build under wine) Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
host pools: keep the single-host wording and _run signature; start/stop commands never inherit across machines Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model 'components' (text_encoder / low_noise / vae → cluster node): the prompt is encoded where the text encoder is, the Wan 2.2 low-noise expert takes over at the expert boundary with the latents handed over, the VAE decodes where it lives and returns the finished video. The far side runs the ordinary generation endpoint with a _handoff block, so loading, acceleration and LoRAs happen as usual there; the scheduler is sliced to resume mid-loop (codai/cluster/components.py). Models page block + tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Synchronous data parallel through torch.distributed with a hand-rolled all-reduce of the adapter gradients (codai/cluster/ddp.py): the engine that takes the job is rank 0, sends the same job with the images inlined to each node in 'nodes' (or the model's lora_train_nodes), all ranks start from rank 0's init, average gradients every step, rank 0 saves. Peers write to scratch and are cancelled with the coordinator. All five in-process trainers (SD1.5, SDXL, Z-Image, flow DiT, Wan incl. 4-bit) take the hooks; two-process test. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model 'distribute': n images/videos, embedding and rerank lists, TTS sentences, transcriptions cut at silences, OCR batches are split over every engine that has the model — here and on cluster nodes — and merged back. Files a node generates are addressed through the head. Failed parts retry once elsewhere. Models page block + tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packaging: llama rebuild stage checks with ldd (libcuda is a runtime library), rpc-server script needs no import Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packaging: Dockerfile.update-llama — rebuild llama-cpp-python with GGML_RPC and ship rpc-server over an existing image Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang multi-node, pools, compat-filtered model form Several coderai installs as one: - Cluster nodes (cluster.nodes): another install is an engine of this front — polled on its /cluster/state, assigned models, pinned by name, routed by capability and load; entries pushed to /cluster/reload-config and localized there; Load/Unload reach it through token-guarded twins. Live add/remove. - llama.cpp RPC: codai/backends/ggml_rpc.py registers rpc-server endpoints through ctypes and gives every load an explicit device list (local cards, then the model's rpc_servers); the split and the VRAM pool see remote memory. cluster.rpc_servers runs this machine's rpc-server processes; packaging/build-rpc-server.sh builds it from the vendored llama.cpp; build.sh and the images build llama-cpp-python with GGML_RPC. - vLLM on Ray (pipeline parallel across nodes) and SGLang --nnodes: per-model vllm/kt blocks over the global config, node start/stop commands, ray head and worker mode in boot.sh. - Pools: several URLs per capability remote and several machines per host model, with failover. - Models page: engine/backend selects narrowed live to compatible pairs, impossible pairs refused; Settings → Cluster; /admin/cluster page. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
docs: nineteen images, their sizes and cores, the CUDA-13 vLLM trap, uv/apt/cache markers, direct TLS by default Co-Authored-By:
-
- 18 Sep, 2026 6 commits
-
-
Stefy Lanza (nextime / spora ) authored
PyPI's CDN throttles a machine that has pulled 100 GB of wheels in a day to ~400 KB/s; a venv rebuild that only needed a new apt layer above it then spends hours re-downloading torch. Cache mounts for pip and uv keep the wheels between builds. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Now that a coderai pod serves TLS on its public ip:port, the proxy's 100 s cutoff and body cap buy nothing for our images — and a voice clone already died on it. direct_tcp becomes three-state: unset means direct for coderai images and the proxy for vLLM / llama.cpp / custom (the proxy is the only TLS those have); true/false force it. Model form, save route, harness (--direct-tcp / --proxy) and docs follow. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two verification runs failed with a 404 issued the same second the pod was declared ready — tts on one card, ocr on another — and both passed unchanged on a rerun: RunPod's proxy answers 404 until its route settles, and a catalogue can lag its listing by a moment. A 404/502/503 from a base seen less than a minute ago is retried three times over ~20 s before it is returned. A base up for longer gets no such doubt. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod died with 'The NVIDIA driver on your system is too old (found version 12080)': vLLM 0.27.1 on PyPI pins torch 2.13+cu130. vLLM's own index has a +cu129 variant with torch's cu129 wheels, and CUDA 12.x minor-version compatibility runs a 12.9 runtime on a 12.8 driver. uv resolves it with --index-strategy unsafe-best-match, since the torch index also carries stale copies of ordinary packages. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
vLLM on the pod died with 'Engine core initialization failed. See root cause above' — and the root cause was above the six-line tail the 503 carried. Neither core has a C compiler; Triton JIT-compiles vLLM's (and SGLang's) kernels at run time and needs one. A <profile>.apt marker keeps runtime apt packages in the image; vllm and engines-kt get gcc. The worker's error now surfaces the lines that name an error first. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod terminated behind the pool's back — the harness reaping by API, a reclaimed spot machine — stayed 'healthy' until the next timed probe, and the next request failed on connection refused for something the pool could have known. The backend now discards it immediately and retries the request once on a fresh acquire; a second refusal is a real outage and raises. Co-Authored-By:
-
- 17 Sep, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai-llama came out at 11.6 GB — the CUDA wheel for five GPU generations on top of the full core — over RunPod's size line. The only thing the CUDA GGUF backend needed torch for was three questions about the card (is there one, total memory, free memory); gpu_probe answers them from torch when present and NVML otherwise. The image drops to the light core plus NVIDIA's runtime, cuBLAS and NCCL wheels. Co-Authored-By:
-