- 04 Oct, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authoredEviction now does the same clean swap the model manager does for a busy model (release_idle_vram waits for a request boundary rather than evicting mid-request). OcrManager._release_vram tore pools down immediately, which lost whatever page was being recognised — and with it the caller's document. Taking turns on one card means the handover waits for the handover point. - _Pool tracks in-flight pages (threading-guarded: incremented on the event loop, read from the manager's eviction thread) and drain_and_release() waits for them to finish before tearing down, bounded by ocr.evict_drain_timeout_s (default 60 s) so a wedged engine cannot block a model load forever. - A rebuild waits out an in-progress drain, instead of re-evicting the model that just took its turn and loading into VRAM the drain is about to free. - Evicted stays temporary: the next OCR request rebuilds and asks for room again, so OCR and models can alternate as many times as the traffic asks for. 4 new tests, including the two that matter: an in-flight page survives an eviction that arrives mid-recognition, and three consecutive OCR/model turns each rebuild and re-reserve. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
-
Stefy Lanza (nextime / spora ) authored
surya-on-vLLM was failing every ~20 minutes in production with "Available KV cache memory: -2.81 GiB", and the cause was the 0.35 share itself. vLLM's --gpu-memory-utilization is a fraction of the card's TOTAL memory and every other process' allocation counts against it, so with 3.6 GB of resident bge-m3 on a 24 GB 3090 a 0.35 budget left surya about 5 GB — less than its weights (1.37) plus activation peak at max_model_len=18432 plus CUDA graphs (0.30). Each attempt burned ~3 minutes of boot and profiling, then died, and the next OCR request started over. - vlm_gpu_memory_utilization is now the engine's OWN share and is translated at launch against what else holds the card (effective_gmu): 0.35 with 3.6 GB resident becomes 0.50, i.e. a 12 GB budget of which 8.4 GB is the engine's. Capped at 0.92 so the driver keeps its room. - The evict-first demand (prelaunch_free_gb) now covers the share plus that headroom, so a genuinely crowded card gets models evicted rather than the share clamped. - stop_service_for reports what the instance actually reserved, recorded at launch, instead of the caller's share — under-reporting sent the evictor hunting for VRAM that was already free. - Stopping the managed vLLM is no longer gated on the engine's current serve mode: a mode changed while a service was up leaked its share of the card until a restart. - ocr.fallback_engines (default "auto"): an engine that cannot serve hands the document to the other enabled engines instead of returning 503, fewest-ways-to-fail first. The response's "engine" field names whoever actually read the pages. This also covers a pool evicted mid-document — the failed pages are re-read rather than failing. 21 new tests; the two older vLLM-worker tests now patch the package attribute as well as sys.modules, without which the fake worker silently did not apply. Co-Authored-By:
-
- 02 Oct, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
`docker import` cannot express "empty list", so a base committed by the in-image upgrader (run_oci.sh passes --change "CMD []") came back as null in the flattened image and the equality check refused to retag — over a difference `docker run` cannot see. Normalise empty Entrypoint/Cmd to absent on both sides. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
New engine `olmocr` — allenai/olmOCR-2 (Apache-2.0), a document VLM for the pages the detector-based engines read badly: old scans, maths, multi-column, dense tables. Equations as LaTeX, tables as HTML, reading order sorted; no bounding boxes, so paddle/surya stay the choice when something has to point at the page. It is served three ways (`ocr.olmocr_serve`): "model" (default) routes through coderai's OWN model manager, so a GGUF+mmproj or HF entry in models.json gets VRAM accounting, eviction and the thermal governor — and it works without CUDA; "vllm" gives it its own vLLM instance; "server" attaches to an OpenAI endpoint already running it. olmOCR's trained no-anchoring-v4 prompt is sent verbatim, the page is rendered at 1288 px, the YAML front matter lands in the new `OcrPage.meta` (omitted for engines that report none), and a page the model calls rotated is turned and asked once more. Four fixes the production Surya failures exposed — 169 consecutive failed boots between 08:00 and 09:35 today, every one against the same 11.34 of 23.56 GiB free against a 14.13 GiB demand, each costing the caller a 35 s hang: - The pool evicted for an engine AFTER loading it. For an engine that boots a server claiming a fixed share of the card, the load is what fails. Engines now declare `prelaunch_vram_gb()` and the pool frees that much first. - That reservation is the WHOLE POOL's, asked for once. Per instance it would have asked the manager to free 24 x 8.4 GB on this install's 24-instance surya pool, and an engine that reserved up front was then evicted for a second time, after its server had already taken the memory. - A failed build was retried on every single request. `ocr.build_retry_cooldown_s` (60 s) now fails fast with the reason the boot gave. - `ocr.vlm_gpu_memory_utilization` (default 0.35): the OCR VLM's own share of the card instead of the one `vllm.gpu_memory_utilization` asks for on behalf of an LLM. 0.35 of 24 GB fits surya-2 / olmOCR-2 and leaves the resident engines alone; 0.6 was more than the card had left. Per-instance footprints corrected to match: a surya worker in a served mode speaks only HTTP (0.05 GB, not 1 GB x 24), and an olmocr instance is one in-flight page, holding nothing of its own. Eviction in the other direction is unchanged and tested: the model manager's external releaser tears down the pools AND stops both VLM engines' vLLM services, and the pools rebuild on the next request. 18 new tests (no model, no GPU, no network). Docs: docs/ocr.md. Co-Authored-By:
-
- 30 Sep, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authored
Motus request 4. The most common reel format is a person talking to camera, and only this server can build it: composition synthesises the narration, so it holds the exact audio of each scene, and the lip-sync engines and identity-locked video generation are here too. - codai/compose/presenter.py: per scene — a face (a character profile's front reference, or a portrait), something to move in (a still, the caller's clip, or a short identity-locked shot generated through the ordinary video endpoint so its model config/acceleration/keyframe identity all apply), then an engine driven by THAT scene's narration, held to the scene's exact length and the canvas size. wav2lip and sadtalker are known by name; anything else installed under its own name can be selected by that name, so a newer engine needs no API change. engine=auto prefers a video-driven engine when there is a clip and a portrait engine otherwise. - Layouts in render.py: full (replaces the scene's visuals — the scene may then have none), pip (a masked circle/rounded/rect bubble with an optional border, inside the caption safe area) and split (stacked halves). With a presenter a scene's visuals are joined without an internal transition and composited as one clip; without one nothing changes. - Bottom captions move above a bottom bubble: ASS takes a per-event MarginV, so only the scenes that need it move (captions.py carries margin_v through). - Failure never loses a render: no engine, no face, a crash or a failed composite falls back to the scene's visuals (a gradient when there are none) and warns, naming the scene. New stage "presenter"; result.scenes[i].presenter says which engine and layout, or null. - Lip-sync is the only GPU step in a composition, so it takes a lease from the central scheduler instead of fighting a generation for the card. - POST /v1/video/talking-head (+ /{id}, /{id}/cancel) is the same pipeline as a one-scene full-screen composition, for a character-page preview, capped at 15 s; GET /v1/video/presenter/engines tells a client what is installed so it can grey the feature out instead of meeting a 501. Both stay local in the capability gateway (a video pod has the shims but not this install's characters). - 400 for an unknown character / invalid layout, position, size, shape, motion, or a pip scene with no visuals; 501 with what to install when no engine is present. - docs/video-compose.md, README; tests/test_compose.py 32 -> 43 (engines stubbed, compositing verified on real pixels). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Motus request 3. The character and voice extractors decoded media with their own helpers, which took base64, data: URIs and http(s) URLs but not the /v1/files/... paths compose accepts — so a client had to inline every source clip as base64 (a third larger) even when the server already held the file it had uploaded minutes earlier. - codai/compose/media.py: one resolver for every accepted reference — blob_path() for a sha256: upload id, local_path() for that or a /v1/files path (traversal-checked), fetch_bytes() for the endpoints that want bytes; a file we hold is read from disk, never fetched over the loopback. - characters._decode_source and voice_clone._decode_b64_or_url now delegate to it, which also gives POST /v1/characters and PATCH add_images the same references (they only base64-decoded before), and _decode_audio takes the stored extension from the CONTENT rather than the caller's claim (a data:audio/mpeg URI holding a wav was written as .mpeg). - uploads.find_blob() reuses media.blob_path so there is one lookup, not two. - A bare server path stays refused; a missing blob is a 400 naming the problem. - docs/video-compose.md lists the endpoints that share the resolver; tests/test_compose.py +2 (every reference form, and a character saved from an upload id). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`coderai-docker --upgrade` was unusable: git.nexlab.net's certificate expired on 2026-09-15, the clone failed, and the script gave up — although the same production branch sits on the public GitHub mirror. Nobody running the published image has a stake in that host, so an unreachable default is now a detour, not a dead end. - packaging/linux/launcher/coderai-upgrade: clone_from() per remote; a connection /TLS/DNS failure on the DEFAULT remote retries the mirror and logs both the reason and the source it ended up using. A repo the operator named (--upgrade-repo / CODERAI_UPGRADE_REPO) or an SSH-key clone is never silently replaced, and a ref that does not exist stays an error — papering over that with the mirror would install the wrong code. - Two bugs found while testing it, both silent: * `if ! cmd; then rc=$?` captured the status of the negation (0 on failure), so the fallback branch could never run; * reqs_content() ran under `set -o pipefail` with a `[[ -f ]] && grep` group, so a tree without packaging/common/requirements-oci.txt (or with no deps at all) aborted the upgrade one line after "upgrading x -> y", with no message. - tests/test_upgrade_launcher.py: the real script against local git repos — no network — for the healthy default, the fallback, the explicit repo, a missing ref and an up-to-date install. The certificate itself is on the host serving git.nexlab.net (94.23.108.54, not lisa) and still needs renewing there. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
CoderAI could make every piece of a short video and nothing turned the pieces into a reel — assembly needs ffmpeg and two facts only the server has (how long the narration it just synthesised is, and when each word falls), so a detached client cannot do it. Requested by the Motus project over the HTTP API. - codai/api/compose.py: scenes -> one uploadable video. Scene length = narration + padding (min_duration floor), visuals split it evenly; async job with stage/progress/message/warnings, cancel (the ffmpeg child is killed), a queue that reports its position, and a blocking form for tests. Narration, music and word timings come from this server's own endpoints, called on a loop this module owns — never the request's loop, which may be closed before the render needs it. 400 names the offending scene/field; 501 when a sub-feature has no model (music.generate, timing=stt, TTS for narration) or ffmpeg is missing. - codai/compose/render.py: one normalised clip per visual (so a failed visual becomes a gradient and a warning instead of a lost render), concat or xfade that preserves the total duration, narration padded per scene and loudness-normalised, music looped/faded/ducked with sidechaincompress, burned captions + overlays with the final H.264 High/AAC/+faststart encode, thumbnail. - codai/compose/captions.py: word timings from STT aligned onto the CALLER's words (a mis-heard word never reaches the screen) or estimated; line breaking; ASS with six presets inside a 10%/6% safe area; SRT/VTT with the same text. - codai/compose/media.py: resolve a URL / /v1/files path / data URI / base64 (never a server path), probe with ffprobe, resolve fonts through fontconfig. - codai/api/uploads.py: POST /v1/files/upload, content-addressed by sha256 with GET /v1/files/blob/{hash} so a client can skip a re-upload; type decided by content, not by the client's claim. - compose and uploads are pinned local in the capability gateway (a pod has ffmpeg but not this install's models or files); both advertised in the capability document; caption fonts (Montserrat/Inter/Noto CJK) added to the three image builds. - docs/video-compose.md; tests/test_compose.py (30, real ffmpeg, no model/GPU). Co-Authored-By:
-
- 20 Sep, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
cluster: mDNS discovery with a shared token, prefix-cache routing, Prometheus /metrics + per-key usage, node logs and re-push on return, GGUF row split, cosign-signed images - codai/cluster/discovery.py: _coderai._tcp announce + browse (zeroconf), HMAC fingerprint of cluster.token as membership; members auto-join as nodes, listed nodes with a blank key get the shared token; nodes and engines accept the cluster token (front /cluster/*, engine BearerAuth); env overrides for capability images (CODERAI_CLUSTER_TOKEN/DISCOVERY/NODE_NAME/ADVERTISE_HOST); coderai-docker --host-network - frontproxy/affinity.py + router prefer=: a conversation goes back to the engine/node that served it when the model is resident in several places; engine_for_model now picks the least busy holder - frontproxy/metrics.py: /metrics (requests per key/model/kind/status/engine, engines/nodes up, VRAM, in-flight, RPC, discovery, RunPod spend), usage tables on the Cluster page, server.metrics_public - node logs: /cluster/logs on the node, /admin/api/cluster/logs on the head, log viewer on the Cluster page; a node that comes back gets its assigned models pushed again - per-model split_mode=row (llama.cpp tensor parallel over local + RPC devices); vLLM TP may span nodes (documented) - packaging/sign-images.sh + packaging/cosign.pub: all 21 GHCR images signed - docs/cluster.md §§ discovery, 7 prefix routing, 9 operating, 10 accelerators evaluation; tests/test_cluster_ops.py (11) Co-Authored-By:
-
- 19 Sep, 2026 17 commits
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
never ship the local video_editor config (it carried an API token) or .claude settings; scrub them in the image overlay Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packages: install-from-packages guide, Windows installer (Docker Desktop + WSL2 launcher, Inno Setup build under wine) Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
host pools: keep the single-host wording and _run signature; start/stop commands never inherit across machines Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model 'components' (text_encoder / low_noise / vae → cluster node): the prompt is encoded where the text encoder is, the Wan 2.2 low-noise expert takes over at the expert boundary with the latents handed over, the VAE decodes where it lives and returns the finished video. The far side runs the ordinary generation endpoint with a _handoff block, so loading, acceleration and LoRAs happen as usual there; the scheduler is sliced to resume mid-loop (codai/cluster/components.py). Models page block + tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Synchronous data parallel through torch.distributed with a hand-rolled all-reduce of the adapter gradients (codai/cluster/ddp.py): the engine that takes the job is rank 0, sends the same job with the images inlined to each node in 'nodes' (or the model's lora_train_nodes), all ranks start from rank 0's init, average gradients every step, rank 0 saves. Peers write to scratch and are cancelled with the coordinator. All five in-process trainers (SD1.5, SDXL, Z-Image, flow DiT, Wan incl. 4-bit) take the hooks; two-process test. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per-model 'distribute': n images/videos, embedding and rerank lists, TTS sentences, transcriptions cut at silences, OCR batches are split over every engine that has the model — here and on cluster nodes — and merged back. Files a node generates are addressed through the head. Failed parts retry once elsewhere. Models page block + tests. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packaging: llama rebuild stage checks with ldd (libcuda is a runtime library), rpc-server script needs no import Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
packaging: Dockerfile.update-llama — rebuild llama-cpp-python with GGML_RPC and ship rpc-server over an existing image Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang multi-node, pools, compat-filtered model form Several coderai installs as one: - Cluster nodes (cluster.nodes): another install is an engine of this front — polled on its /cluster/state, assigned models, pinned by name, routed by capability and load; entries pushed to /cluster/reload-config and localized there; Load/Unload reach it through token-guarded twins. Live add/remove. - llama.cpp RPC: codai/backends/ggml_rpc.py registers rpc-server endpoints through ctypes and gives every load an explicit device list (local cards, then the model's rpc_servers); the split and the VRAM pool see remote memory. cluster.rpc_servers runs this machine's rpc-server processes; packaging/build-rpc-server.sh builds it from the vendored llama.cpp; build.sh and the images build llama-cpp-python with GGML_RPC. - vLLM on Ray (pipeline parallel across nodes) and SGLang --nnodes: per-model vllm/kt blocks over the global config, node start/stop commands, ray head and worker mode in boot.sh. - Pools: several URLs per capability remote and several machines per host model, with failover. - Models page: engine/backend selects narrowed live to compatible pairs, impossible pairs refused; Settings → Cluster; /admin/cluster page. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
docs: nineteen images, their sizes and cores, the CUDA-13 vLLM trap, uv/apt/cache markers, direct TLS by default Co-Authored-By:
-
- 18 Sep, 2026 6 commits
-
-
Stefy Lanza (nextime / spora ) authored
PyPI's CDN throttles a machine that has pulled 100 GB of wheels in a day to ~400 KB/s; a venv rebuild that only needed a new apt layer above it then spends hours re-downloading torch. Cache mounts for pip and uv keep the wheels between builds. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Now that a coderai pod serves TLS on its public ip:port, the proxy's 100 s cutoff and body cap buy nothing for our images — and a voice clone already died on it. direct_tcp becomes three-state: unset means direct for coderai images and the proxy for vLLM / llama.cpp / custom (the proxy is the only TLS those have); true/false force it. Model form, save route, harness (--direct-tcp / --proxy) and docs follow. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two verification runs failed with a 404 issued the same second the pod was declared ready — tts on one card, ocr on another — and both passed unchanged on a rerun: RunPod's proxy answers 404 until its route settles, and a catalogue can lag its listing by a moment. A 404/502/503 from a base seen less than a minute ago is retried three times over ~20 s before it is returned. A base up for longer gets no such doubt. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The pod died with 'The NVIDIA driver on your system is too old (found version 12080)': vLLM 0.27.1 on PyPI pins torch 2.13+cu130. vLLM's own index has a +cu129 variant with torch's cu129 wheels, and CUDA 12.x minor-version compatibility runs a 12.9 runtime on a 12.8 driver. uv resolves it with --index-strategy unsafe-best-match, since the torch index also carries stale copies of ordinary packages. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
vLLM on the pod died with 'Engine core initialization failed. See root cause above' — and the root cause was above the six-line tail the 503 carried. Neither core has a C compiler; Triton JIT-compiles vLLM's (and SGLang's) kernels at run time and needs one. A <profile>.apt marker keeps runtime apt packages in the image; vllm and engines-kt get gcc. The worker's error now surfaces the lines that name an error first. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A pod terminated behind the pool's back — the harness reaping by API, a reclaimed spot machine — stayed 'healthy' until the next timed probe, and the next request failed on connection refused for something the pool could have known. The backend now discards it immediately and retries the request once on a fresh acquire; a second refusal is a real outage and raises. Co-Authored-By:
-
- 17 Sep, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai-llama came out at 11.6 GB — the CUDA wheel for five GPU generations on top of the full core — over RunPod's size line. The only thing the CUDA GGUF backend needed torch for was three questions about the card (is there one, total memory, free memory); gpu_probe answers them from torch when present and NVML otherwise. The image drops to the light core plus NVIDIA's runtime, cuBLAS and NCCL wheels. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai-text was transformers only, so a GGUF or a vLLM-served repo still went to an upstream image: no TLS on the direct path, no /boot, no local LoRAs, nothing on the Tasks page. coderai-llama compiles llama-cpp-python with CUDA for every pod GPU on the full core; coderai-vllm is vLLM in its own uv-resolved venv on the light core, found by the worker through CODERAI_VLLM_VENV, with the local vLLM settings forwarded. The upstream engines stay selectable as vllm / llamacpp. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
So callers and tests that patch requests.get/post keep working; only https-to-a-bare-IP takes the pinned path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
direct_tcp traded the proxy's 100 s cutoff for plain HTTP. Now a coderai pod on that path brings its own TLS: a per-install CA (openssl, kept beside the pod registry) signs a short-lived leaf per pod, sent as env like the token; boot.sh and the venv-on-volume bootstrap start uvicorn on it. This side verifies https-to-a-bare-IP against that CA only — pod_http, a requests shim every pod-facing call now goes through. No hostname check: the IP is unknown at issue time, and a foreign cert fails the chain before a name is compared. Proven end to end in a test with a real https server. vLLM/llama.cpp images stay plain on this path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Every pod request went through <pod>-<port>.proxy.runpod.net, which sits behind Cloudflare: a 100 s idle limit per request and a body-size cap. A slow non-streaming render can be cut off with a 524 while the pod is still working. direct_tcp (off by default) exposes the port as tcp and talks to the public ip:port the runtime reports — plain HTTP, no limits. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
pip spent an hour and gigabytes of RAM on sglang-kt's forty exact pins without installing a byte; uv resolved the same set in seconds. A <profile>.venv-<name>.uv marker opts a venv in. Also drop __pycache__ from baked venvs. Co-Authored-By:
-