- 30 Jul, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authoredThe RX 580 (Polaris/Vulkan) keeps timing out its GPU ring under relentless back-to-back embedding load — a global throttle can't distinguish a fragile card from a robust one. Make the admission gate PER-MODEL: each model id gets its own semaphore + backlog counter + pacing, resolved from the models.json entry (embed_max_concurrency / embed_max_backlog / embed_min_interval_ms), falling back to the CODERAI_EMBED_* env vars, then defaults. Default min interval is now 200ms (was 0) so GPU starts are paced out of the box, giving the ring breathing room; a robust CUDA model can set it to 0 per-model. Bump 0.1.82. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0196Gm4PNkcuybDj9yfcz3CR
-
- 29 Jul, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authored
The mux serve path (SERVE_BATCH=1, run_serve_mux) force-disables the int8 MTP speculative-decode head ("speculation is not ragged-safe across KV slots") and its batched decode loop has no MTP path at all — MTP lives only in run_serve's single-sequence spec_decode. Since coderai drives colibri with kv_slots=1, drive run_serve instead (SERVE_BATCH=0): accepted MTP drafts mean fewer forward passes = fewer expert-streaming rounds, a real decode win on this IO-bound engine. MuxEngine now selects single-client mode when kv_slots==1: launches run_serve, drains its one-shot startup telemetry, and runs turns synchronously over the run_serve byte protocol (\x02PROMPT frame in; raw text + \x01\x01END\x01\x01 + STAT out) with SIGINT-based cancellation. Multi-slot keeps the batched mux path. pause/resume are no-ops in single-client mode (run_serve would misread the frame; thermal falls back to the front's SIGSTOP). Bump 0.1.81. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
allow_cross ("use both cards for one model") was applied to EVERY engine the moment any model enabled gpu_split — which un-isolated the radeon engine's Vulkan ICD too. Combined with the embedding loader not pinning non-split models to their native card, the radeon engine's embeddings (Qwen3-Embedding, gme-Qwen2-VL) drifted onto the NVIDIA 3090 after a Vulkan device-order change. Scope it: each engine is isolated to its own backend by default; a per-model split names its lead engine (the model's `engine` field), and only that lead's card gets cross-backend visibility. A global offload.gpu_split still crosses all engines (explicit opt-in). Preserves nvidia-led splits onto the RX 580 while keeping radeon/intel Vulkan engines pinned to their own card. Bump 0.1.80. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Embeddings bypassed the front request queue (relayed unqueued) and the per-model lease scheduler, so the only guard was ratelimit.py's per-IP fixed window. A single authenticated client could pipeline unlimited embedding requests straight at the GPU — which on the Polaris/RX580 Vulkan backend wedged the SDMA ring (ring timeout -> GPU reset -> device lost from bus). Add _embed_admission() around create_embeddings: bounded concurrency, backlog shedding (429 when in-flight >= concurrency + backlog), and optional pacing between GPU starts. Tunable via CODERAI_EMBED_MAX_CONCURRENCY (2), CODERAI_EMBED_MAX_BACKLOG (32), CODERAI_EMBED_MIN_INTERVAL_MS (0). Bump 0.1.79. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
GLM output was garbage: colibri keeps only EOS as a hard stop in serve mode (it filters the non-EOS special-token stops for tool-call safety, #401), so the model ran past its turn and emitted <|user|>/<think>/</think> and repeated the prompt. - backend: clean_glm_output cuts the reply at the first turn marker (<|user|>, <|observation|>, <|assistant|>, <|system|>, <|endoftext|>) and strips control tokens (<think>,</think>,[gMASK],<sop>). Applied to both non-stream and stream paths; the stream holds back a short tail so a marker split across chunks never leaks. - Early stop: generate paths pass a `cancelled` hook that fires the moment a turn marker appears, so colibri CANCELs instead of decoding to max_tokens — a big saving at streaming-bound decode speeds. MuxEngine.run now treats our own CANCELLED ack as a clean early stop (returns partial stats) instead of raising. - alias: _resolve_container now resolves the configured colibri model_id (e.g. glm-5.2-colibri) and colibri.model_path to the container, not only the full repo path — so `{"model":"glm-5.2-colibri"}` loads instead of "no container resolved". Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Per design: a hot CPU is a global hardware event, so every engine pauses — not just the CPU-heat source. Reverts the per-engine CPU exemption from 0.1.75 (front _thermal_loop) and 0.1.76 (engine-level wait_until_safe): the shared-CPU term again applies to all engines. The original "radeon stuck while colibri ran free" bug is addressed the right way — by making colibri ACTUALLY honour the pause (targeted SIGSTOP that keeps the engine's HTTP alive, and colibri's cooperative PAUSE/RESUME mux frame), both retained — so colibri stops, the CPU cools, and all engines resume together instead of the innocent engine being stranded. Removed the now-unused _scan_pgroup_cpu/_engine_cpu_relevant helpers and _self_cpu_relevant. Co-Authored-By:
-
- 28 Jul, 2026 9 commits
-
-
Stefy Lanza (nextime / spora ) authored
Companion to the front-side fix (0.1.75): the ENGINE-level guard (wait_until_safe in codai/models/thermal.py) still applied the global CPU term to every engine, so a GPU-bound engine (radeon embeddings, ~0 CPU) kept "Cooling … CPU>78 — waiting" while colibri (596% CPU, dense-on-CPU) held the shared CPU above resume. It never cooled because the engine that was cooking it wasn't the one being made to wait. wait_until_safe now honours the CPU term (pause / join-cooldown / stay-in-cooldown / soft-throttle) only when THIS engine's own process tree is a real CPU-heat source (_self_cpu_relevant: rolling-max of read_process_tree_cpu ≥ 1.5 cores). The radeon engine no longer waits on colibri's CPU heat; the nvidia engine (colibri in its tree) still throttles itself. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Fixes the "only radeon was stopped while everything under throttle" case and gives colibri a clean pause instead of SIGSTOP. Thermal (engine_supervisor): - Per-engine CPU gating: measure each engine's process-group CPU (rolling max) and apply the GLOBAL cpu_hot/cpu_warm term ONLY to real CPU-heat sources. A GPU-bound engine (embeddings) is never paused/stranded in the resume<CPU<high dead-band that a CPU-heavy engine (colibri) keeps the shared CPU parked in. - Targeted SIGSTOP: _thermal_signal now freezes the engine's native compute CHILDREN (colibri/ds4/whisper-server), leaving the Python HTTP server alive to ack pause/resume — killing the resume-fail / SIGSTOP
↔ SIGCONT thrash. Falls back to killpg for in-process (torch) engines. colibri graceful PAUSE/RESUME: - packaging/patch-colibri.py: idempotent serve-mux patch adding PAUSE/RESUME control frames — colibri idles the decode loop between tokens (keeping KV) instead of being frozen; build.sh applies it after clone. - MuxEngine.pause()/resume() + colibri_worker.pause_all()/resume_all(); the engine's /internal/thermal-pause|-resume now drives them, so the cooperative thermal throttle can cool the box without SIGSTOP. Credits (requested): clear acknowledgements for the brilliant engines coderai builds on — colibri (JustVugg), ds4/DwarfStar (antirez), llama.cpp/whisper.cpp (Gerganov), and VPR/geo research — in the admin settings UI, README, and docs (new docs/glm-colibri.md, ds4 doc footer). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
colibri's context is the CTX env (default 4096), DISTINCT from NGEN (max new tokens). We only set NGEN, so any prompt over ~4095 tokens was rejected ("prompt does not fit … CTX=4096") — coding clients with big system prompts always tripped it. MuxEngine now sets CTX from the model's configured n_ctx (a caller/extra_env CTX still wins). GLM-5.2 (glm_moe_dsa) uses compressed MLA KV held in host RAM, so large contexts cost RAM, not VRAM — the model itself supports up to 1,048,576 positions. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
colibri's GPU tiering is OFF unless COLI_CUDA=1 (CUDA/HIP) / COLI_METAL=1 is set — it is NOT implied by shipping a CUDA binary. We passed CUDA_EXPERT_GB without it, so the engine aborted at startup: "CUDA_EXPERT_GB requires COLI_CUDA=1" -> exited before READY. _build_env now defaults the backend on for cuda/hip (COLI_CUDA=1) and metal (COLI_METAL=1) targets — coderai ships a CUDA build and routes colibri to the CUDA engine — and only sets CUDA_EXPERT_GB when the CUDA backend is enabled. extra_env is applied last so COLI_CUDA=0 can force a CPU run. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The engine is a pre-built binary, bundled in the image and built on a host with the CUDA toolkit — never compiled per-request in the CUDA-*runtime* container (which has no nvcc). Build it with the host CUDA 13.x toolkit like ds4: the binary links libcudart.so.13, resolved in-container from /opt/coderai/local-libs (the CUDA-13 runtime coderai already ships for PyTorch), so it adds no portability constraint. - worker: _detect_build_target now needs a REAL nvcc (a runtime container has /usr/local/cuda but no compiler); ensure_built fails early with a clear "pre-compile the binary" message instead of cloning and dying deep in `make` with the confusing "nvcc not found … backend_cuda.o Error". _make_args defaults CUDA to a PORTABLE arch (sm_80..120 + PTX) and points at the resolved nvcc/CUDA_HOME. - build.sh --colibri: require a real nvcc and default CUDA_ARCH=portable (like ds4's cuda-generic), so the bundled binary isn't locked to the build host's GPU. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
hf leaves a stale <sha>.<etag>.incomplete orphan in blobs/ after a resumed or retried chunk even though the real file finished — so "download complete" and the model page disagreed (settings said done, the page showed
⚠ incomplete). - model list: an .incomplete file now counts as an unfinished download ONLY when its final blob (<sha>) is absent; an orphan whose final blob exists is ignored. Fixes the false badge for ALL HF models, not just colibri. - colibri catalog: _colibri_repo_complete now tests for dangling snapshot symlinks (the true "file missing" signal) instead of .incomplete presence, so the settings "present" indicator matches reality. - colibri catalog label → the real size (~429 GB / 400 GiB). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The GLM-5.2 model is a DIRECTORY (an HF repo of int4 .safetensors shards + config + tokenizer/MTP head), not a single file/GGUF — so it downloads via the normal whole-repo snapshot and lives in the HF cache snapshot dir. - routes: _COLIBRI_DEFAULT_MODELS catalog (mastouri/GLM-5.2-colibri-int4-g64-with- int8-mtp) + GET /admin/api/colibri/default-models; extend /admin/api/model-add to accept `backend` and `alias` so a downloaded repo registers as a colibri-backed model in one step. - settings.html: a "Download the GLM-5.2 container" picker + button in the colibri card — full-repo snapshot (no file_pattern → a directory), then registers it as a colibri model addressable by the configured model id. - backends/colibri: _resolve_container resolves an HF repo id → its local snapshot DIRECTORY (snapshot_download local_files_only), so a normally-downloaded model serves without any manual path wiring. - models.html: add "colibri" to the per-model backend dropdown (manual add path). - app: allow the new catalog endpoint through the no-auth admin allowlist. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- parser: add GLMParser + parse_glm_tool_calls/strip_glm_tool_calls for GLM-5.2's <tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value>...</tool_call> format (byte-compatible with colibri's own parser). Gated on the <arg_key> marker so a generic <tool_call>{json} from other families is never hijacked. Declared-type coercion keeps string args verbatim (no "12345"->int). Unclosed-box recovery for budget-truncated calls. Wired into family selection ('glm'/'colibri'), the model-agnostic ToolCallParser path, and both strip_tool_calls_from_content paths. - manager: colibri no longer seizes the whole GPU like ds4. It pins only a configurable expert tier (CUDA_EXPERT_GB) and streams the rest, so it coexists and is evicted like any other model. VRAM footprint estimated from cuda_expert_gb (+overhead) until measured. Verified: typed/untyped parse, strip, unclosed recovery, non-GLM gating, and the by-name dispatcher all correct. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Add JustVugg/colibri as a managed engine, mirroring the ds4 integration but driving the pure-C `colibri` binary DIRECTLY over its stdin/stdout mux protocol (docs/serve_protocol.md) instead of proxying to a Python gateway — coderai reproduces openai_server.py's engine client and GLM-5.2 chat template itself. - config: ColibriConfig (disabled by default; model is a directory container, not a GGUF — routes by model_id/alias/name, no arch sniff) - codai/api/colibri_worker.py: clone+build (make colibri CUDA=1), MuxEngine protocol client (READY handshake, SUBMIT/DATA/DONE, KV-slot pool, CANCEL, stderr log pump), per-container engine registry - codai/backends/colibri.py: ColibriBackend + render_chat (byte-exact port of colibri's GLM-5.2 template; the engine tokenizes what we send) - front routing: `colibri` capability on nvidia/cuda/auto; router/assignment/ app/engine_supervisor thread config.colibri - manager: colibri_should_handle, backend selection, /v1/models surfacing, exclusive-VRAM eviction (wants the whole GPU like ds4) - admin: config get/set + per-model overrides; settings.html card + models.html row - packaging: build.sh --colibri, OCI bundle (repo+binary, no 372GB model), entrypoint seed, CODERAI_COLIBRI_DIR, smoke test Ships OFF: no routing changes until colibri.enabled + colibri.model_path are set. Verified offline: config round-trip, routing predicate, render_chat byte-match vs colibri's own, MuxEngine end-to-end against a fake engine, CPU engine builds. Co-Authored-By:
-
- 26 Jul, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authored
Second VPR option for REQUEST-vpr-embeddings.md, alongside the in-process EigenPlaces. SALAD is near-SOTA for place recognition but its deps (pytorch_lightning, pytorch_metric_learning) are heavy and version-sensitive, so it runs in a SEPARATE process with those deps quarantined in a --system-site-packages venv (shares the main torch, adds only the extras). The main engine venv never imports them — verified clean. - codai/api/vpr_salad_server.py: stdin image-path -> stdout JSON embedding server (same protocol as dinov2-embed), GPU-preferred with CPU fallback, announces its true width (8448) on ready. - embeddings.py: 'salad'/'dinov2-salad' -> 'vpr-server' backend that spawns the server via /cache/salad-venv/bin/python; reuses the dinov2cpp reader/cleanup. - Self-healing: _EmbeddingModel gains a respawn hook. If the manager kills the child under VRAM pressure but keeps the cached model, the embed path now respawns the server in place instead of failing forever with a stale handle (this was causing persistent "process died" 500s once salad was evicted). - packaging/setup-salad-venv.sh: idempotent creator for the isolated venv. Both VPR models pass the decisive ordering test on same-place/different-place image sets (zero overlap between same-place min and different-place max); SALAD's margin (0.516 vs 0.150) is wider than EigenPlaces' (0.479 vs 0.349). Verified: 8448-d L2=1.0, deterministic, ~0.2s warm / ~8s cold, GPU, subprocess isolation intact. Config: models.json 'salad' + alias 'dinov2-salad', engine nvidia. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Implements REQUEST-vpr-embeddings.md: a visual place recognition model that turns a photo into ONE L2-normalised descriptor trained so two images of the SAME place land close — the building-identity discrimination dinov2/gme/geoclip lack (they rate similar-looking houses as matches). HomeHunter uses it to match listing exteriors against Street View panoramas. Backed by EigenPlaces (gmberton, ResNet50, 2048-d) loaded via torch.hub. Chosen over SALAD/MixVPR for a clean dependency footprint: torch + torchvision only (no pytorch_lightning). ImageNet-normalised, 512x512 eval transform (matches training crops); output L2-normalised (idempotent — the net already ends in an L2 layer). On GPU (nvidia engine device); CPU only as fallback. The image data URI arrives in `input` (per the contract) or the `image` field. torch.hub cache pinned to a persistent TORCH_HOME (/cache/torchhub) with the repo + weights + trusted_list pre-seeded, so the engine loads offline and non-interactively (a cold torch.hub.load would otherwise hit an interactive trust prompt that EOF-crashes a server). Verified: 2048-d L2=1.0, deterministic, one vector/image; and the decisive ordering test — same-place pairs (min cos 0.479) rank strictly above every different-place pair (max 0.349), zero overlap. Config: models.json 'eigenplaces' + alias 'vpr', engine nvidia. SALAD (pytorch_lightning, subprocess-isolated) to follow as a second, higher-accuracy option. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Implements the HomeHunter request (REQUEST-geoclip-embeddings.md): two model ids on the existing embeddings endpoint, both returning 512-d L2-normalised vectors in ONE shared space so a property photo can be scored directly against candidate GPS coordinates. geoclip — image encoder (CLIP ViT-L/14 + projection MLP), on GPU. Accepts the image data URI in EITHER `input` (per the request) or the `image` field (like dinov2). geoclip-location — GPS encoder (equal-earth + RFF MLP), pinned to CPU: it is a tiny MLP, so CPU is fast, frees VRAM, and is bit-exact across batch sizes (GPU reduction order otherwise perturbs a cached coordinate by ~1e-7). Batches an array of "lat,lon" strings to one vector per element, in input order. Both sides load from the SAME bundled GeoCLIP checkpoint, so the spaces coincide by construction — the mismatched-checkpoint failure the request warns about cannot happen. Loaded as separate _EmbeddingModel('geoclip', ...) instances so each is independently evictable. transformers>=5 compat: CLIPModel.get_image_features now returns a BaseModelOutputWithPooling (768-d projected features in pooler_output) instead of a bare tensor, which breaks GeoCLIP's internal mlp() call. The image path extracts the 768-d tensor itself (tolerant of both the old tensor and the new object) before the projection MLP. geoclip (+geopy, geographiclib) added to requirements.txt; its heavy deps (torch, transformers, Pillow, pandas, numpy) are already pinned, so no ML-stack churn. Verified end-to-end: 512-d L2-normalised both sides; batching + index order; same request bit-exact; both ids in GET /v1/models; image·location scoring varies by image. Co-Authored-By:
-
- 24 Jul, 2026 17 commits
-
-
Stefy Lanza (nextime / spora ) authored
The nvidia container runtime only injects the NVIDIA Vulkan ICD when the 'graphics' driver capability is requested. --nvidia ran with the runtime default (compute,utility), so inside the container Vulkan could see only the AMD card (RADV) and llvmpipe — never the NVIDIA GPU. A Vulkan-only GGUF embedder (dinov2-embed, built with GGML_VULKAN) therefore could not run on the NVIDIA card at all; it was pinned to the AMD RX 580 or forced to CPU. Add `-e NVIDIA_DRIVER_CAPABILITIES=all` (honouring any caller-exported value) to the docker --nvidia args so the NVIDIA Vulkan ICD is injected and Vulkan enumerates the NVIDIA GPU. Enables running the GGUF dinov2 embedder on the 3090 via Vulkan under the nvidia-gguf engine (device pinned via GGML_VK_VISIBLE_DEVICES / VK_ICD_FILENAMES; the build hardcodes ggml_backend_vk_init(0)). Takes effect on the next container (re)launch. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
POST /v1/images/generations for a masked image transformer (Z-Image) crashed with "`attn_mask` is not supported for flash-attn 2." → HTTP 500. The diffusers "active attention backend" is PROCESS-WIDE global mutable state (_AttentionBackendRegistry class attribute) shared by the image and video paths. The video path sets it to flash-attn-2. A transformer whose per-module backend is None reads that global at dispatch time; flash-attn-2 rejects attn_mask, so Z-Image (which uses caption masks) crashes. The previous fix reset the global to native before generating, but that reset is NOT atomic with the pipeline() call (it runs in a to_thread worker), so a concurrent/subsequent video generation races and flips the shared global back to flash before image attention dispatches. Fix: pin the image denoiser's per-module attention backend to native (SDPA, mask-supporting) via set_attention_backend("native"). The dispatcher passes processor._attention_backend explicitly, bypassing the shared global — so image attention is immune to the video path flipping it. Video is undisturbed (it sets its own explicit per-module 'flash'). Image pipelines were never intentionally on flash, so this is non-regressive. Global reset kept as a fallback. Verified: unsloth/Z-Image-Turbo-unsloth-bnb-4bit now returns 200 with an image; no "attn_mask is not supported" in the log. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
POST /v1/embeddings to the GME-Qwen2-VL GGUF embedder returned HTTP 500 for large photos. The qwen2-vl vision tower turns a big image into >n_ctx image tokens (observed batches of 1012–2048), and mtmd's decode then fails — "decode: failed to find a memory slot for batch of size N" / "failed to eval chunk 1" — because the image tokens plus the chat-prompt frame exceed the model's KV context. Small images (few hundred tokens) were unaffected. mtmd's own image_max_tokens budget is NOT honoured by this llama.cpp build (2048-token batches slipped through), so add a hard, Python-side backstop: _resize_image_to_token_budget() downscales any oversize PIL image (aspect preserved, 28px patches) to at most n_ctx-128 tokens BEFORE it reaches mtmd, so a request can never overflow the context regardless of input size. The budget derives from the live n_ctx, so it self-adjusts to any model config. Pairs with raising the GME model's n_ctx (instance models.json) so a full-size image (~2048 tokens) fits comfortably; n_batch/n_ubatch follow n_ctx in the loader. Verified on the radeon Vulkan embedder: 336² / 896² / 1400² / 1680² images all return 200 (the 1400²/1680² are logged resizing to ~2400 tokens to fit the 2432 budget); previously the ≥1024-token cases 500'd. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
On a global CPU cooldown the thermal supervisor already pauses ALL engines (want_pause = gpu_hot or cpu_hot, applied per-engine), and the logs confirm every engine gets a pause. But the Tasks page often showed only ONE engine as cooling while the others looked like they were still running. Cause: two threads write engine.cooling. The thermal loop sets it when it pauses an engine; the health poll loop overwrites it every tick with the engine's OWN self-reported cooldown (cooling=d.get("cooling")). An engine only self-reports cooling while sitting in its own wait_until_safe loop (it has an in-flight request). An engine the front paused while IDLE reports cooling=None, so the poll loop cleared the front's pause indicator — the UI then showed that engine as running mid-cooldown. Fix: while the front holds an engine paused (engine.therm_paused), pass the update_state sentinel (cooling=False, "don't touch") instead of the engine's self-report, so the front's indicator survives. Once the front resumes the engine, the engine's self-report flows through again as before. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A GGUF vision model (e.g. Gemma-4-14B) served correct image descriptions on its FIRST load after a restart but hallucinated an identical answer for every image on every subsequent load — the image was silently flattened to a "[image_url content]" text placeholder and the model never saw pixels. Root cause was a config-key mismatch that self-polluted the in-memory config. On-demand loads arrive as a basename (Gemma-...gguf) while the real models.json entry is keyed by full path. record_vram_delta() resolved the write target via _config_for_model_key(), which — unlike _config_for_model() — did NOT fall back to basename/alias matching, so it returned {} and then persisted a NEW basename-keyed entry holding ONLY the measured_* fields (no mmproj, no n_ctx). On the next load _config_for_model()'s exact-match hit that stripped basename entry FIRST, before the basename loop that would have found the real full-path config, so mmproj was dropped, supports_vision went False, and the vision projector never loaded. Fix: - Add _resolve_config_key(): returns the actual self.config key for a model (exact -> alias -> basename), the single canonical key readers and writers must agree on. - Route _config_for_model() through it; give _config_for_model_key() the same basename/alias fallback so it can no longer return {} for a basename. - record_vram_delta()/_persist() now read and persist measured fields under the canonical key, merging into the real config instead of spawning a stripped shadow entry. Verified: after restart the 14B loads with "mmproj ... (vision enabled)" on every reload and three distinct test images produce three distinct, accurate descriptions; the measured-VRAM writeback now logs "(force_vram_update)" (real config resolved) instead of the old "(no used_vram_gb)" (empty config). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reworked engine_request_min_interval_ms from start-spacing to a proper post-completion gap: _rate_acquire holds a per-engine lock for the whole request, _rate_release frees it only `interval` ms AFTER completion (via loop.call_later, non-blocking) — so consecutive requests to the engine are ALWAYS separated by at least that idle GPU time regardless of request duration. Wired acquire/release into all 3 inference dispatch paths with release in every finally/early-return. _rate_acquire refreshes config on mtime change so a value saved in the web UI applies to the next request. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Adds a "Rate limit (ms)" column to the Settings per-engine overrides table for engine_request_min_interval_ms (0 = no limit). Backend GET returns the map, POST saves it via the int-override sanitizer. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
New server.engine_request_min_interval_ms (engine name → ms, 0/unset = off). _rate_gate spaces inference dispatch STARTS to an engine by at least the interval, capping request rate and inserting idle time between GPU submissions — a stability lever for a marginal card (e.g. RX 580) that wedges under sustained back-to-back Vulkan compute. Wired into all three inference dispatch paths after the swap-gate; the request itself runs unthrottled, only the start cadence is gated. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
New server.engine_env_overrides (engine name → {VAR: val}), merged into the engine's process env at spawn. Lets low-level driver knobs be set without hardcoding — set radeon → RADV_DEBUG=syncshaders to serialize RADV shader dispatch and test whether the Polaris compute-ring async race (ring comp_x.y.z timeout) is what wedges the RX 580 under sustained Vulkan compute. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
At front startup _build_engines runs vulkaninfo to enumerate GPUs; on a wedged Polaris card that process blocks UNINTERRUPTIBLY (D-state), and subprocess.run(timeout=) can't kill a D-state child — so the front's uvicorn startup hung forever ("Waiting for application startup"), taking down the healthy NVIDIA engine too (502 everywhere). Add _call_with_timeout: runs the probe in a daemon thread and STOPS WAITING after a wall-clock timeout, leaving the un-killable call orphaned instead of blocking. Wrap vulkan_devices() (12s) so startup always completes, and _amd_stats() (6s) so a hung Radeon can't freeze the health/thermal poll threads that also monitor the healthy card. A wedged engine now degrades to "that engine down", not "everything down". Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The per-engine concurrency-overrides table on the Settings page already exposes max_parallel_requests per engine; add a "GPU power lock" column (dpm level select) so dpm_force_performance_level_overrides is editable too. Backend: GET returns the map, POST saves it via a level-validating sanitizer. Takes effect on engine restart. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Devuan's udev rejects the 'bind' trigger action, so the udev-rule path can't reliably apply the DPM lock at boot. sysvinit's /etc/rc.local runs race-free after amdgpu binds — this root-owned helper, called from rc.local, sets power_dpm_force_performance_level=high on every AMD card by PCI vendor id. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
An ATTR{}= assignment on the add event races the amdgpu probe (the power_dpm attribute doesn't exist yet), so it silently no-ops at boot (observed: level stayed 'auto'). Fire on the bind action — driver bound and sysfs attrs created — and write via RUN with %p devpath. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Devuan (sysvinit/OpenRC, no systemd) — a udev rule is the init-agnostic way to make power_dpm_force_performance_level=high stick. Matches by DRIVER==amdgpu so it survives DRM card renumbering across GPU resets. Complements the in-container best-effort apply (unprivileged, can't write root sysfs) shipped in 0.1.53. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
New server.dpm_force_performance_level_overrides (engine name → level, e.g. {"radeon":"high"}). At engine startup the engine writes the level to every amdgpu card's power_dpm_force_performance_level — locking a Polaris/GCN card to fixed top clocks avoids the DPM power-state transitions that hang these cards under sustained Vulkan compute (current default is 'auto', the hang-prone mode). Card is matched by PCI vendor id (0x1002), robust to DRM card renumbering across resets. Best-effort: an unprivileged container logs the exact host command on PermissionError. Pairs with two config-only stability levers applied for the radeon: n_ctx reduction (qwen3 1024→512, gme 1536→1024) to keep all three embedders in real VRAM with headroom (no GTT-over-PCIe spill during compute — another Polaris hang trigger), and max_parallel_requests override {"radeon":1} to serialize Vulkan submissions. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A vision-capable GGUF model (gemma-4-26B + mmproj) replied "none" to real-estate photos because the client sends AVIF, and llama.cpp's mtmd chat handler decodes data-URIs with stb_image, which has no AVIF/WebP support — the model silently received no image. _normalize_vision_content now transcodes any non-stb format to PNG via PIL (which has libavif) before handing it to the handler. The radeon GME embedder was unaffected (it already decodes via PIL). Text-only models (gemma-2-9b, no mmproj) are untouched — the image is still flattened, since they have no vision tower. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The 0.1.50 guard only covered _read_gpu_temp_uncached's FIRST probe path; its rocm/psutil fallbacks and gpu_eval() (which reads engine_gpu_stats directly) still saw the raw 511°C and cooling-waited a healthy card forever. Filter at the real source — the amdgpu sysfs temp1_input read in gpu_detect — so temp is None for any ≥150°C reading and NO downstream reader can act on it; plus a final chokepoint in read_gpu_temp(). Co-Authored-By:
-
- 23 Jul, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authored
After a GPU reset the amdgpu SMU can return a constant garbage reading (observed: 511°C = 0x1FF invalid-ADC, with nonsense voltage/fan values). The thermal supervisor believed it, paused the radeon engine and escalated to SIGSTOP — freezing a healthy card forever. Both readers (engine-side thermal.py and the front's supervisor loop) now discard readings ≥150°C with a one-time warning naming the broken sensor; thermal protection for that card resumes when it reads sane values. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Engines run in their own sessions, so supervisord's killasgroup cannot reach them; teardown relies on the front's shutdown handler + the engines' PDEATHSIG. That chain leaked a full front+engine tree across a restart — two trees then fought over one GPU (phantom VRAM, evictions freeing nothing, duplicated model workers, requests served by one tree while logs came from the other). The launcher now pkills any surviving coderai-front/engine/dinov2-embed processes before starting: at that point none can legitimately exist. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Multiple loaded models must each have their own queue — one model's backlog or GPU ownership must never block requests for another model that is loaded and idle: - queue admission is per model (is_full_for): 429 only when the REQUEST'S model already has queue_max_size waiters (global 4× backstop bounds memory); previously ONE hot model's backlog 429'd every other model. - GpuSwapGate: requests for a model already RESIDENT on the engine bypass the ownership gate — no swap is needed and its own per-model queue governs concurrency. The gate now serializes only requests that would actually trigger a model swap (its real purpose). Three co-resident embedders can each serve a request in parallel. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
An engine crash re-parented its dinov2-embed child (still holding VRAM) and the respawned engine stacked a second copy — the child now gets PR_SET_PDEATHSIG (dies with the engine) and stale orphans for the same model are reaped before spawn. Engines-card footprints: a model loaded onto an already-full card measures a ~0 VRAM delta and reported "0.01 GB" — report max(measured, config estimate) instead. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The tooltip matched loaded_info entries (raw engine keys) against canonicalized display names client-side and mostly missed — footprints now canonicalize server-side with the same mapping as the model list, keyed by canonical id, so every loaded model shows its VRAM (+RAM when offloading). llama-vl: bound mtmd image_max_tokens (config `image_max_tokens`, default 1024) — large photos otherwise expand to thousands of vision tokens whose transient compute buffer evicts co-resident models on a small card. Co-Authored-By:
-