- 24 Jul, 2026 9 commits
-
-
Stefy Lanza (nextime / spora ) authoredNew server.engine_env_overrides (engine name → {VAR: val}), merged into the engine's process env at spawn. Lets low-level driver knobs be set without hardcoding — set radeon → RADV_DEBUG=syncshaders to serialize RADV shader dispatch and test whether the Polaris compute-ring async race (ring comp_x.y.z timeout) is what wedges the RX 580 under sustained Vulkan compute. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
-
Stefy Lanza (nextime / spora ) authored
At front startup _build_engines runs vulkaninfo to enumerate GPUs; on a wedged Polaris card that process blocks UNINTERRUPTIBLY (D-state), and subprocess.run(timeout=) can't kill a D-state child — so the front's uvicorn startup hung forever ("Waiting for application startup"), taking down the healthy NVIDIA engine too (502 everywhere). Add _call_with_timeout: runs the probe in a daemon thread and STOPS WAITING after a wall-clock timeout, leaving the un-killable call orphaned instead of blocking. Wrap vulkan_devices() (12s) so startup always completes, and _amd_stats() (6s) so a hung Radeon can't freeze the health/thermal poll threads that also monitor the healthy card. A wedged engine now degrades to "that engine down", not "everything down". Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The per-engine concurrency-overrides table on the Settings page already exposes max_parallel_requests per engine; add a "GPU power lock" column (dpm level select) so dpm_force_performance_level_overrides is editable too. Backend: GET returns the map, POST saves it via a level-validating sanitizer. Takes effect on engine restart. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Devuan's udev rejects the 'bind' trigger action, so the udev-rule path can't reliably apply the DPM lock at boot. sysvinit's /etc/rc.local runs race-free after amdgpu binds — this root-owned helper, called from rc.local, sets power_dpm_force_performance_level=high on every AMD card by PCI vendor id. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
An ATTR{}= assignment on the add event races the amdgpu probe (the power_dpm attribute doesn't exist yet), so it silently no-ops at boot (observed: level stayed 'auto'). Fire on the bind action — driver bound and sysfs attrs created — and write via RUN with %p devpath. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Devuan (sysvinit/OpenRC, no systemd) — a udev rule is the init-agnostic way to make power_dpm_force_performance_level=high stick. Matches by DRIVER==amdgpu so it survives DRM card renumbering across GPU resets. Complements the in-container best-effort apply (unprivileged, can't write root sysfs) shipped in 0.1.53. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
New server.dpm_force_performance_level_overrides (engine name → level, e.g. {"radeon":"high"}). At engine startup the engine writes the level to every amdgpu card's power_dpm_force_performance_level — locking a Polaris/GCN card to fixed top clocks avoids the DPM power-state transitions that hang these cards under sustained Vulkan compute (current default is 'auto', the hang-prone mode). Card is matched by PCI vendor id (0x1002), robust to DRM card renumbering across resets. Best-effort: an unprivileged container logs the exact host command on PermissionError. Pairs with two config-only stability levers applied for the radeon: n_ctx reduction (qwen3 1024→512, gme 1536→1024) to keep all three embedders in real VRAM with headroom (no GTT-over-PCIe spill during compute — another Polaris hang trigger), and max_parallel_requests override {"radeon":1} to serialize Vulkan submissions. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A vision-capable GGUF model (gemma-4-26B + mmproj) replied "none" to real-estate photos because the client sends AVIF, and llama.cpp's mtmd chat handler decodes data-URIs with stb_image, which has no AVIF/WebP support — the model silently received no image. _normalize_vision_content now transcodes any non-stb format to PNG via PIL (which has libavif) before handing it to the handler. The radeon GME embedder was unaffected (it already decodes via PIL). Text-only models (gemma-2-9b, no mmproj) are untouched — the image is still flattened, since they have no vision tower. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The 0.1.50 guard only covered _read_gpu_temp_uncached's FIRST probe path; its rocm/psutil fallbacks and gpu_eval() (which reads engine_gpu_stats directly) still saw the raw 511°C and cooling-waited a healthy card forever. Filter at the real source — the amdgpu sysfs temp1_input read in gpu_detect — so temp is None for any ≥150°C reading and NO downstream reader can act on it; plus a final chokepoint in read_gpu_temp(). Co-Authored-By:
-
- 23 Jul, 2026 9 commits
-
-
Stefy Lanza (nextime / spora ) authored
After a GPU reset the amdgpu SMU can return a constant garbage reading (observed: 511°C = 0x1FF invalid-ADC, with nonsense voltage/fan values). The thermal supervisor believed it, paused the radeon engine and escalated to SIGSTOP — freezing a healthy card forever. Both readers (engine-side thermal.py and the front's supervisor loop) now discard readings ≥150°C with a one-time warning naming the broken sensor; thermal protection for that card resumes when it reads sane values. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Engines run in their own sessions, so supervisord's killasgroup cannot reach them; teardown relies on the front's shutdown handler + the engines' PDEATHSIG. That chain leaked a full front+engine tree across a restart — two trees then fought over one GPU (phantom VRAM, evictions freeing nothing, duplicated model workers, requests served by one tree while logs came from the other). The launcher now pkills any surviving coderai-front/engine/dinov2-embed processes before starting: at that point none can legitimately exist. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Multiple loaded models must each have their own queue — one model's backlog or GPU ownership must never block requests for another model that is loaded and idle: - queue admission is per model (is_full_for): 429 only when the REQUEST'S model already has queue_max_size waiters (global 4× backstop bounds memory); previously ONE hot model's backlog 429'd every other model. - GpuSwapGate: requests for a model already RESIDENT on the engine bypass the ownership gate — no swap is needed and its own per-model queue governs concurrency. The gate now serializes only requests that would actually trigger a model swap (its real purpose). Three co-resident embedders can each serve a request in parallel. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
An engine crash re-parented its dinov2-embed child (still holding VRAM) and the respawned engine stacked a second copy — the child now gets PR_SET_PDEATHSIG (dies with the engine) and stale orphans for the same model are reaped before spawn. Engines-card footprints: a model loaded onto an already-full card measures a ~0 VRAM delta and reported "0.01 GB" — report max(measured, config estimate) instead. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The tooltip matched loaded_info entries (raw engine keys) against canonicalized display names client-side and mostly missed — footprints now canonicalize server-side with the same mapping as the model list, keyed by canonical id, so every loaded model shows its VRAM (+RAM when offloading). llama-vl: bound mtmd image_max_tokens (config `image_max_tokens`, default 1024) — large photos otherwise expand to thousands of vision tokens whose transient compute buffer evicts co-resident models on a small card. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Eviction cleanup() closed the llama ctx / killed the dinov2 subprocess while another thread was mid-encode (embeddings hold no pool ref, so _is_key_busy saw idle) — use-after-free segfaults that crash-looped the radeon engine. cleanup() now takes the same per-model lock the embed paths hold, so it waits for the in-flight call. Engines card: the loaded-models hover now lists each model with its measured VRAM footprint and, separately, the host-RAM slice when CPU-offloading. Plumbed engine-state loaded_info → supervisor → registry → engines_list → tooltip. dinov2cpp: build the quantize tool too (dinov2-large-Q8_0 is lossless at cos 0.9973 and drops 612→329 MB). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
dinov2: packaging/dinov2cpp/ builds dinov2.cpp with ggml's Vulkan backend plus a coderai `embed` stdin/stdout server (HF-matching preprocessing: short-side 256 bicubic, center-crop 224; CLS read via ggml_backend_tensor_get so GPU backends work; DINOV2_FORCE_CPU escape hatch). embeddings.py 'dinov2cpp' backend drives one persistent subprocess per model (evict = terminate, reload = respawn); dinov2 GGUF entries route there. Verified on the RX 580: cos 0.9973 vs the HF dinov2-large backend. Tasks page: every task now shows the compute device it runs on and whether the model is CPU-offloading. record_vram_delta records per-model device+offload ("GPU RTX 3090 · CPU offload 12.4 GB"); api_tasks tags each task from that map (engine default as fallback); tasks.html renders a green/amber/gray badge (GPU / offload / CPU). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A GGUF embedder with a configured `mmproj` now loads as a vision-capable 'llama-vl' backend: the mtmd image tower feeds an embeddings context (pooling LAST) under the GME chat prompt — the same scheme as the HF qwenvl backend, so both produce the same vector space (~0.89 cosine agreement at Q4). Runs natively on whatever the llama.cpp build targets (Vulkan on radeon). Text goes through the same prompt+last-token path so both modalities share the GME space; PIL-decoded images are passed as raw RGB bitmaps (mtmd's stb can't read AVIF). Serialized on the per-model lock; cleanup frees the mtmd ctx. Reverts the front's image-reroute-to-HF-sibling: a model pinned to an engine must be served there, not silently moved. The gme mmproj was generated from the local HF checkpoint with llama.cpp's convert_hf_to_gguf.py --mmproj (F16, 1.3 GB). Co-Authored-By:
-
- 22 Jul, 2026 20 commits
-
-
Stefy Lanza (nextime / spora ) authored
llama.cpp's embedding API has no image tower, so `image` requests to a GGUF embedder 400'd. When the same model is also registered as a non-GGUF HF entry (e.g. gme-Qwen2-VL GGUF on radeon + HF on nvidia), the front now rewrites the image request to that sibling — one model name for clients; text stays pinned to the GGUF's engine, images go to the engine that has the vision tower. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two remaining GGUF crashers: (1) detokenize→retokenize is not token-count-stable, so a chunk cut at exactly n_ctx could re-inflate past it inside embed() and trip the GGML_ASSERT — chunks are now shrunk until their re-tokenized length verifiably fits (n_ctx-64 margin); (2) llama.cpp contexts are not thread-safe and parallel indexer requests segfaulted the engine (-11) — embed calls now hold a per-model lock. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Inputs beyond the context window are split into context-sized token windows, each embedded, and combined with a token-count-weighted mean (then normalized) — no content dropped. Single-chunk inputs keep the direct path. Pairs with raising the GGUF embedders' n_ctx in config. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
llama.cpp embedding mode SIGABRTs (GGML_ASSERT out_ids == n_outputs) when an input exceeds the batch size — with the default n_batch=512 any long listing killed the radeon engine (front kept respawning it). Size n_batch/n_ubatch to n_ctx at load, and truncate each input to the context window before embed so no accepted input can trip the assert. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A .gguf path in the embedding registry now loads through llama_cpp (embedding=True, n_ctx/n_gpu_layers from the model config) instead of failing in SentenceTransformer. Per-token outputs are mean-pooled, vectors normalized, matryoshka `dimensions` truncation reuses the shared path. cleanup() closes the llama ctx so eviction frees VRAM. Text-only: image requests against a GGUF embedder keep returning 400. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A bulk indexer's burst of first-requests all saw model_obj None and EACH loaded its own copy of the embedder (observed pool_instances=3 for the 4B model + 6 for DINOv2 ≈ the whole card) — this duplicate stacking, not a single load, was what exhausted VRAM and poisoned the measured footprint (26.8 GB recorded for an 8 GB model, measured across overlapping loads). Loads now take a per-model asyncio lock with a re-check, so one request loads and the rest reuse the loaded model. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Teardown leak: _free_pipeline_vram now records the pipeline's CUDA storage pointers up front and, after the normal teardown+gc, walks gc.get_objects() and nulls any surviving tensors out of their list/dict referrers (the proven pass from the CUDA text backend's cleanup) — the "~22 GB untracked (teardown leak; referenced elsewhere)" VRAM is now reclaimed instead of poisoning every later load until a restart. Names the holders when it fires so future leak sources identify themselves. Tracking: a video pipeline only registers in manager.models AFTER its multi-minute load, so concurrent loads raced the free-VRAM check into mutual OOM. note_loading()/clear_loading() reserve the model's estimate for the whole load window; _get_free_vram_gb subtracts reservations and the orphan-VRAM check counts them (no more false "teardown leak" for an in-progress load). The embeddings OOM-retry waits out active reservations (bounded 300s) before evicting and retrying. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
When the GPU is held by another workload (video job, concurrent load racing request_model's free-VRAM check), the embedding load/encode hit CUDA OOM and 500'd. Now both paths behave like other model loads: run the manager's eviction (which frees idle models and waits for busy ones to go idle) and retry once. No CPU fallback — the request participates in the same VRAM queue as everything else. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
This reverts commit 7908670c.
-
Stefy Lanza (nextime / spora ) authored
Bulk video jobs (township) occupy the whole card while active, so the embedding model 500'd with CUDA OOM on load and CUBLAS alloc failures on encode. Both paths now fall back to CPU — degraded throughput beats hard-failing every request until the GPU frees up. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The final usage chunk joined message contents with str.join, which raised "expected str instance, list found" for multipart (vision) content and surfaced as an error event at the end of otherwise-successful streams. Flatten via _content_to_text instead (all three token-estimate sites). Embeddings limit 120→1200 req/min per IP: bulk indexers legitimately embed at several req/s; keep the limit an abuse guard, not a throttle. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The plain-Jinja fast path in generate_chat_stream renders the template itself and generates via create_completion — which never invokes the multimodal (mmproj) chat handler, so image_url parts were silently dropped and vision models answered as if no image was sent. Streaming vision requests now fall through to create_chat_completion. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Declare turboquant-py in the OCI requirements so the image (and the upgrade delta-sync) installs it — the admin UI offered the 'library' backend but the package was never present. The upstream 0.1.x package only accepts bit_width 1-4, so an explicitly selected library backend with turbo6/turbo8 now raises a precise "supports 1-4 bit" error instead of the misleading "unavailable or failed"; 'auto' still falls back to the builtin quantizer silently. Verified: library turbo4 cos>0.994 vs originals, auto turbo8 builtin fallback intact. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The `dimensions` request param truncated AFTER normalizing, so truncated vectors were no longer unit-norm and downstream cosine-via-dot-product math broke. Truncate then re-normalize (OpenAI semantics), shared by all backends. Verified on MRL-trained Qwen3-Embedding-4B: at 1024/2560 dims retrieval quality is unchanged (0.900 vs 0.181 relevant/unrelated; full-dim was 0.898 vs 0.189). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
docker commit snapshots the container's RUNTIME config, so the upgrade flow's --entrypoint coderai-upgrade override and -e CODERAI_UPGRADE_* vars were being committed into the image — leaving coderai:base with the upgrader as its default ENTRYPOINT, so the server "failed to start" (every new container ran the upgrade script, which as --user also dies on the root-owned app tree). Capture the image's original ENTRYPOINT/CMD before the upgrade run and re-apply them via docker commit --change, and blank the injected env vars. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Support the three planned embedding models end-to-end (all verified on GPU): - 'vision' backend (DINOv2/ViT family): image-only encoders have no text tower or tokenizer, so none of the existing paths could load them. AutoImageProcessor + AutoModel, CLS/pooler embedding, normalized. Text input returns a clear 400. Verified: facebook/dinov2-large, 1024-d, similar-image 0.966 vs 0.750 cross. - 'qwenvl' backend (GME-Qwen2-VL…): the repo's remote code (auto_map + custom_st) pins transformers<4.52 and refuses to load on this stack, so load the plain Qwen2-VL weights through the CONCRETE native classes (which never consult auto_map) and reproduce GME's embedding recipe: their chat prompt format and last-token pooling, fp16, normalized. Shared 1536-d text+image space verified cross-modally (OCR-style test). No trust_remote_code needed. - ST image path: custom multimodal ST modules take {"image": …} dicts, not bare PIL (kept PIL-first for ST's native CLIP wrapper). - Tighten _supports_images: ST's plain Transformer module also exposes a `processor` attribute, which made text-only models (Qwen3-Embedding) look image-capable; now require an actual image-capable processor. Verified: Qwen/Qwen3-Embedding-4B (ST, 2560-d), facebook/dinov2-large (vision, 1024-d), Alibaba-NLP/gme-Qwen2-VL-2B-Instruct (qwenvl, 1536-d cross-modal). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A full `pip install -r requirements.txt` on upgrade is fragile: the repo's requirements list can contain entries that don't cleanly reinstall in the image env (e.g. kokoro-tts has no py3.13 wheel — the image provisions it another way), so reprocessing the whole file fails on a pre-existing line and aborts the upgrade. Instead, diff the normalized requirement specs (old tree vs fetched) and pip-install only the new/changed lines; satisfied/unclean existing lines are left untouched. Empty delta => no pip run. --force no longer implies a full reinstall (it forces the code refresh; deps still follow the delta). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- Add openssh-client to the image (Dockerfile.oci + the update overlay): git ships in the base but the CUDA runtime has no ssh, so `--upgrade --ssh-key` failed with `ssh: not found`. HTTPS upgrades already worked without it. - run_oci.sh: fix the --upgrade banner — `(force)` showed even when force was 0 (`${VAR:+…}` fires on the string "0"), and the auth line printed the ssh key path twice. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Make `coderai-docker --upgrade` a complete in-image update, not just an app-code swap: - Dependency sync: fingerprint the declared deps (requirements.txt + the OCI extras) before/after fetch; when they change (or with --force) re-run pip into /opt/coderai/python using the same effective set as the image build. Native CUDA builds pinned `>=` stay satisfied so pip never clobbers them. A pip failure aborts before commit so a half-upgraded image is never produced. --no-pip (CODERAI_UPGRADE_SKIP_PIP) refreshes code only. - System-file refresh: the launchers + nginx/supervisord configs live outside /opt/coderai/app, so the code swap alone never updated them (including this upgrade script itself). Mirror Dockerfile.update's COPY set, writing each via temp+atomic-rename so replacing the running coderai-upgrade is safe. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Add a runtime code-upgrade path so shipped images can pick up new code without rebuilding or shipping an overlay image. - launcher/coderai-upgrade: in-image upgrader. Shallow-clones the configured ref (default: production branch of the nexlab repo), compares its codai/__init__.py __version__ against the installed one, and rsyncs the fetched tree over /opt/coderai/app when newer (or with --force). Mirrors the build's path exclusions (.git/venv*/dist/…). Exit 0=updated, 10=up-to-date, else error. Auth over HTTPS by default, or SSH when a key path is provided. - run_oci.sh: add --upgrade/--force/--upgrade-ref/--upgrade-repo/--ssh-key. Runs the in-image upgrader in a throwaway container (as root, --entrypoint) and, only when it reports an update, docker-commits the result back onto the SAME image tag — no rebuild, no new overlay. SSH key is bind-mounted read-only. - Dockerfile.update / Dockerfile.oci: install /usr/local/bin/coderai-upgrade. Co-Authored-By:
-
- 21 Jul, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
The session cookie is named session_<port> so two instances on one host don't clobber each other's cookie, but the lightweight front telemetry auth only checked the literal "session" name — 401'ing every front-served status call. Match the engine's get_current_user and accept any session / session_* cookie. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Wire up real image embeddings and make embedding models first-class in the VRAM lifecycle. - api/embeddings.py: detect CLIP/SigLIP dual encoders and drive them through transformers get_text_features/get_image_features so text and images share one projected space (ST path kept for repos shipping a native recipe). request.image is now actually read (URL/data-URI/path/base64), vectors are appended after the text ones, and text-only models return a clear 400. Handle the transformers 5.x pooled-output return shape. - Wrap the loaded model in _EmbeddingModel (unpacks as (backend, model) but exposes cleanup()) and register it via add_model() + record_vram_delta() on the request path, so it is measured, LRU-tracked, and cleanly evicted like every other model type instead of leaking as a bare tuple. - admin: fix the model-load button for embeddings (was routed to the diffusers loader) to use _load_embedding_model, matching the request path. - admin: backfill used_vram_gb after a download completes, since the entry is saved before the weights exist on disk; factor the estimate into a shared _estimate_used_vram_gb helper. A freshly-downloaded CLIP/SigLIP entry now always carries a reasonable estimate so pre-load eviction sizes correctly. Co-Authored-By:
-