- 22 Jun, 2026 34 commits
-
-
Stefy Lanza (nextime / spora ) authored- /admin/api/system-stats now built on the front (psutil for CPU/RAM, torch-free gpu_detect for GPU/VRAM) instead of poll-proxying to the GIL-busy engine, so the Tasks header tiles stay live during generation. - Removed from codai/admin/routes.py (engine), now all owned by the front: the dead page-GET routes (login/admin/models/tokens/users/tasks/chat/settings/ archive/change-password GET — the front renders these), api_gpu_stats and api_system_stats. Kept the auth POSTs (login/logout/change-password), _tmpl, _do_task_cancel, build_settings_dict and api_save_settings. Verified: front _build_system_stats returns cpu/ram/gpu/vram; routes.py imports. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
-
Stefy Lanza (nextime / spora ) authored
With --single-process gone, the engine only ever runs behind the front, which serves these before its catch-all — so the engine copies were dead code. Removed from codai/admin/routes.py: - api_status (GET /admin/api/status) — front builds status from its own state - api_list_tokens/api_create_token/api_delete_token — front admin_data (auth.json) - api_create_user/api_delete_user — front admin_data (auth.json) - api_get_settings (GET) — front serves via the shared build_settings_dict Kept: build_settings_dict (imported by the front), api_save_settings (POST — the engine still persists/applies config), api_list_models, and the auth POSTs (login/logout/change-password) the front still proxies. The engine now does generation + model management only; the front owns pages, stats and these data reads. Verified routes.py imports and compiles. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai now always boots as the front proxy + supervised engine subprocess(es); the one-process "UI/API and model work in the same process" mode is gone. This unblocks removing the engine's now-dead UI/stats/data handlers (the engine only ever runs as --engine-only behind the front). Removed: the --single-process CLI flag (cli.py), config.server.single_process (config.py + save), the proc-title branch and the run-mode branch (main.py), and the arg from the engine spawn filter (engine_supervisor.py). Existing config.json files with a stale single_process key are tolerated (_dc ignores unknown keys). Verified: argparse rejects --single-process; --engine-only still parses. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per "always use the front when it can serve it; the engine only generates and gets stats from the front": /admin/api/status no longer calls the engine at all. It's built entirely from front state — registry (loaded models, in-flight), config (backend/load_mode), models.json (enabled models/aliases), live torch-free GPU stats (VRAM) and psutil (system RAM). Recent activity is now tracked on the front (it relays every request) via a ring buffer recorded in both the direct proxy and broker dispatch paths, so the Overview activity table is front-native too. Removed the now-dead _overlay_engine_totals/_cached_status/_status_cache. Engine-side removal of the now-unused handlers (api_status, tokens/users CRUD, api_get_settings) is pending a decision on the legacy --single-process mode, which still serves the UI from the engine app. Verified: _native_status() returns vram+ram+recent_activity via standalone test. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Overview's only call (/admin/api/status) was proxied to the engine, so during a generation (engine GIL-busy) it timed out and the dashboard showed nothing. The front now builds status from its OWN state — registry (loaded models, per- engine in-flight), config (backend, load_mode), models.json (enabled models + aliases) and live torch-free GPU stats (VRAM summed across cards) — with no engine round-trip. It's used whenever the engine can't answer, and the front skips the engine entirely when the primary is mid-generation (inflight > 0) so Overview is instant instead of waiting out the timeout. Engine-only fields (recent_activity, RAM watcher) are carried over from the last good status; when the engine is idle the richer engine status is still used. Verified: _native_status() builds a complete payload (status/backend/load_mode/ enabled_models/vram/requests) via TestClient/standalone. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Root-cause finding (Path B): the engine event loop is genuinely freed during GGUF generation (work runs via asyncio.to_thread; llama_decode releases the GIL), so it's not a hard-blocked loop. But under CONTINUOUS generation the per-token Python work in llama-cpp-python keeps re-grabbing the GIL and starves the engine's own API handlers (sync ones especially) — an intrinsic ceiling that can't be cleanly removed while generating. So per "B first, A fallback", falling back to A: serve page data from the front (a separate process the GIL can't touch). This commit: /admin/api/settings GET is now served by the front. api_get_settings was refactored into a pure build_settings_dict(config, gpu_cards) shared by the engine handler and the front (front holds the same Config), so both return an identical payload. Front uses its torch-free gpu_detect.gpu_cards(). POST (save) still falls through to the engine, which persists + applies it. Verified: build_settings_dict standalone + front /admin/api/settings via TestClient. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Pages opened from the front but their DATA was still proxied to the engine, so while a generation ran (engine event loop GIL-busy) Tokens/Users/etc. showed no data. Tokens and users are backed purely by auth.json under config_dir, which the front already has — so it can answer them from disk with zero engine round-trip. New codai/frontproxy/admin_data.py (register_admin_data): front handlers for GET/POST/DELETE /admin/api/tokens and /admin/api/users, mirroring the engine handlers exactly (same shapes/status codes), with sessions validated locally against the shared secret and admin gating. Registered before the catch-all so they're served here, not forwarded. The engine still reads auth.json fresh for bearer/session auth, so tokens created via the front work for inference. This fully unblocks the Tokens and Users pages during generation. Settings/ status/models-list are the next candidates (config + models.json + registry are all local to the front too). Verified via TestClient: full CRUD, 401/403 gating, and pass-through isolation. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The Models page fired ~10 /admin/api reads on load. Browsers cap ~6 connections per host, so during a generation (engine event loop GIL-busy) the proxied reads pile up, saturate the limit and freeze the whole page behind them. Two fixes: 1. New front endpoint POST /admin/api/batch: fans out several engine GET reads CONCURRENTLY server-side (front's own un-capped pool), each individually bounded, and returns them in one response. A slow/blocked sub-call returns an error marker without holding up the rest. base.html gains bootstrapPage() + pageFetch(): the page batch-loads its first-paint reads in ONE request, and the existing loaders transparently serve from the bundle (falling back to a normal fetch on miss). models.html now collapses settings/cache-stats/cached-models/ models/quantize-status/downloads into a single batched call, plus an in-flight guard on the downloads poller so ticks can't stack up. 2. Bound /admin/api/* GET proxying (catch-all) with a finite read timeout + graceful 503 fallback, so a busy engine can never hang a page forever (HF hub lookups and streaming endpoints keep the unbounded path). Verified: batch unit test (concurrent aggregation, error markers, path filtering, auth) and node syntax-check of the bootstrap script. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Move request queuing from the engine to the front so the engine does only generation. New codai/frontproxy/reqqueue.py FrontQueue: a per-model concurrency gate sized to the model's max_instances (per-model from models.json, else the global models.max_model_instances). A text-generation request acquires a slot before dispatch; when all slots are busy it waits (FIFO) and shows as "queued" with its position on the Tasks page; beyond server.queue_max_size waiting it gets HTTP 503. A client disconnect while queued cancels the wait and drops it from the queue with no slot leak. Wired into both dispatch paths in app.py (direct proxy + broker route) so direct and brokered requests share one queue, scoped to text generation only (images/ audio/embeddings pass through unqueued). Slots are released on stream completion, send failure, or disconnect. Queue key is the canonical model id so all aliases/ paths of a model share one gate. Front never over-subscribes the engine, so the engine's own queue stays a safety net that effectively never fills. The engine-side QueueManager + KV-affinity instance pool are intentionally left in place for now (trimming them is a follow-up once this is verified live). Verified: FrontQueue unit tests (capacity, FIFO hand-off, QueueFull, cancel cleanup, position, capacity>1) and a FrontProxy integration test (capacity resolution, alias-key collapsing, queued task surfacing). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The front reverse-proxied every page navigation to the primary engine, so opening any page while that engine was mid-generation (event loop busy) made the whole UI hang until generation finished. Now the front renders the UI pages itself (dashboard, models, tokens, users, tasks, settings, archive, chat, login, change-password) and serves the admin static assets + favicon locally. Sessions are validated locally: the cookie is HMAC-signed with the secret under config_dir (shared via auth.json), so the front authenticates it with no round-trip to a busy engine — and matches the port-derived session_<port> cookie name by signature, not by exact name. Mutating auth actions (login/logout/change-password POST) and all /admin/api/* data calls still fall through to the catch-all proxy and reach the engine. New: codai/frontproxy/ui_pages.py (register_ui_pages), wired in build_app before the catch-all. Verified via TestClient: local render, unauth->login redirect, authed render, static, and POST/logout pass-through. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
When models.json changes, the front recomputed the per-engine assignment and immediately POSTed /internal/reload-config to every engine. If an engine was mid-generation it was GIL-busy, so the push blocked and timed out, stalling the front's poll thread ("reload-config push to 'nvidia' failed: timed out"). Now the front updates its local router instantly but queues the reload push per engine, and _flush_pending_reloads() delivers it only once that engine's inflight count drops to 0 (generation finished). Failed pushes are requeued. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
On the Tasks page a running generation's token count (step) appeared frozen and only refreshed every several seconds, while the speed kept moving. Cause: the primary engine's task list is frequently served from the front's last-good cache (when the engine is briefly busy), so the task's step froze — but rate, a cumulative average over growing elapsed time, kept drifting, so it looked like "only the speed updates". The front is itself relaying the SSE stream and counting tokens per request (engine.active, refreshed ~2×/s by the streaming proxy), but that live count was deduped away in favor of the stale real task. Fix: in _merge_engine_tasks, overlay the live in-flight step/rate (from engine.active) onto the matching running task before the synthetic-task dedup, so the count stays live regardless of how stale the engine's own task snapshot is. Match on (engine, model); fall back to the engine's sole live generation when the client's model string (alias/path) differs from the task's resolved name. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
gemma-4 GGUF finetunes don't emit a clean EOS after a tool call — they continue and hallucinate a fake tool result (<tool_call|><|tool_response>response:NAME{…} … <turn|>), wasting tokens on a result llama.cpp never runs. Add a per-model stop augmentation (_augment_model_stops): for gemma, inject <|tool_response>, <tool_response|>, <turn|> and the real <end_of_turn> as stop strings (merged with any client-supplied stops). llama.cpp matches and trims stop strings, so the call ends cleanly and the fake-response tail is never generated. Applied to both chat methods and the raw completion paths; non-gemma models are unaffected. Complements 5135f8f8 (parser repair of <|"|> quotes and malformed markers). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
gemma-4 GGUF finetunes emit a corrupted tool-call scheme that broke both extraction and display, e.g.: <|tool_call>call:bash{command:<|"|>wc -l README.md<|"|>}<tool_call|><|tool_response>… Three distinct corruptions, all now handled by normalize_gemma_tool_tokens(): - <|"|> is the model's stand-in for a " quote — it leaked verbatim into string args, so a bash command became <|"|>wc -l<|"|> and failed with "syntax error near unexpected token |". Restored to a real ". - malformed open/close markers <|tool_call> / <tool_call|> (vs the canonical <|tool_call|>) — stripped, so the native call:NAME{…} body parses and the markers don't leak as text. The canonical <|tool_call|> / <|tool_call_end|> are deliberately left intact so Phi-style parsing is unaffected. - a hallucinated <|tool_response>…/<turn|> tail (the model fakes the tool result) — stripped, and strip_tool_calls_from_content now also drops response:NAME{…} spans so the fake result never reaches the user or gets executed. Wired into ModelParserAdapter + ToolCallParser (extract & strip) and cleanup_control_tokens, covering the streaming, non-streaming and fallback paths. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The task page's engine tiles vanished whenever the primary (NVIDIA) engine was busy generating. /admin/api/engines gated on is_admin(), which validates the session by round-tripping to the primary engine's /admin/api/status — that call times out while the engine is saturated, so the endpoint 401s and the page hides the whole engines card. But the tile DATA (name/health/VRAM/loaded models) comes entirely from the front's own cached registry, not the engines. Fix: gate /admin/api/engines on a light credential check (session cookie or bearer present), exactly as /admin/api/gpu-stats already does and for the same reason. Extracted that check into Front._has_cred() and reused it for both. The mutating /admin/api/engines/{id}/restart keeps full is_admin validation. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Problem 1 — the per-card cap was read and printed ("VRAM capped at 5.0 GB on amd:...") but the PERFORMANCE split branch ignored it: it pushed the whole overflow onto the slow card (_overflow * adj[i]/others == all of it when there's one other card), so a 5 GB-capped RX 580 got ~47% of the layers and filled up. Fix: treat each card's (capped) free VRAM as an absolute ceiling — fill the fast lead card first up to its cap, then each other card up to ITS cap; the GPU budget is bounded by sum(caps) and the remainder spills to CPU via the n_gpu_layers auto-offload. RX 580 @5 GB now yields tensor_split [0.805, 0.195] (was [0.527, 0.473]). The VRAM (proportional) strategy already used the capped values. Problem 2 — process naming: extract _set_proc_title() and ALSO re-assert it right before the engine binds uvicorn (defensive), with CODERAI_ENGINE_BACKEND as a fallback name. The per-engine CODERAI_ENGINE_NAME is verified distinct, so this guarantees coderai-front / coderai-nvidia / coderai-radeon regardless of timing. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The global "secondary card VRAM cap" was a single number applied to every non-lead card. Replace it with a per-CARD map so each physical card on the machine can be capped independently — e.g. RX 580 → 4 GB, RTX 3090 → 20 GB. - gpu_detect: card_key() (nvidia:<uuid> / amd:<pci>) + gpu_cards() enumerating every physical GPU. Stable keys the front and engines both compute identically. - vulkan: _per_device_card_key() (parallel to _per_device_free_vram_gb, same llama.cpp order) + _resolve_device_caps() combining the per-model secondary cap (non-main devices) with the global per-card map, lowest wins per device. Applied in both the auto-offload pool and the auto-split ratio. - config: OffloadConfig.split_card_caps_gb {key: gb}; per-model split_secondary_cap_gb stays a single value (one cap is enough per model). - manager: pass _global_card_caps to the engine; keep per-model scalar. - settings UI: render one cap input per detected card (name + vendor), keyed by the stable card key; GET returns gpu_cards + saved caps, POST saves the map. Applies on the next engine (re)load (global args are read at engine startup), matching the previous global cap's behavior. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Root cause of "engine 'nvidia' not responding" during generation: in --debug mode the streaming response generator print()s the ENTIRE generated text (raw + repr), and a prompt-echoing/runaway generation makes that multi-MB. The engine writes stdout to a pipe drained by the front; once the pipe fills, that synchronous print() — running on the event-loop thread inside the SSE generator — BLOCKS, freezing the engine so health polls time out and the front flips it to "not responding" (seen ~3s into each debug flood in debug.log). - api/text.py: add _clip_for_log() and bound every large debug/dump print (generated_text, second_pass/reasoning/final text, formatted_response, extracted_tool_calls) to ~4KB head+tail. Shared layer, so both engines covered. - engine_supervisor: enlarge each engine's stdout pipe to 1 MiB (F_SETPIPE_SZ) so bursts can't stall the event loop even if the pump lags briefly. Also: name processes by role — coderai-front for the front, coderai-<name> for each engine (CODERAI_ENGINE_NAME passed at spawn), coderai for single-process. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Add a cap on how much VRAM the auto-split may place on EACH secondary (non-main) card, so a slow second GPU (e.g. RX 580) stays lightly loaded and bottlenecks throughput less — the remainder stays on the fast card or spills to CPU. - config offload.split_secondary_cap_gb: global default, persisted + pushed live to every engine's global_args (each engine reads config at startup, so it's global). - per-model split_secondary_cap_gb overrides — but only to TIGHTEN: the effective cap is min(global, per-model) of whichever are set; a per-model value higher than the global is ignored. - applied in vulkan auto-split: caps each non-main device's free VRAM in the ratio (both vram & performance strategies), and in the auto-offload pool so the excess proactively spills to CPU instead of only reacting to an OOM. - UI: model config modal "Secondary card VRAM cap" + global Settings field. - added to _LOAD_AFFECTING so changes re-apply live on the next request. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The supervisor health-polls /internal/engine-state every couple seconds. While the engine is GIL-busy generating, that poll can't be answered in time and the engine was flipped to healthy=False — flapping out of the UI/routing mid-generation even though it's perfectly alive. Now a poll timeout only downgrades health when the PROCESS IS GONE (true death, already caught by the restart check); a timeout with the process alive keeps the last-known-healthy state. Also bump proxy_status_timeout 2s→4s so transient GIL contention doesn't trip it. (Pairs with the engine-state VRAM cache that removed the per-poll CUDA call.) Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two fixes for the performance split OOM loop at large context: 1) The performance "fits on the fast card?" check used expected_vram_gb (WEIGHTS only), ignoring the KV cache + compute buffers. A ~20 GB model at 178k ctx looked like it fit a 24 GB card → tensor_split=[1.0,0.0] → llama_context creation OOM'd (KV didn't fit) in a retry loop. Add a context-scaled KV/compute estimate (~n_ctx/16000 + 1.5 GB, calibrated to observed ~10 GB at 178k) to the footprint before deciding what fits — tight enough not to over-load the slow card. 2) On llama_context creation failure, progressively offload to CPU: move ~20% more layers to CPU and retry, up to 5 times, then fail. A slow CPU-spilled load beats a hard failure / crash loop. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The previous commit (be9950a5) left a try: with no except/finally in the SSE throughput counter, breaking import of the front. Remove the stray try (the BackgroundTask _release already closes the upstream response). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The Tasks page lost the it/s indicator under load: when the engine is too busy to report its own task, the front showed a synthetic in-flight task with rate 0. Now the front counts SSE "data:" events (~one token each for chat/text) as it relays the stream and publishes step + rate (tokens/s, refreshed ~2×/s) onto the in-flight metadata, so the synthesized task shows a live it/s even while the engine can't answer its own /admin/api/tasks. Only for streaming 200 responses; the engine's real task (with its own rate) still wins via the (engine,model) dedup when reachable. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Harden the max-instances guard against the concurrent case: request_model now resolves a GGUF text model to its absolute .gguf path (via _resolve_local_gguf) before building model_key, so every name form — bare id, basename.gguf, full path, or alias (e.g. "lisa") — collapses to ONE model_key. That gives a single self.models entry and a single instance pool, so two simultaneous first-requests under different forms converge on the same pool and the second queues instead of loading a second instance. Complements the earlier already-loaded fuzzy match (which only covered the sequential case). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
request_model's "already loaded?" fuzzy match compared basenames literally, so a bare request "gemma-…-Q4_0" missed the loaded "/AI/…/gemma-…-Q4_0.gguf" (the .gguf suffix differed) and a SECOND instance was loaded even with max_model_instances=1. A "lisa" alias resolved to the full path and matched, but the bare name didn't — so two requests for the same model (one via alias, one via name) ran as two instances instead of queueing on one. Normalize the .gguf extension (and compare short basenames) when matching, so every name form of one model maps to the same loaded instance and the second request queues. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The front health-polls /internal/engine-state every ~2s. It called torch.cuda.mem_get_info + get_device_name on EVERY poll, touching the CUDA context, which can serialize behind the running forward pass and stall the handler past the poll timeout — flipping a busy engine to "not responding". Cache the VRAM snapshot (4s TTL) and device names (permanent), so mid-generation polls return instantly from cache instead of blocking on CUDA. (llama-cpp-python 0.3.30 uses ctypes, which already releases the GIL during eval, so the compute itself wasn't the blocker.) Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The auto split is VRAM-proportional, which hands a big share to a slow second GPU (e.g. RX 580) so it bottlenecks throughput (Radeon ~100%, 3090 ~30%). Add a "performance" strategy that fills the fast lead card (main_gpu) first and spills only the overflow to the slower card(s), using the model's expected size — so the weak GPU holds the fewest layers (and none at all if the model fits on the lead card). "vram" (default) keeps the capacity-maximizing behavior. - config offload.split_strategy ("vram"|"performance") global default, persisted + pushed to live global_args; per-model split_strategy overrides it. - plumbed via build_runtime_kwargs + manager _cfg_or_global into vulkan auto-split. - model config modal: "Split strategy" select shown with the split options; global Settings: default split strategy select. - split_strategy added to the live-reload (_LOAD_AFFECTING) set so changing it re-applies on the next request without a restart. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
#1 Front-side in-flight task tracking: Engine.enter_request now records per-request metadata (model/kind/path/started_at) in engine.active; _merge_engine_tasks injects synthetic "running" task entries for in-flight requests not already reported by the engine (deduped by engine+model). Both the direct proxy and the broker route register/clear it. So the Tasks page shows work the front dispatched even when the engine is too GIL-busy generating to answer its own /admin/api/tasks. (Combines with the last-good cache.) #3 Thermal stopping-criteria cadence: the per-token callback held the GIL every token, starving admin handlers. Now it does real work every 50 tokens for text, scaling to every 100 when generation is fast (>50 tok/s) — far less GIL contention, negligible thermal-drift risk between checks. #2 (no change needed): generation already runs OFF the event loop via _aiter_blocking (asyncio.to_thread per token step) for streaming and asyncio.to_thread(manager.generate) for non-streaming. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The Tasks page polls the primary engine via the short-timeout client. While the primary is GIL-busy generating it can't answer, so poll() hit its except branch and returned the primary's tasks as EMPTY — the running generation vanished from the page under load. Cache the primary's last-good task list (120s TTL) and reuse it (marked stale) when the live poll fails, so an in-flight generation stays visible. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three related fixes for cross-GPU split of a vision model at large context: 1) gpu_detect: when exposing GPUs for cross-split (allow_cross), include only REAL hardware Vulkan devices (nvidia/amd/intel) and EXCLUDE software rasterizers (llvmpipe/lavapipe/virtio = vendor "other"). They're CPU-backed with no real VRAM, slower than native CPU offload, and their presence skewed the device list so the tensor_split ratio didn't line up with the actual cards (and layers could land on a fake GPU). 2) vulkan auto tensor_split now reserves the mmproj's size + compute margin on main_gpu (the projector always loads there), so the proportional split doesn't fill main_gpu to the brim and abort when CLIP can't allocate (GGML_ASSERT(buffer) failed). 3) auto-offload (n_gpu_layers sizing) subtracts the same mmproj reserve from the POOLED free VRAM, so when model+KV+projector exceed BOTH cards combined it reduces GPU layers and spills to CPU gracefully instead of crashing. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
gpu/live: honor per-model gpu_split (was offloading to CPU); apply model config live on any load-affecting change Two fixes: 1) build_runtime_kwargs never promoted gpu_split/tensor_split to top-level kwargs (only into _raw_cfg), but the manager reads config['gpu_split'] via _cfg_or_global (which doesn't consult _raw_cfg). So a model set to "Split — <card> first" loaded gpu_split=False → confined to one card → its big context spilled to CPU instead of distributing to the second GPU. Now promoted, so the split is honored and the GGUF auto-offload sizes against the POOLED VRAM across both cards. 2) apply_model_entry_live only evicted a loaded model when *acceleration* changed. Generalize to ANY load-affecting field (n_ctx, n_gpu_layers, gpu_split, tensor_split, cache types, kv_offload, n_batch/ubatch/seq_max, flash, quant, engine pin, vae/precision, …): if it changed and the model is loaded, evict it so the next request reloads with the new config — config changes apply immediately, no server restart. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
- Model list: a model with cross-GPU split now tags as "<lead> ⇄split
📌 " (with a tooltip naming the lead card + ratio/auto), so the list makes "loads here, spills onto the other GPU(s)" visible instead of looking like a plain pin. - gpu_detect._amd_stats reported VRAM in decimal GB (bytes/1e9) while _nvidia_stats used GiB (MiB/1024). That made the two cards ~7% out of step in the dashboard and in any cross-card free/total sum. AMD now reports GiB to match, so free-VRAM is consistent and comparable across both cards. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The per-model engine selector was confusing: "Auto" implied single-card (correct) but the only split choice was a generic "All GPUs". Replace with hardware-named, ordered choices: - "Auto (single GPU — by capability / free VRAM)" → one card (unchanged) - "<engine> (single card)" → pin to that card - "Split — <engine> first" → pool across all GPUs with that engine's card leading (main_gpu / larger share). Generated per engine, so on a 3090+Radeon box you get "Split — nvidia first" and "Split — radeon first". Saving a split option sets gpu_split + pins engine=<lead>; the weight-ratio field (blank = auto by free VRAM) shows only for a split choice. Make per-model split actually function: - engine_supervisor: expose every GPU to engines (allow_cross) when the global toggle OR ANY model entry has gpu_split — otherwise the lead engine couldn't see the foreign card. - vulkan.py: when an engine sees BOTH backends' cards but THIS model isn't a split, confine it to its own backend (zero the foreign devices in tensor_split) so it can't accidentally spread — while same-backend multi-card split still works. Split models pool across all visible devices (lead card = device 0 = main_gpu). Co-Authored-By:
-
- 21 Jun, 2026 6 commits
-
-
Stefy Lanza (nextime / spora ) authored
Two distinct "config goes back to what it was after restart" bugs: 1) save_config dropped broker websocket_path + websocket_ping_interval (and the new offload gpu_split/tensor_split). The settings UI set them in memory, but save_config never serialized them, so the next load() fell back to dataclass defaults — the broker protocol/websocket settings reverted every boot even with a fully persistent config dir. Now serialized. 2) Per-model fields (e.g. n_ctx) reverted via a multi-process models.json write race: every engine loads models.json at its own boot, so a SECONDARY engine's in-memory models_data is stale w.r.t. a later UI edit on the primary. When that secondary engine auto-persisted measured_vram_gb it called save_models(), which dumps its whole stale state and clobbered the edit. Added ConfigManager.persist_model_field(): a single-field, atomic read-modify-write that re-reads models.json from disk, updates only the one field, and refreshes the in-memory copy. record_vram_delta now uses it, so VRAM measurement can never revert user model config. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Per-model (model config modal): the "Engine / card" selector gains an "All GPUs (split across every backend)" option. Choosing it sets gpu_split and reveals a "Weight distribution" field (tensor_split, e.g. 0.8,0.2); blank = auto. A real engine name still pins as before. Global (Settings): "Split models across all GPUs" checkbox + default weight distribution field, persisted to offload.gpu_split / offload.tensor_split and pushed to live global_args. Per-model "All GPUs" overrides the global default. api: settings GET/POST now expose+accept offload.gpu_split / tensor_split. backend: when gpu_split is on and no ratio is given, auto-derive tensor_split proportional to each device's FREE VRAM (new _per_device_free_vram_gb, llama.cpp device order: CUDA first, then Vulkan/AMD) — e.g. 24 GB 3090 + 8 GB RX 580 -> ~0.75/0.25 — so the bigger card carries more without the user computing it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A multi-stage build leaves untagged <none> intermediate layers behind that pile up across rebuilds. After the build (and the dist export, which still needs the final image), run `docker image prune -f` to drop them. Only DANGLING images are removed — the final image, the pulled CUDA base, and any other tagged images are kept. Default on; disable with --no-prune. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Gemma emits tool calls as text — <|tool_call>call:NAME{…} — which the first-pass streaming loop forwarded straight to the client as assistant content. So the call showed up as a visible message and only got parsed/executed at end-of-stream. Add _ToolCallStreamGate: as chunks stream, emit visible content only up to the first tool-call marker (gemma's <|tool_call>/<|tool_response> plus the common tag formats), holding back a small tail so a marker split across chunk boundaries isn't leaked; once a marker appears, emit nothing further. The full text still accumulates for the existing end-of-stream tool extraction (GemmaParser), which emits a proper tool_calls delta. The second pass is already buffered+extracted, so it never leaked. Also add gemma's "<|tool_call>" to the reasoning/tool split pattern lists. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
gpu-stats fell through to the generic proxy, which forwards via the long-timeout client to the primary engine. When that engine is saturated generating (sync llama.cpp holding its event loop), the request blocked → the dashboard's temp/GPU polling hung → the whole web UI went unresponsive and the task page stopped showing temperatures. Serve it from the front instead, using the already torch-free gpu_detect.gpu_stats() (nvidia-smi + AMD sysfs, reports every card regardless of CUDA_VISIBLE_DEVICES), run in a thread so the subprocess never blocks the front's event loop, and registered before the catch-all so it's not proxied to a busy engine. Light auth (session cookie / bearer presence) since GPU telemetry is low-sensitivity and full session validation lives on the engine — which is the component we're decoupling from. Note: the thermal THROTTLE (pausing a generation when hot) stays in the engine — it must, to pause that engine's own forward pass — but its stats are now front-served. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Broker (AISBF) chat requests were rejected by the engine with 401 "Invalid API key" — they're authenticated at the aisbf layer (registration token) and carry no end-user Bearer, but the engine's BearerAuthMiddleware still demanded one, so every brokered request came back as a constant ~472-byte error and never ran inference. Fix (token auto-managed, secure): - Front marks broker-relayed requests with x-coderai-broker-authed = the internal shared token (which the engine already trusts via CODERAI_INTERNAL_TOKEN). Added that header to _DROP_REQ so a client-supplied copy is always stripped first — unforgeable from outside. - Engine BearerAuthMiddleware skips the Bearer check when x-coderai-broker-authed matches CODERAI_INTERNAL_TOKEN. This is NOT the plain internal token (which rides on every front→engine request, direct included), so DIRECT API requests still require a real Bearer token. Co-Authored-By:
-