- 22 Jun, 2026 16 commits
-
-
Stefy Lanza (nextime / spora ) authoredRoot cause of "engine 'nvidia' not responding" during generation: in --debug mode the streaming response generator print()s the ENTIRE generated text (raw + repr), and a prompt-echoing/runaway generation makes that multi-MB. The engine writes stdout to a pipe drained by the front; once the pipe fills, that synchronous print() — running on the event-loop thread inside the SSE generator — BLOCKS, freezing the engine so health polls time out and the front flips it to "not responding" (seen ~3s into each debug flood in debug.log). - api/text.py: add _clip_for_log() and bound every large debug/dump print (generated_text, second_pass/reasoning/final text, formatted_response, extracted_tool_calls) to ~4KB head+tail. Shared layer, so both engines covered. - engine_supervisor: enlarge each engine's stdout pipe to 1 MiB (F_SETPIPE_SZ) so bursts can't stall the event loop even if the pump lags briefly. Also: name processes by role — coderai-front for the front, coderai-<name> for each engine (CODERAI_ENGINE_NAME passed at spawn), coderai for single-process. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
-
Stefy Lanza (nextime / spora ) authored
Add a cap on how much VRAM the auto-split may place on EACH secondary (non-main) card, so a slow second GPU (e.g. RX 580) stays lightly loaded and bottlenecks throughput less — the remainder stays on the fast card or spills to CPU. - config offload.split_secondary_cap_gb: global default, persisted + pushed live to every engine's global_args (each engine reads config at startup, so it's global). - per-model split_secondary_cap_gb overrides — but only to TIGHTEN: the effective cap is min(global, per-model) of whichever are set; a per-model value higher than the global is ignored. - applied in vulkan auto-split: caps each non-main device's free VRAM in the ratio (both vram & performance strategies), and in the auto-offload pool so the excess proactively spills to CPU instead of only reacting to an OOM. - UI: model config modal "Secondary card VRAM cap" + global Settings field. - added to _LOAD_AFFECTING so changes re-apply live on the next request. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The supervisor health-polls /internal/engine-state every couple seconds. While the engine is GIL-busy generating, that poll can't be answered in time and the engine was flipped to healthy=False — flapping out of the UI/routing mid-generation even though it's perfectly alive. Now a poll timeout only downgrades health when the PROCESS IS GONE (true death, already caught by the restart check); a timeout with the process alive keeps the last-known-healthy state. Also bump proxy_status_timeout 2s→4s so transient GIL contention doesn't trip it. (Pairs with the engine-state VRAM cache that removed the per-poll CUDA call.) Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two fixes for the performance split OOM loop at large context: 1) The performance "fits on the fast card?" check used expected_vram_gb (WEIGHTS only), ignoring the KV cache + compute buffers. A ~20 GB model at 178k ctx looked like it fit a 24 GB card → tensor_split=[1.0,0.0] → llama_context creation OOM'd (KV didn't fit) in a retry loop. Add a context-scaled KV/compute estimate (~n_ctx/16000 + 1.5 GB, calibrated to observed ~10 GB at 178k) to the footprint before deciding what fits — tight enough not to over-load the slow card. 2) On llama_context creation failure, progressively offload to CPU: move ~20% more layers to CPU and retry, up to 5 times, then fail. A slow CPU-spilled load beats a hard failure / crash loop. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The previous commit (be9950a5) left a try: with no except/finally in the SSE throughput counter, breaking import of the front. Remove the stray try (the BackgroundTask _release already closes the upstream response). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The Tasks page lost the it/s indicator under load: when the engine is too busy to report its own task, the front showed a synthetic in-flight task with rate 0. Now the front counts SSE "data:" events (~one token each for chat/text) as it relays the stream and publishes step + rate (tokens/s, refreshed ~2×/s) onto the in-flight metadata, so the synthesized task shows a live it/s even while the engine can't answer its own /admin/api/tasks. Only for streaming 200 responses; the engine's real task (with its own rate) still wins via the (engine,model) dedup when reachable. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Harden the max-instances guard against the concurrent case: request_model now resolves a GGUF text model to its absolute .gguf path (via _resolve_local_gguf) before building model_key, so every name form — bare id, basename.gguf, full path, or alias (e.g. "lisa") — collapses to ONE model_key. That gives a single self.models entry and a single instance pool, so two simultaneous first-requests under different forms converge on the same pool and the second queues instead of loading a second instance. Complements the earlier already-loaded fuzzy match (which only covered the sequential case). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
request_model's "already loaded?" fuzzy match compared basenames literally, so a bare request "gemma-…-Q4_0" missed the loaded "/AI/…/gemma-…-Q4_0.gguf" (the .gguf suffix differed) and a SECOND instance was loaded even with max_model_instances=1. A "lisa" alias resolved to the full path and matched, but the bare name didn't — so two requests for the same model (one via alias, one via name) ran as two instances instead of queueing on one. Normalize the .gguf extension (and compare short basenames) when matching, so every name form of one model maps to the same loaded instance and the second request queues. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The front health-polls /internal/engine-state every ~2s. It called torch.cuda.mem_get_info + get_device_name on EVERY poll, touching the CUDA context, which can serialize behind the running forward pass and stall the handler past the poll timeout — flipping a busy engine to "not responding". Cache the VRAM snapshot (4s TTL) and device names (permanent), so mid-generation polls return instantly from cache instead of blocking on CUDA. (llama-cpp-python 0.3.30 uses ctypes, which already releases the GIL during eval, so the compute itself wasn't the blocker.) Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The auto split is VRAM-proportional, which hands a big share to a slow second GPU (e.g. RX 580) so it bottlenecks throughput (Radeon ~100%, 3090 ~30%). Add a "performance" strategy that fills the fast lead card (main_gpu) first and spills only the overflow to the slower card(s), using the model's expected size — so the weak GPU holds the fewest layers (and none at all if the model fits on the lead card). "vram" (default) keeps the capacity-maximizing behavior. - config offload.split_strategy ("vram"|"performance") global default, persisted + pushed to live global_args; per-model split_strategy overrides it. - plumbed via build_runtime_kwargs + manager _cfg_or_global into vulkan auto-split. - model config modal: "Split strategy" select shown with the split options; global Settings: default split strategy select. - split_strategy added to the live-reload (_LOAD_AFFECTING) set so changing it re-applies on the next request without a restart. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
#1 Front-side in-flight task tracking: Engine.enter_request now records per-request metadata (model/kind/path/started_at) in engine.active; _merge_engine_tasks injects synthetic "running" task entries for in-flight requests not already reported by the engine (deduped by engine+model). Both the direct proxy and the broker route register/clear it. So the Tasks page shows work the front dispatched even when the engine is too GIL-busy generating to answer its own /admin/api/tasks. (Combines with the last-good cache.) #3 Thermal stopping-criteria cadence: the per-token callback held the GIL every token, starving admin handlers. Now it does real work every 50 tokens for text, scaling to every 100 when generation is fast (>50 tok/s) — far less GIL contention, negligible thermal-drift risk between checks. #2 (no change needed): generation already runs OFF the event loop via _aiter_blocking (asyncio.to_thread per token step) for streaming and asyncio.to_thread(manager.generate) for non-streaming. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The Tasks page polls the primary engine via the short-timeout client. While the primary is GIL-busy generating it can't answer, so poll() hit its except branch and returned the primary's tasks as EMPTY — the running generation vanished from the page under load. Cache the primary's last-good task list (120s TTL) and reuse it (marked stale) when the live poll fails, so an in-flight generation stays visible. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three related fixes for cross-GPU split of a vision model at large context: 1) gpu_detect: when exposing GPUs for cross-split (allow_cross), include only REAL hardware Vulkan devices (nvidia/amd/intel) and EXCLUDE software rasterizers (llvmpipe/lavapipe/virtio = vendor "other"). They're CPU-backed with no real VRAM, slower than native CPU offload, and their presence skewed the device list so the tensor_split ratio didn't line up with the actual cards (and layers could land on a fake GPU). 2) vulkan auto tensor_split now reserves the mmproj's size + compute margin on main_gpu (the projector always loads there), so the proportional split doesn't fill main_gpu to the brim and abort when CLIP can't allocate (GGML_ASSERT(buffer) failed). 3) auto-offload (n_gpu_layers sizing) subtracts the same mmproj reserve from the POOLED free VRAM, so when model+KV+projector exceed BOTH cards combined it reduces GPU layers and spills to CPU gracefully instead of crashing. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
gpu/live: honor per-model gpu_split (was offloading to CPU); apply model config live on any load-affecting change Two fixes: 1) build_runtime_kwargs never promoted gpu_split/tensor_split to top-level kwargs (only into _raw_cfg), but the manager reads config['gpu_split'] via _cfg_or_global (which doesn't consult _raw_cfg). So a model set to "Split — <card> first" loaded gpu_split=False → confined to one card → its big context spilled to CPU instead of distributing to the second GPU. Now promoted, so the split is honored and the GGUF auto-offload sizes against the POOLED VRAM across both cards. 2) apply_model_entry_live only evicted a loaded model when *acceleration* changed. Generalize to ANY load-affecting field (n_ctx, n_gpu_layers, gpu_split, tensor_split, cache types, kv_offload, n_batch/ubatch/seq_max, flash, quant, engine pin, vae/precision, …): if it changed and the model is loaded, evict it so the next request reloads with the new config — config changes apply immediately, no server restart. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
- Model list: a model with cross-GPU split now tags as "<lead> ⇄split
📌 " (with a tooltip naming the lead card + ratio/auto), so the list makes "loads here, spills onto the other GPU(s)" visible instead of looking like a plain pin. - gpu_detect._amd_stats reported VRAM in decimal GB (bytes/1e9) while _nvidia_stats used GiB (MiB/1024). That made the two cards ~7% out of step in the dashboard and in any cross-card free/total sum. AMD now reports GiB to match, so free-VRAM is consistent and comparable across both cards. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The per-model engine selector was confusing: "Auto" implied single-card (correct) but the only split choice was a generic "All GPUs". Replace with hardware-named, ordered choices: - "Auto (single GPU — by capability / free VRAM)" → one card (unchanged) - "<engine> (single card)" → pin to that card - "Split — <engine> first" → pool across all GPUs with that engine's card leading (main_gpu / larger share). Generated per engine, so on a 3090+Radeon box you get "Split — nvidia first" and "Split — radeon first". Saving a split option sets gpu_split + pins engine=<lead>; the weight-ratio field (blank = auto by free VRAM) shows only for a split choice. Make per-model split actually function: - engine_supervisor: expose every GPU to engines (allow_cross) when the global toggle OR ANY model entry has gpu_split — otherwise the lead engine couldn't see the foreign card. - vulkan.py: when an engine sees BOTH backends' cards but THIS model isn't a split, confine it to its own backend (zero the foreign devices in tensor_split) so it can't accidentally spread — while same-backend multi-card split still works. Split models pool across all visible devices (lead card = device 0 = main_gpu). Co-Authored-By:
-
- 21 Jun, 2026 14 commits
-
-
Stefy Lanza (nextime / spora ) authored
Two distinct "config goes back to what it was after restart" bugs: 1) save_config dropped broker websocket_path + websocket_ping_interval (and the new offload gpu_split/tensor_split). The settings UI set them in memory, but save_config never serialized them, so the next load() fell back to dataclass defaults — the broker protocol/websocket settings reverted every boot even with a fully persistent config dir. Now serialized. 2) Per-model fields (e.g. n_ctx) reverted via a multi-process models.json write race: every engine loads models.json at its own boot, so a SECONDARY engine's in-memory models_data is stale w.r.t. a later UI edit on the primary. When that secondary engine auto-persisted measured_vram_gb it called save_models(), which dumps its whole stale state and clobbered the edit. Added ConfigManager.persist_model_field(): a single-field, atomic read-modify-write that re-reads models.json from disk, updates only the one field, and refreshes the in-memory copy. record_vram_delta now uses it, so VRAM measurement can never revert user model config. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Per-model (model config modal): the "Engine / card" selector gains an "All GPUs (split across every backend)" option. Choosing it sets gpu_split and reveals a "Weight distribution" field (tensor_split, e.g. 0.8,0.2); blank = auto. A real engine name still pins as before. Global (Settings): "Split models across all GPUs" checkbox + default weight distribution field, persisted to offload.gpu_split / offload.tensor_split and pushed to live global_args. Per-model "All GPUs" overrides the global default. api: settings GET/POST now expose+accept offload.gpu_split / tensor_split. backend: when gpu_split is on and no ratio is given, auto-derive tensor_split proportional to each device's FREE VRAM (new _per_device_free_vram_gb, llama.cpp device order: CUDA first, then Vulkan/AMD) — e.g. 24 GB 3090 + 8 GB RX 580 -> ~0.75/0.25 — so the bigger card carries more without the user computing it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A multi-stage build leaves untagged <none> intermediate layers behind that pile up across rebuilds. After the build (and the dist export, which still needs the final image), run `docker image prune -f` to drop them. Only DANGLING images are removed — the final image, the pulled CUDA base, and any other tagged images are kept. Default on; disable with --no-prune. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Gemma emits tool calls as text — <|tool_call>call:NAME{…} — which the first-pass streaming loop forwarded straight to the client as assistant content. So the call showed up as a visible message and only got parsed/executed at end-of-stream. Add _ToolCallStreamGate: as chunks stream, emit visible content only up to the first tool-call marker (gemma's <|tool_call>/<|tool_response> plus the common tag formats), holding back a small tail so a marker split across chunk boundaries isn't leaked; once a marker appears, emit nothing further. The full text still accumulates for the existing end-of-stream tool extraction (GemmaParser), which emits a proper tool_calls delta. The second pass is already buffered+extracted, so it never leaked. Also add gemma's "<|tool_call>" to the reasoning/tool split pattern lists. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
gpu-stats fell through to the generic proxy, which forwards via the long-timeout client to the primary engine. When that engine is saturated generating (sync llama.cpp holding its event loop), the request blocked → the dashboard's temp/GPU polling hung → the whole web UI went unresponsive and the task page stopped showing temperatures. Serve it from the front instead, using the already torch-free gpu_detect.gpu_stats() (nvidia-smi + AMD sysfs, reports every card regardless of CUDA_VISIBLE_DEVICES), run in a thread so the subprocess never blocks the front's event loop, and registered before the catch-all so it's not proxied to a busy engine. Light auth (session cookie / bearer presence) since GPU telemetry is low-sensitivity and full session validation lives on the engine — which is the component we're decoupling from. Note: the thermal THROTTLE (pausing a generation when hot) stays in the engine — it must, to pause that engine's own forward pass — but its stats are now front-served. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Broker (AISBF) chat requests were rejected by the engine with 401 "Invalid API key" — they're authenticated at the aisbf layer (registration token) and carry no end-user Bearer, but the engine's BearerAuthMiddleware still demanded one, so every brokered request came back as a constant ~472-byte error and never ran inference. Fix (token auto-managed, secure): - Front marks broker-relayed requests with x-coderai-broker-authed = the internal shared token (which the engine already trusts via CODERAI_INTERNAL_TOKEN). Added that header to _DROP_REQ so a client-supplied copy is always stripped first — unforgeable from outside. - Engine BearerAuthMiddleware skips the Bearer check when x-coderai-broker-authed matches CODERAI_INTERNAL_TOKEN. This is NOT the plain internal token (which rides on every front→engine request, direct included), so DIRECT API requests still require a real Bearer token. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Previously pooling was gated on gpu_split, so a 2-card same-backend engine (e.g. 2× 3090, or 2× Radeon) still measured only one device for the fit/eviction math. Now both manager._get_free_vram_gb() and vulkan._pooled_free_vram_gb(): - ALWAYS sum every visible CUDA device (torch honours CUDA_VISIBLE_DEVICES, so it is scoped to this engine's NVIDIA cards) → same-backend split is accounted for with no flag. - add AMD card(s) (amdgpu sysfs) only when cross-backend split is on OR no CUDA device is visible (a Radeon/Vulkan engine), so a Radeon engine counts its own cards and an NVIDIA engine only reaches across to Radeon when split is enabled. So: 2× NVIDIA → summed across both NVIDIAs; 2× Radeon → summed across both Radeons; split on → summed across all NVIDIA + Radeon. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
When cross-backend pooling (offload.gpu_split) is on a model can span several cards, so VRAM math must consider the pool, not one device: - vulkan.py: new _pooled_free_vram_gb() sums free VRAM across every visible card (all CUDA devices + AMD via amdgpu sysfs; no double count). The GGUF auto-offload fit decision uses the pooled figure when the model is gpu_split, so layers aren't needlessly pushed to CPU just because one card's free VRAM looks small. - manager._get_free_vram_gb(): when gpu_split is enabled, report pooled free across all GPUs (used by the eviction loop + budget checks), so eviction frees/measures capacity across both cards. Off → unchanged single-device behavior. Pooling is aggregate (sum); strict per-device fitting/targeted eviction can be a later refinement. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
gpu: per-engine backend isolation (fix docker cross-GPU split) + opt-in per-model cross-backend pooling Phase 1 — fix the docker-only "model loads on both GPUs": - gpu_detect.vendor_env detects each Vulkan device's vendor and pins each engine to ONLY its own backend's cards by real indices (not assumed 0..n-1). When a vendor has no Vulkan device (e.g. NVIDIA in a container that lacks nvidia_icd.json because the toolkit only injects it with the graphics capability), the engine gets ZERO Vulkan and runs CUDA-only instead of falling back to all ICDs and grabbing the Radeon via RADV. Same-backend split (e.g. 2x 3090) is preserved. Phase 2 — opt-in cross-backend GPU pooling, per model: - OffloadConfig.gpu_split (default off) + tensor_split ("0.8,0.2", llama.cpp device order: CUDA first then Vulkan); global default + per-model override. - vendor_env(allow_cross=…) exposes the foreign card when enabled; the engine supervisor passes it from config. - manager threads gpu_split/tensor_split (per-model via _raw_cfg, else global via global_args) into the GGUF loader; vulkan.py sets llama.cpp tensor_split when on and otherwise leaves split_mode=LAYER so same-backend split still works. - admin model-configure accepts gpu_split + tensor_split per model. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
frontproxy: - _DROP_RESP now drops date/server when relaying engine responses. The front's own ASGI server adds Date/Server, so keeping the engine's too produced DUPLICATE header lines — which nginx logged as a warning on every request, flooding the terminal. Now each appears once. - Brokered route logs the engine's actual status/bytes/preview (or "NO ENGINE -> 503") so a brokered request that "doesn't get executed" (instant tiny reply) is diagnosable from the log. coderai-oci: - Use the host-tailable file log only when it's actually writable. When the container runs as --user but the log dir/file was created by an earlier root run, it's root-owned; tee then spammed "Permission denied" and the file stayed empty. Now we detect that and log to stdout only with a clear note. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
launcher: image auto-resolution, --user in-place config persistence, --debug SPEC, install/uninstall image handling run_oci.sh: - Resolve the installed coderai image when none is given (single -> use it, several -> menu); no longer hardcodes a possibly-wrong tag. - --debug accepts SPEC as the next token (not just --debug=SPEC) and won't swallow an image-looking arg, fixing "image: <debug spec>" mishaps. - --local puts the runtime dir under ~/.config/coderai-runtime (override --data-dir). - New --user[=UID[:GID]]: run as that user and switch a config dir to an IN-PLACE mount so the app's config edits persist (owned by you). Without --user, --local/--config-dir stay a throwaway copy. Banner shows user. Non-destructive config persistence (no startup rewrite): - codai/cli.py: new --host/--port that override config.server in memory only. - codai/main.py: apply those overrides right after config load (never written to config.json). - coderai-oci: pass --host/--port to the server and stop rewriting an existing config.json (only create one on true first run) — so an in-place-mounted config dir is never modified at startup. install.sh / uninstall.sh: - install: after loading, offer to remove OTHER coderai images (default no); no auto-launch. - uninstall: resolve the installed coderai image(s) to remove (single -> confirm, several -> menu / all / none) instead of a hardcoded tag. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
coderai-oci (in-image): append ${CODERAI_EXTRA_ARGS} to the server argv in both exec paths. supervisord runs a fixed command, so arbitrary server flags arrive via this env var rather than argv. (Takes effect after an image rebuild.) run_oci.sh: --coderai-arg ARG (repeatable, one token each) and --coderai-args "STR" build CODERAI_EXTRA_ARGS and inject it with -e; new cdr-args banner line. `--` still goes to the container engine, not coderai. Docs (AI.PROMPT, dist-bundle README.md/.txt) updated accordingly. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
build_oci_image.sh: - --versioned derives a deterministic tag from codai.__version__ + build mode + GPU: local-venv -> coderai:full_<gpu>_<version>, from-scratch -> coderai:base_<gpu>_<version> (overrides -t/--tag). - --gpu all|nvidia|vulkan (default all) sets the <gpu> token; label only — the image always bundles both CUDA and Vulkan. run_oci.sh: - --host ADDR binds the published port to a specific interface (-p ADDR:PORT:8776); default unchanged (all interfaces). CODERAI_HOST stays 0.0.0.0 in-container. Banner URL reflects the bind host. Docs (AI.PROMPT, dist-bundle README.md/.txt, README-RUN.txt): - Document --versioned/--gpu, additive --nvidia/--vulkan/--all, --vulkan auto-libcuda + --with-libcuda, graceful llama-cpp degradation, --host, and the new uninstall.sh + confirm gates. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
docker/backend: graceful llama-cpp load + additive GPU modes + libcuda mapping; admin GGUF batch/slots tuning Backend robustness: - vulkan.py catches Exception (not just ImportError) around the llama_cpp import: a CUDA-built llama-cpp missing libcuda.so.1 raised RuntimeError/OSError that crash-looped the whole server. Now it logs a warning and marks the Vulkan/GGUF backend unavailable; CUDA/CPU/ds4 keep working. - detect_available_backends() reads LLAMA_CPP_AVAILABLE instead of re-importing (which re-raised the same error). Docker launcher (run_oci.sh): - GPU backends are now additive: --nvidia --vulkan enables both (maps libcuda via --gpus all AND /dev/dri). Added --all and --with-libcuda[=PATH]. - --vulkan auto bind-mounts the host's libcuda.so.1 (the bundled llama-cpp is a CUDA build), so Vulkan GGUF loads without full --gpus all. Banner shows mode set and libcuda status. Dist bundle: - New uninstall.sh (removes runner + optional image), wired into make_dist_bundle. - install.sh + uninstall.sh print what they'll do and confirm before proceeding, bypassable with --yes/-y. Admin GGUF tuning: - Expose n_batch / n_ubatch / n_seq_max (llama.cpp -b/-ub/-np) in the model config UI and apply them in the Vulkan backend to shrink VRAM at the ceiling; n_seq_max gated on llama-cpp-python support. Co-Authored-By:
-
- 20 Jun, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authored
The model-upload endpoint joined a client-supplied filename straight onto the cache dir, so an admin-authenticated request with a traversal filename (or upload_id) could write outside it. Reduce both to a safe basename, reject separators/.., and add a commonpath containment check before committing the upload. SessionManager only locked the write half of each load->mutate->save, so concurrent writers could clobber each other's changes (lost sessions or tokens). Add update_auth_data(mutator), which holds the lock across the whole read-modify-write and persists only when the mutator asks to; route every mutating method (and the token create/delete endpoints) through it. Read-only callers keep the lock-free load since writes are atomic via os.replace. While migrating the token endpoints, switch IDs to max+1 (no reuse after deletion) and to timezone-aware timestamps. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Large contexts make the KV cache huge (a 256k q4_0 cache is several GB), which won't fit in VRAM alongside the weights. llama.cpp can't page KV to disk, but it can keep it in system RAM via --no-kv-offload. Expose that as a per-model kv_offload flag (default unchanged = KV in VRAM): set kv_offload=false to pass offload_kqv=False to llama.cpp, freeing VRAM for big contexts at the cost of slower decode (KV ops cross PCIe). Also allow the key in the admin model-config endpoint so it's persistable from the UI. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Add a canonical codai.__version__ = "0.1.0" as the single source of truth (kept separate from Config.version, which is the config-schema/migration version). The admin template renderer injects it as coderai_version, and base.html shows it as a small "v0.1.0" pill next to the CoderAI logo on every page. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Some quantized fine-tunes (seen with an "Aggressive" Qwen3.6-35B Q4_K_M) collapse into a runaway repetition loop — emitting a malformed parallel tool-call flood of 1700+ tokens that never terminates — when top_p=1.0 and no repetition penalty are in effect (exactly the conditions Qwen's own docs warn cause endless repetitions). Two fixes: 1. Anti-loop generation stop in stream_chat_response: a model-agnostic detector normalises away the variable parts of the tail (quoted strings, filesystem paths, whitespace) so a loop whose only per-cycle difference is an arg/path still reads as periodic, then breaks generation when a short structural unit repeats >=5x back-to-back. Tuned to not trip on prose, repetitive code, or a legit handful of distinct tool calls. 2. Honor client-supplied repetition controls. The chat paths previously forwarded only temperature/top_p, silently dropping repeat/presence/frequency penalty — so a caller (e.g. Kilo) setting them per-model had no effect. Plumb them through generate_chat_stream / generate_chat to both backends (cuda already accepts them; vulkan now does too) with graceful signature fallbacks. Defaults are no-ops, so unset clients are unaffected. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Auto-compaction never triggered: multi_model_manager.config stores the whitelisted build_runtime_kwargs() dict, which drops the per-model auto_compact* keys (they survive only under _raw_cfg), so _resolve_compaction always read the global default (False) and returned None. Read the keys via a _raw_cfg fallback so per-model compaction config is honoured. Also rework the over-context handling to count the reply reservation, since the reply is generated into the same window (prompt + max_tokens <= n_ctx). Four layers, cheapest first: 1. fits as-is -> nothing 2. overflow within tol -> trim max_tokens to fit (lossless) 3. beyond tol & big prompt -> compact history (drop/summarize) 4. single message too big -> slice it (summarize its middle, keep head/tail) The chars/4 estimate undercounts token-dense code/JSON, so trimming to the exact n_ctx edge could still overflow; inflate the estimate by a configurable estimate_safety (default 1.15) for all physical-fit decisions. New CompactionConfig knobs (per-model overridable): tolerance_pct (20), min_output (512), estimate_safety (1.15). Effective max_tokens is threaded back to both the streaming and non-streaming generation paths. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An engine restart (admin button / config change) previously SIGTERM'd the process immediately, severing any active SSE stream mid-response — the client saw httpcore.RemoteProtocolError "peer closed connection without sending complete message body". Now restart_engine marks the engine `draining` first: the router stops routing NEW requests to it (Engine.is_alive() reports false while draining, and the poll loop can't flip it back healthy), and the supervisor waits up to server.engine_restart_drain_grace seconds (default 30, 0 = immediate) for the in-flight count to reach zero before killing the process. Stragglers past the grace window are still bounced. In-flight is tracked per engine in the front proxy: proxy() increments on send and decrements once the streamed response is fully drained (or the send failed). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Qwen-style chat templates pre-fill the opening <think> in the prompt, so the model emits only the reasoning body + a bare closing </think> — and they think by DEFAULT regardless of the API enable_thinking flag. The old paired-tag reasoning extractor missed the bare close, leaking the whole thought (and the </think>) into content and conversation history. - extract_reasoning_content: handle a bare </think|/thinking|/thought> with no opening tag (treat the prefix as reasoning). - streaming: a chunk-safe reasoning gate routes the thought into delta.reasoning / reasoning_content until </think>, then flips to content; tool extraction runs on the post-</think> answer only. - non-streaming: extract reasoning, set message.reasoning(+_content), clean content; tools parsed from the answer. - activate whenever the model auto-thinks (qwen3/qwq/deepseek-r1/… name) OR reasoning is explicitly enabled — not just on the API flag. - configurable suppression: per-model `suppress_reasoning`, or per-request via the standard reasoning:{exclude:true} / reasoning_effort:"none" / suppress_reasoning fields. Emits both `reasoning` and DeepSeek-style `reasoning_content` for client compatibility. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Optional CPU-temperature source for thermal control. Not essential — the thermal monitor reads CPU temp from psutil and the kernel's /sys/class/thermal zones first (both work in-container); `sensors` is only a last-resort text parse for hosts whose sysfs doesn't expose a CPU zone. Added to both the from-venv and from-scratch runtime stages for completeness. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The plaintext <tool> rescue could turn a failing 2-bit model's repetition loop (<tool>glob</tool><tool>glob</tool>… / bare names, no args) into a flood of bogus tool calls. Harden it: reject a batch with >6 <tool> blocks (that's model degeneration, not many real calls) and drop any bare <tool>name</tool> that carries no key: value argument (the spam signature). Genuine single/few calls with arguments still parse; combined with the existing trailing-action, declared-name, and DeepSeek-only scoping. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
- ds4 kv janitor: a checkpoint is deleted only when ALL hold — untouched by max(mtime, atime) for the age (so a checkpoint ds4 merely READS, which bumps atime not mtime, is spared); not currently open (fd/mmap) by a ds4-server; and ds4 is not serving any request. New in-flight counter on Ds4Backend (any_request_active) gates the sweep. - settings: "Download a default DeepSeek V4 model" — select + button backed by new /admin/api/ds4/default-models catalog (q2-imatrix / q2-q4 / q4 / mtp from antirez/deepseek-v4-gguf). Reuses the normal downloader, which flattens the gguf into the cache and surfaces it in the model list; live progress. - parser: rescue the degraded plaintext <tool>name arg: value</tool> form that heavy quants (ds4 q2-imatrix) emit when they can't reproduce DSML. Scoped to DeepSeekParser only (never the shared ToolCallParser, so other families are untouched), requires a DECLARED tool name, plaintext-only inner, and the block(s) to be the message's trailing action — so a <tool> example inside a prose reply is not misread as a call. - settings: corrected ds4 perf note (i-quants/Q2_K fail CUDA prefill; use Q4_K+). Co-Authored-By:
-