- 20 Jun, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authoredAdd a "tuning ds4 for performance" note to the global ds4 Settings section and a condensed inline note to the per-model ds4 streaming section on the Models page: NVMe/SSD placement (≈10× prefill), capping the expert cache by count, sizing the VRAM reserve, DS4_CUDA_WEIGHT_ARENA_CHUNK_MB=512, avoiding DS4_CUDA_WEIGHT_CACHE, and that decode of a model larger than VRAM is streaming-bound (smaller quant is the real fix). Distilled from measured tuning on the 154GB DeepSeek-V4 MoE. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com>
-
Stefy Lanza (nextime / spora ) authored
The streaming tool-content gate withheld <tool>/<|tool_call>/call: markers but not DeepSeek V4's native <｜DSML｜tool_calls>… block (｜ = U+FF5C), so during a streamed tool call the raw markup reached the client token-by-token as visible content (even though the post-stream parser extracted the tool_calls correctly). _gate_tool_content now withholds everything from the first <｜DSML｜ marker to the end (dropped on final, surfaced as structured tool_calls), and the trailing- partial hold list includes the DSML open tag so a marker split across chunks doesn't leak its leading chars. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Per-model ds4 tuning (these vary by quant/size/context, so they belong on the model, not globally): - Optional `ds4` block on a model entry overrides the global Ds4Config for ssd_streaming / expert_cache_reserve_gb / extra_args / extra_env; unset fields inherit the global config (the default/template). Ds4Backend looks up its own model entry and applies the overrides via dataclasses.replace. - admin: api_model_configure accepts + normalizes the per-model `ds4` block, dropping it when empty. - models page: a "ds4 streaming" section shown only when ds4 is enabled globally and the model is a deepseek4; n_ctx stays the context knob. Fix garbled / truncated ds4 replies: the streaming reader used iter_lines(decode_unicode=True), which decodes each network chunk independently and corrupts a multibyte UTF-8 char split across chunks ('—' -> 'â'); the broken JSON then made json.loads fail and the token was silently dropped (truncated tails). Parse the SSE byte stream and split on the b"\n" byte (never inside a UTF-8 sequence), decoding whole lines; also flush a final newline-less line. UI: slow-reply notice reworded to "Waiting for model reply..." with a trailing newline so the real reply starts on its own line. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
DeepSeek V4 (ds4) emits native tool calls as <｜DSML｜invoke name="…"> <｜DSML｜parameter name="p" string="true">val</｜DSML｜parameter></｜DSML｜invoke> (the ｜ is U+FF5C). No parser recognised this, so ToolCallParser returned None and the raw markup leaked to the client as content even though ds4 reported finish=tool_calls. - parse_deepseek_dsml_tool_calls(): extract (name, args); string="false" params are JSON-decoded, others kept as strings; ASCII | tolerated. - Wired into DeepSeekParser and ToolCallParser.extract_tool_calls (the live path). - strip_dsml_tool_calls(): drop the DSML block from displayed content in both strip_tool_calls_from_content paths. Guarded by 'DSML' in text -> no effect on other models. Also reword the slow-reply notice from "Waiting for model to load..." to "Waiting for model reply..." (the model is usually loaded, just slow). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
ds4-server exposes several CUDA tunables only via environment, not CLI flags. By default ds4 reserves half the card for non-cache use and allocates the model weight arena in 1792 MiB chunks — both starve / OOM the streaming expert cache on small-weight MoE models served from SSD. Pass an explicit env to ds4-server (Popen now sets env=) with: - expert_cache_reserve_gb: typed knob -> DS4_CUDA_STREAMING_EXPERT_CACHE_RESERVE_GB (0 = leave ds4's default). - extra_env: free-form KEY=VALUE passthrough for the rest, e.g. DS4_CUDA_WEIGHT_ARENA_CHUNK_MB=512 to shrink the weight-arena chunk so it fits a heap fragmented by the expert cache. Both surfaced in Settings (config + admin GET/POST + UI), default to no-op so behaviour is unchanged unless set. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Add a GitHub-flavored README.md alongside the existing README.txt in the all-in-one docker distribution bundle, and have make_dist_bundle.sh stage it so future builds include both. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
- models page: the multimodal-projector select now defaults to None unless a projector is a strong, unambiguous name match. Scores only distinctive tokens (drops generic words + quant tokens, keeps size tokens like 14b), requires covering at least half the model's tokens, and rejects ties. Stops a lone shared family token from pairing the wrong-size projector. - task page: the per-engine loaded-model hover now lists each model once by its canonical id instead of its aliases (auto gguf stem, explicit alias, type prefix). engines_list() resolves loaded keys via the pin index's new model_id. Co-Authored-By:
-
- 19 Jun, 2026 33 commits
-
-
Stefy Lanza (nextime / spora ) authored
By default a per-model engine pin is a hard constraint: if the pinned engine is down/incompatible the request fails (no duplicate on another card). Add an `engine_fallback` model-config flag (admin form checkbox + persisted in models.json) that opts into the old behaviour — fall back to a compatible engine when the pin can't be honoured. A pinned engine that's merely busy-but-alive is still routed to (queues) in both modes; fallback only applies when it's actually down or can't serve the model. Threads pin_fallback through pick_engine; the front reads engine_fallback via _load_pins/_model_info. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
An engine mid-generation is GIL-blocked and fails the health poll, so it reads as unhealthy. pick_engine required e.healthy at every step, so a second request for a model pinned to that engine fell through to the least-loaded engine — which loaded a DUPLICATE copy (and ignored the model's configured n_ctx, e.g. 2048 vs 32000 → "exceeds context window"). Honour the pin (and the assigned owner) when the engine is alive but transiently busy: route there so the request queues on its gen-lock and the owner handles serialization/eviction. Only fall back to another engine when the owner's process is actually dead. Adds Engine.is_alive() (process liveness) and registry.engine_owning() (health-agnostic owner lookup). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The gemma loose object/array parser could spin forever: when a recursive value parse can't advance past a stray delimiter (e.g. '}' where ']' was expected, as in the broken `{"files":[{"path":"x"]}}` a looping Gemma finetune emits), the array/object loop kept iterating without consuming input. parse_gemma_native_tool_calls and the new parse_tool_tag_json_calls both feed model output through this parser, so a malformed tool call would hang the request (not just be missed). Add a forward-progress guard to both loops: bail when an iteration consumes no input. Best-effort recovers the tool name + good fields from malformed JSON; clean input is unaffected. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
collect_models unioned only registry.healthy() engines and re-fetched each engine's /v1/models live. An engine loading a model is GIL-blocked and misses the 2s health poll, so it goes "unhealthy" and ALL its models — including a freshly-added one — drop out of the aggregated /v1/models until the load finishes. A client (e.g. the kilo model script) polling during a load then sees models vanish. Cache each engine's last-good /v1/models and, when it's transiently unhealthy/unreachable, serve that cached list instead of dropping it. The models are still assigned to the engine and will serve once it's free. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Some Gemma finetunes emit tool calls as <tool>{"name":..,"arguments":..} using <tool> (or <tool_call>) as BOTH delimiters — no closing slash — and sometimes append a stray quote. Every existing <tool>…</tool> pattern requires a slashed closer, so GemmaParser found nothing and the call was returned as plain text: kilocode saw no tool_call and showed no reply. Add parse_tool_tag_json_calls, which extracts the brace-balanced object after each marker via the tolerant loose parser (so trailing junk and stray quotes don't break it) and reads name + arguments/parameters, restricted to declared tool names and de-duplicated. Wired into GemmaParser before the generic fallback. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Whether a model rejects the 'system' role is a property of the chat template baked into the specific GGUF, not the architecture: the gemma-2 template and the official gemma template raise "System role not supported", while 'heretic' gemma4 quant conversions ship a permissive template that accepts system. Detect from the embedded tokenizer.chat_template (raise_exception/"system role") and fold only when it actually rejects system; fall back to architecture (Gemma) when no template is readable. Avoids needlessly folding permissive Gemma models while still covering gemma-2-9b and strict non-Gemma templates. The runtime "System role not supported" retry remains as a safety net. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Gemma's chat template has no 'system' role; llama.cpp raises "System role not supported" and the generation fails (the Kilo client always sends a system prompt). On that specific error, retry with the system message(s) folded into the first user turn — Gemma's own convention, and a no-op for models that accept system. Handles both streaming and non-streaming paths and preserves multimodal (list) content. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
VulkanBackend.load_model treated any id that isn't a '.gguf' path / file / URL as a HuggingFace repo to download. A configured gguf addressed by its automatic alias ('coe-…-q4_k_m', no extension) thus 404'd against the Hub instead of loading the local file. Resolve the alias via _resolve_local_gguf (configured-entry + cache-dir match) first; only fall back to the HF path when no local gguf is found. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
pick_engine honours the front's assignment (radeon) only if the engine can_serve the request's required capability. But _required_cap derived that capability from the bare alias 'coe-…-q4_k_m' — no literal 'gguf' — so required_capability returned 'transformers' (CUDA-only). radeon is gguf-only, failed can_serve, and the request fell through to the default engine (nvidia), even though compute_assignment had correctly placed the model on radeon (it sees the full '…-q4_k_m.gguf' path). Resolve the model's configured path in _load_pins (now indexed by the .gguf-stripped stem too) and, when the name heuristic yields 'transformers' but that path is a .gguf, correct the capability to 'gguf'. whisper/ds4 precedence is unchanged. Combined with the registry stem-matching, a bare-alias request now lands on the owning Vulkan/AMD engine. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
_free_vram_gb() used torch.cuda.mem_get_info, which returns 0 on an AMD/Vulkan engine (no CUDA). That made the auto-offload sizing guard (_free > 0) silently false, so n_gpu_layers stayed at -1 (all) and a model larger than VRAM was forced entirely onto the GPU — OOM, "Failed to load model from file" (e.g. a 13 GB gemma4 model on an 8 GB RX 580). A 24 GB CUDA card has room for all layers, so the bug was invisible there. Fall back to amdgpu sysfs (mem_info_vram_total - vram_used, indexed by Vulkan device order) so AMD GPUs size partial offload too. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
load_model() decided gguf-vs-HF purely from the literal string 'gguf' in the model name. A gguf whose alias carries only the quant suffix (e.g. 'coe-gemma4-coding-hc-14b-a4b-q4_k_m', no literal 'gguf') was mis-routed to the HF/transformers backend, which then failed with "is not a valid model identifier" (503). Fall back to _resolve_local_gguf(): if the alias maps to an actual local .gguf, treat it as gguf and route to llama.cpp. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A gguf model's assigned/loaded key is its file path, but /v1/models advertises it — and clients address it — by the filename without the .gguf suffix (the automatic alias). engine_for_assigned / engine_for_model / _key_matches_path compared short names verbatim, so the automatic alias never matched the .gguf key and routing fell through (404 / wrong engine). Normalize both sides via _short_stem so the automatic alias resolves to the owning engine with no manual alias. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two bugs made a freshly-configured model unusable until a full restart on a multi-engine node: 1. Name mismatch: list_models advertises a gguf's filename WITHOUT .gguf as an id, but get_all_allowed_identifiers only allowed the name WITH .gguf, so a request using the id from /v1/models was 404'd as "not an allowed model". Now the .gguf-stripped stem is allowed too. 2. Stale per-engine assignment: each engine's /v1/models is filtered by the assignment set fixed at startup, and secondary engines never re-read models.json — so an added/removed model didn't show up or route until restart. The front now watches models.json mtime, recomputes the assignment, updates its router, and pushes it to every engine via a new internal POST /internal/reload-config (re-reads models.json + set_assigned_models). /v1/models and routing now reflect add/remove live. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The whisper-server runner's model_id is inherited from the gguf MODEL config's alias, which links the two. So: - Adding a model config creates one runner whose id is the config's alias (auto-minted + stamped onto the config when no alias is given). - Removing a config (by config_id or by path) tears down the runner whose id matches that config's alias — one config removed = one runner removed/killed. Replaces the interim model_config_id link. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Starting a whisper-server runner loads the gguf onto the GPU, but it was invisible to the VRAM-eviction logic — it never evicted others to make room, recorded no footprint, and (lacking a cleanup()) couldn't itself be evicted. - WhisperServerManager.cleanup() -> stop(), so _evict_one/unload_model can free its VRAM like any other model. - MultiModelManager.start_whisper_server(): estimate the gguf footprint, evict other models if free VRAM is short, start the subprocess, and register it in models/models_in_vram/_measured_vram_gb (active_in_vram). It's now both a trigger for eviction and an eviction candidate. - stop_whisper_server(): stop + clear all that accounting (frees VRAM). - Routed every start/stop through these: on-request transcription, engine startup pre-load, admin model-load (Load button) and model-unload/disable. So: starting a runner = a model load (evicts as needed); unloading = frees VRAM. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Model a whisper gguf as two things: a MODEL config (a .gguf entry with backend=whisper-server and NO model_path — enables the model, holds load strategy, shown on the GGUF row) and a RUNNER (backend=whisper-server WITH model_path — the subprocess, shown in the whisper card). - Enabling a .gguf with speech_to_text marks it backend=whisper-server and auto-creates exactly one runner (1:1) on a free port. - Disabling the model removes + kills all its runners (cascade by model_path). - Removing a runner (or model) now stops the subprocess + drops registry entries, instead of leaving it running until restart. - cached-models shows the model config on the GGUF row but excludes runners; the whisper card shows only runners (require model_path). - engine startup only launches runners (entries with model_path), never the bare model config. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A GGUF model config (configure + enable the model, load strategy) and a whisper-server config (the runner: port, gpu, which model) are two distinct things. Showing whisper-servers as the backing file's configs made the GGUF row's "Configure" open the whisper form — conflating them. Whisper-server entries are again excluded from a GGUF file's editable config list (they live only in their own card); the GGUF row's Configure opens the general model config modal. The file still reflects "loaded" via its model_path in the loaded-status sets. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Reverted the over-correction that hid whisper-server entries from the backing GGUF file's config list — they should appear there (the file is configured through them). To avoid the duplicate-config bug, editing a whisper-server config from the GGUF row now routes to the whisper editor (which updates in place by id) instead of the general config modal. Pills for whisper-server configs are labelled by id so the two instances are distinguishable (they share the "whisper" alias). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Editing a GGUF model's config kept appending duplicate entries. Root cause: api_cached_models added whisper-server entries to the backing GGUF file's config list (keyed by model_path). With whisper0/whisper1 both pointing at the file, the GGUF row's configs[0] became a whisper-server entry, which carries no config_id — so "Configure" on that row treated every save as a brand-new config and spawned a fresh duplicate each time. - cached-models: skip whisper-server entries entirely (they're managed in their own card; the file still shows "loaded" via its model_path key). - model-configure (whisper-server): update an existing entry in place when the id matches instead of 409-or-append, preserving unmanaged fields (engine, config_id). - model-disable: guard against whisper-server entries' path=None so a path-based disable can't crash on basename(None). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Clicking Load/Unload (or any refreshLocal) re-ran loadCachedModels, which blanked the HF/GGUF lists to "Loading…" every time. That collapsed the page height and threw the viewport to the end, so a whisper-server unload looked like it "did nothing and scrolled to the bottom" even though it worked. Now the "Loading…" placeholder only shows on the first (empty) load; on a refresh the existing rows stay in place and the scroll position is captured and restored around the re-render. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Removed the redundant "Load mode" dropdown from both whisper-server forms (builder + edit modal). A whisper-server is backed by a GGUF model, so its load mode now derives from that GGUF's configured load_mode (default on-request) instead of being set separately in two places. Load/offload strategy stays solely in the GGUF model config. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two whisper-server UI issues on a multi-engine node: - Unload didn't visibly take effect: a whisper-server is registered both as a subprocess (whisper_servers) AND in the generic .models/.model_pools registry under audio:<id>. The unload stopped the subprocess but left the registry entry, so the row kept showing "Unload". Now it also drops the audio:<id>/<id> registry entries (and matches by id, audio:<id>, or the gguf model_path, so unloading the file stops every server using it). - The backing gguf file showed "Load" while its whisper-server was running. Surface each running server's _model_path in the loaded-key sets (engine-state, model-loaded-status, status) so the GGUF-file row reflects that the file is in use. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
/admin/api/models is served by the primary engine and ran list_models() with the per-engine assignment filter, so models pinned to a secondary engine (whisper-servers configured with engine=radeon) were dropped. That emptied the whisper-server config form and left those models unmatched in the loaded-status check, so they never showed as loaded. Add list_models(all_engines=True) to bypass _entry_assigned, and call it from the admin model list. Combined with the whisper-servers now reported in /internal/engine-state, the radeon whisper-servers reappear in their own form and show as loaded. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The supervisor polls /internal/engine-state (not /admin/api/status) to populate each engine's loaded_models in the registry, which the front then aggregates for the models page. That endpoint only read .models, so a whisper-server running on a secondary engine still showed as not loaded. Fold each running server (id + `audio:` alias) into its loaded_models. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Two issues when unloading/reporting models on a multi-engine node: - Unload didn't free VRAM for pooled models. api_model_unload only popped multi_model_manager.models and never touched model_pools, so a model served with max_instances>1 (which lives only in the pool) kept all its instances resident. Now it searches both dicts and calls unload_model(), which cleans up the whole pool + runs gc/empty_cache. Also handles whisper-server models (their own subprocess) by stopping the server. - whisper-server showed as "not loaded". It runs as a subprocess tracked in whisper_servers, not in .models. Fold each running server (id + `audio:` alias) into both the model-loaded-status list and the /admin/api/status loaded_keys, so the models page, dashboard count and per-engine box all reflect it (incl. on a secondary engine). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
model-load/model-unload were proxied to the primary engine, so unloading (or loading) a model that lives on a secondary engine hit the wrong process and silently no-op'd (was_loaded=False). Add front-proxy interceptors: - unload: find the engine whose loaded_models matches the path and forward the request there; fall back to the primary. - load: reuse an engine already serving the model, else the model's engine pin from models.json, else the primary. Registered before the catch-all proxy, mirroring /admin/api/engines. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
- Tasks page: the engine box "N models" count now has a hover tooltip listing every loaded model key on that engine (dotted underline + help cursor); "no models loaded" when empty. - Models page: models loaded on a non-primary engine were shown as idle. /admin/api/model-loaded-status is served by the primary engine and only reported its own pool. Added a front-proxy interceptor that proxies to the primary then unions in every other engine's loaded_models, mirroring the existing /admin/api/engines and /admin/api/status aggregation. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The image's demo tool UIs (video editor :8420, videogen :7790, township :7788, parler TTS :8123) are now individually toggleable at container start — no rebuild. supervisord.conf drives each program's autostart from a CODERAI_TOOL_* env var; the entrypoint seeds defaults (three UIs on, parler off) so the %(ENV_...)s expansion always resolves. run_oci.sh (shipped as the dist coderai-docker runner) gains: --no-tools disable all three demo UIs --enable-tool NAME force-enable one (also turns on parler) --disable-tool NAME disable one; explicit toggles override --no-tools NAME ∈ {video-editor, videogen, township, parler}. README + --help updated. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
coderai disables hf_xet by default (it bypasses the tqdm progress hook and can segfault the worker), but some HF blobs are Xet-only and the plain HTTPS path refuses them with "file too large … install hf_xet" — even though hf_xet is bundled. The first pass now holds that error instead of surfacing it, detects the Xet-required message, and transparently retries with Xet enabled (force_xet → HF_HUB_DISABLE_XET=0). Non-Xet errors are surfaced as before; the existing crash→disable-Xet retry is unchanged. _attempt now returns the held error message as a 4th tuple element. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
When ds4.auto_download is enabled and a deepseek4 request resolves no local GGUF, the downloaded weight variant is now relocated into coderai's GGUF cache (get_model_cache_dir; move on same FS, symlink across devices) and registered in models.json as a text_models entry that mimics the requested ("failed") model's config — backend auto, on-request, enabled and visible (removed from unloaded/to_download). model_name is threaded ds4 backend → ensure_service → ensure_model so the registration mirrors the right entry. Also: settings "Extra ds4-server args" hint/placeholder updated to reflect the auto --kv-disk-dir and SSD-streaming expert-cache sizing (--ssd-streaming-cache-experts), noting Q2_K can fail ds4's CUDA prefill. Diagnosis (no code change): ds4-server's "cuda prefill failed" on the 93GB Q2_K variant is a quant-specific ds4 CUDA bug — the 154GB Q4_K completes prefill fine (verified: "prompt done 434s" vs Q2_K instant failure), with 15.8GB VRAM free either way (not OOM, not cache budget, not coderai). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
ds4-server streams MoE experts and wants the whole GPU for its expert cache, but coderai's modest VRAM estimate for a ds4 model let it co-reside another model — starving the cache so ds4's layer-0 FFN expert encode failed ("gpu layer 0 ffn batch encode failed"). When loading a ds4 model on demand, unload all other models first so ds4-server gets the full GPU. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
- Always keep the CURRENT request (last message) intact and as the very last message after compaction (the compacted history/summary precedes it). - summarize strategy now CHUNKs the older history and summarizes map-reduce (per-chunk then a combined pass) so the summarization prompt can't itself overflow. - If compaction still can't fit the window (e.g. a single huge final message), return HTTP 400 "request too big for context" instead of failing mid-generation. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
When enabled for a model, if the prompt would exceed auto_compact_pct% of the model's context window, the conversation is shrunk to ~65% before generation instead of erroring on overflow. Per-model config (auto_compact / auto_compact_pct / auto_compact_strategy) with three strategies: - drop_oldest : keep system messages + the most recent turns that fit. - keep_head_tail : also keep the first user turn as an anchor + a count note. - summarize : replace the dropped middle with a best-effort LLM summary (generated by the loaded model; falls back to a count note). Token size is a cheap chars/4 estimate; membership uses object identity so value-equal turns don't collide. Wired into the chat path (codai/api/text.py), the model-configure whitelist, and the model config modal UI. Co-Authored-By:
-