Gemma emits tool calls as text — <|tool_call>call:NAME{…} — which the first-pass
streaming loop forwarded straight to the client as assistant content. So the call
showed up as a visible message and only got parsed/executed at end-of-stream.

Add _ToolCallStreamGate: as chunks stream, emit visible content only up to the
first tool-call marker (gemma's <|tool_call>/<|tool_response> plus the common tag
formats), holding back a small tail so a marker split across chunk boundaries
isn't leaked; once a marker appears, emit nothing further. The full text still
accumulates for the existing end-of-stream tool extraction (GemmaParser), which
emits a proper tool_calls delta. The second pass is already buffered+extracted, so
it never leaked.

Also add gemma's "<|tool_call>" to the reasoning/tool split pattern lists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gpu-stats fell through to the generic proxy, which forwards via the long-timeout
client to the primary engine. When that engine is saturated generating (sync
llama.cpp holding its event loop), the request blocked → the dashboard's temp/GPU
polling hung → the whole web UI went unresponsive and the task page stopped showing
temperatures.

Serve it from the front instead, using the already torch-free gpu_detect.gpu_stats()
(nvidia-smi + AMD sysfs, reports every card regardless of CUDA_VISIBLE_DEVICES), run
in a thread so the subprocess never blocks the front's event loop, and registered
before the catch-all so it's not proxied to a busy engine. Light auth (session
cookie / bearer presence) since GPU telemetry is low-sensitivity and full session
validation lives on the engine — which is the component we're decoupling from.

Note: the thermal THROTTLE (pausing a generation when hot) stays in the engine —
it must, to pause that engine's own forward pass — but its stats are now front-served.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Broker (AISBF) chat requests were rejected by the engine with 401 "Invalid API
key" — they're authenticated at the aisbf layer (registration token) and carry no
end-user Bearer, but the engine's BearerAuthMiddleware still demanded one, so every
brokered request came back as a constant ~472-byte error and never ran inference.

Fix (token auto-managed, secure):
- Front marks broker-relayed requests with x-coderai-broker-authed = the internal
  shared token (which the engine already trusts via CODERAI_INTERNAL_TOKEN). Added
  that header to _DROP_REQ so a client-supplied copy is always stripped first —
  unforgeable from outside.
- Engine BearerAuthMiddleware skips the Bearer check when x-coderai-broker-authed
  matches CODERAI_INTERNAL_TOKEN. This is NOT the plain internal token (which rides
  on every front→engine request, direct included), so DIRECT API requests still
  require a real Bearer token.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Previously pooling was gated on gpu_split, so a 2-card same-backend engine (e.g.
2× 3090, or 2× Radeon) still measured only one device for the fit/eviction math.

Now both manager._get_free_vram_gb() and vulkan._pooled_free_vram_gb():
- ALWAYS sum every visible CUDA device (torch honours CUDA_VISIBLE_DEVICES, so it
  is scoped to this engine's NVIDIA cards) → same-backend split is accounted for
  with no flag.
- add AMD card(s) (amdgpu sysfs) only when cross-backend split is on OR no CUDA
  device is visible (a Radeon/Vulkan engine), so a Radeon engine counts its own
  cards and an NVIDIA engine only reaches across to Radeon when split is enabled.

So: 2× NVIDIA → summed across both NVIDIAs; 2× Radeon → summed across both Radeons;
split on → summed across all NVIDIA + Radeon.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

When cross-backend pooling (offload.gpu_split) is on a model can span several
cards, so VRAM math must consider the pool, not one device:

- vulkan.py: new _pooled_free_vram_gb() sums free VRAM across every visible card
  (all CUDA devices + AMD via amdgpu sysfs; no double count). The GGUF auto-offload
  fit decision uses the pooled figure when the model is gpu_split, so layers aren't
  needlessly pushed to CPU just because one card's free VRAM looks small.
- manager._get_free_vram_gb(): when gpu_split is enabled, report pooled free across
  all GPUs (used by the eviction loop + budget checks), so eviction frees/measures
  capacity across both cards. Off → unchanged single-device behavior.

Pooling is aggregate (sum); strict per-device fitting/targeted eviction can be a
later refinement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gpu: per-engine backend isolation (fix docker cross-GPU split) + opt-in per-model cross-backend pooling

Phase 1 — fix the docker-only "model loads on both GPUs":
- gpu_detect.vendor_env detects each Vulkan device's vendor and pins each engine
  to ONLY its own backend's cards by real indices (not assumed 0..n-1). When a
  vendor has no Vulkan device (e.g. NVIDIA in a container that lacks nvidia_icd.json
  because the toolkit only injects it with the graphics capability), the engine
  gets ZERO Vulkan and runs CUDA-only instead of falling back to all ICDs and
  grabbing the Radeon via RADV. Same-backend split (e.g. 2x 3090) is preserved.

Phase 2 — opt-in cross-backend GPU pooling, per model:
- OffloadConfig.gpu_split (default off) + tensor_split ("0.8,0.2", llama.cpp
  device order: CUDA first then Vulkan); global default + per-model override.
- vendor_env(allow_cross=…) exposes the foreign card when enabled; the engine
  supervisor passes it from config.
- manager threads gpu_split/tensor_split (per-model via _raw_cfg, else global via
  global_args) into the GGUF loader; vulkan.py sets llama.cpp tensor_split when on
  and otherwise leaves split_mode=LAYER so same-backend split still works.
- admin model-configure accepts gpu_split + tensor_split per model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

frontproxy:
- _DROP_RESP now drops date/server when relaying engine responses. The front's
  own ASGI server adds Date/Server, so keeping the engine's too produced DUPLICATE
  header lines — which nginx logged as a warning on every request, flooding the
  terminal. Now each appears once.
- Brokered route logs the engine's actual status/bytes/preview (or "NO ENGINE ->
  503") so a brokered request that "doesn't get executed" (instant tiny reply) is
  diagnosable from the log.

coderai-oci:
- Use the host-tailable file log only when it's actually writable. When the
  container runs as --user but the log dir/file was created by an earlier root
  run, it's root-owned; tee then spammed "Permission denied" and the file stayed
  empty. Now we detect that and log to stdout only with a clear note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

launcher: image auto-resolution, --user in-place config persistence, --debug SPEC, install/uninstall image handling

run_oci.sh:
- Resolve the installed coderai image when none is given (single -> use it,
  several -> menu); no longer hardcodes a possibly-wrong tag.
- --debug accepts SPEC as the next token (not just --debug=SPEC) and won't
  swallow an image-looking arg, fixing "image: <debug spec>" mishaps.
- --local puts the runtime dir under ~/.config/coderai-runtime (override
  --data-dir).
- New --user[=UID[:GID]]: run as that user and switch a config dir to an
  IN-PLACE mount so the app's config edits persist (owned by you). Without
  --user, --local/--config-dir stay a throwaway copy. Banner shows user.

Non-destructive config persistence (no startup rewrite):
- codai/cli.py: new --host/--port that override config.server in memory only.
- codai/main.py: apply those overrides right after config load (never written
  to config.json).
- coderai-oci: pass --host/--port to the server and stop rewriting an existing
  config.json (only create one on true first run) — so an in-place-mounted
  config dir is never modified at startup.

install.sh / uninstall.sh:
- install: after loading, offer to remove OTHER coderai images (default no);
  no auto-launch.
- uninstall: resolve the installed coderai image(s) to remove (single ->
  confirm, several -> menu / all / none) instead of a hardcoded tag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderai-oci (in-image): append ${CODERAI_EXTRA_ARGS} to the server argv in both
exec paths. supervisord runs a fixed command, so arbitrary server flags arrive
via this env var rather than argv. (Takes effect after an image rebuild.)

run_oci.sh: --coderai-arg ARG (repeatable, one token each) and --coderai-args
"STR" build CODERAI_EXTRA_ARGS and inject it with -e; new cdr-args banner line.
`--` still goes to the container engine, not coderai.

Docs (AI.PROMPT, dist-bundle README.md/.txt) updated accordingly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

build_oci_image.sh:
- --versioned derives a deterministic tag from codai.__version__ + build mode +
  GPU: local-venv -> coderai:full_<gpu>_<version>, from-scratch ->
  coderai:base_<gpu>_<version> (overrides -t/--tag).
- --gpu all|nvidia|vulkan (default all) sets the <gpu> token; label only — the
  image always bundles both CUDA and Vulkan.

run_oci.sh:
- --host ADDR binds the published port to a specific interface (-p ADDR:PORT:8776);
  default unchanged (all interfaces). CODERAI_HOST stays 0.0.0.0 in-container.
  Banner URL reflects the bind host.

Docs (AI.PROMPT, dist-bundle README.md/.txt, README-RUN.txt):
- Document --versioned/--gpu, additive --nvidia/--vulkan/--all, --vulkan
  auto-libcuda + --with-libcuda, graceful llama-cpp degradation, --host, and the
  new uninstall.sh + confirm gates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docker/backend: graceful llama-cpp load + additive GPU modes + libcuda mapping; admin GGUF batch/slots tuning

Backend robustness:
- vulkan.py catches Exception (not just ImportError) around the llama_cpp
  import: a CUDA-built llama-cpp missing libcuda.so.1 raised RuntimeError/OSError
  that crash-looped the whole server. Now it logs a warning and marks the
  Vulkan/GGUF backend unavailable; CUDA/CPU/ds4 keep working.
- detect_available_backends() reads LLAMA_CPP_AVAILABLE instead of re-importing
  (which re-raised the same error).

Docker launcher (run_oci.sh):
- GPU backends are now additive: --nvidia --vulkan enables both (maps libcuda via
  --gpus all AND /dev/dri). Added --all and --with-libcuda[=PATH].
- --vulkan auto bind-mounts the host's libcuda.so.1 (the bundled llama-cpp is a
  CUDA build), so Vulkan GGUF loads without full --gpus all. Banner shows mode set
  and libcuda status.

Dist bundle:
- New uninstall.sh (removes runner + optional image), wired into make_dist_bundle.
- install.sh + uninstall.sh print what they'll do and confirm before proceeding,
  bypassable with --yes/-y.

Admin GGUF tuning:
- Expose n_batch / n_ubatch / n_seq_max (llama.cpp -b/-ub/-np) in the model config
  UI and apply them in the Vulkan backend to shrink VRAM at the ceiling; n_seq_max
  gated on llama-cpp-python support.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The model-upload endpoint joined a client-supplied filename straight onto
the cache dir, so an admin-authenticated request with a traversal filename
(or upload_id) could write outside it. Reduce both to a safe basename,
reject separators/.., and add a commonpath containment check before
committing the upload.

SessionManager only locked the write half of each load->mutate->save, so
concurrent writers could clobber each other's changes (lost sessions or
tokens). Add update_auth_data(mutator), which holds the lock across the
whole read-modify-write and persists only when the mutator asks to; route
every mutating method (and the token create/delete endpoints) through it.
Read-only callers keep the lock-free load since writes are atomic via
os.replace. While migrating the token endpoints, switch IDs to max+1 (no
reuse after deletion) and to timezone-aware timestamps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Large contexts make the KV cache huge (a 256k q4_0 cache is several GB), which
won't fit in VRAM alongside the weights. llama.cpp can't page KV to disk, but it
can keep it in system RAM via --no-kv-offload. Expose that as a per-model
kv_offload flag (default unchanged = KV in VRAM): set kv_offload=false to pass
offload_kqv=False to llama.cpp, freeing VRAM for big contexts at the cost of
slower decode (KV ops cross PCIe). Also allow the key in the admin model-config
endpoint so it's persistable from the UI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add a canonical codai.__version__ = "0.1.0" as the single source of truth (kept
separate from Config.version, which is the config-schema/migration version). The
admin template renderer injects it as coderai_version, and base.html shows it as a
small "v0.1.0" pill next to the CoderAI logo on every page.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Some quantized fine-tunes (seen with an "Aggressive" Qwen3.6-35B Q4_K_M) collapse
into a runaway repetition loop — emitting a malformed parallel tool-call flood of
1700+ tokens that never terminates — when top_p=1.0 and no repetition penalty are
in effect (exactly the conditions Qwen's own docs warn cause endless repetitions).

Two fixes:

1. Anti-loop generation stop in stream_chat_response: a model-agnostic detector
   normalises away the variable parts of the tail (quoted strings, filesystem
   paths, whitespace) so a loop whose only per-cycle difference is an arg/path
   still reads as periodic, then breaks generation when a short structural unit
   repeats >=5x back-to-back. Tuned to not trip on prose, repetitive code, or a
   legit handful of distinct tool calls.

2. Honor client-supplied repetition controls. The chat paths previously forwarded
   only temperature/top_p, silently dropping repeat/presence/frequency penalty —
   so a caller (e.g. Kilo) setting them per-model had no effect. Plumb them through
   generate_chat_stream / generate_chat to both backends (cuda already accepts
   them; vulkan now does too) with graceful signature fallbacks. Defaults are
   no-ops, so unset clients are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Auto-compaction never triggered: multi_model_manager.config stores the
whitelisted build_runtime_kwargs() dict, which drops the per-model
auto_compact* keys (they survive only under _raw_cfg), so _resolve_compaction
always read the global default (False) and returned None. Read the keys via a
_raw_cfg fallback so per-model compaction config is honoured.

Also rework the over-context handling to count the reply reservation, since the
reply is generated into the same window (prompt + max_tokens <= n_ctx). Four
layers, cheapest first:
  1. fits as-is              -> nothing
  2. overflow within tol     -> trim max_tokens to fit (lossless)
  3. beyond tol & big prompt -> compact history (drop/summarize)
  4. single message too big  -> slice it (summarize its middle, keep head/tail)

The chars/4 estimate undercounts token-dense code/JSON, so trimming to the exact
n_ctx edge could still overflow; inflate the estimate by a configurable
estimate_safety (default 1.15) for all physical-fit decisions.

New CompactionConfig knobs (per-model overridable): tolerance_pct (20),
min_output (512), estimate_safety (1.15). Effective max_tokens is threaded back
to both the streaming and non-streaming generation paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

An engine restart (admin button / config change) previously SIGTERM'd the
process immediately, severing any active SSE stream mid-response — the client
saw httpcore.RemoteProtocolError "peer closed connection without sending
complete message body".

Now restart_engine marks the engine `draining` first: the router stops routing
NEW requests to it (Engine.is_alive() reports false while draining, and the poll
loop can't flip it back healthy), and the supervisor waits up to
server.engine_restart_drain_grace seconds (default 30, 0 = immediate) for the
in-flight count to reach zero before killing the process. Stragglers past the
grace window are still bounced.

In-flight is tracked per engine in the front proxy: proxy() increments on send
and decrements once the streamed response is fully drained (or the send failed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Qwen-style chat templates pre-fill the opening <think> in the prompt, so the
model emits only the reasoning body + a bare closing </think> — and they think
by DEFAULT regardless of the API enable_thinking flag. The old paired-tag
reasoning extractor missed the bare close, leaking the whole thought (and the
</think>) into content and conversation history.

- extract_reasoning_content: handle a bare </think|/thinking|/thought> with no
  opening tag (treat the prefix as reasoning).
- streaming: a chunk-safe reasoning gate routes the thought into
  delta.reasoning / reasoning_content until </think>, then flips to content;
  tool extraction runs on the post-</think> answer only.
- non-streaming: extract reasoning, set message.reasoning(+_content), clean
  content; tools parsed from the answer.
- activate whenever the model auto-thinks (qwen3/qwq/deepseek-r1/… name) OR
  reasoning is explicitly enabled — not just on the API flag.
- configurable suppression: per-model `suppress_reasoning`, or per-request via
  the standard reasoning:{exclude:true} / reasoning_effort:"none" /
  suppress_reasoning fields. Emits both `reasoning` and DeepSeek-style
  `reasoning_content` for client compatibility.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Optional CPU-temperature source for thermal control. Not essential — the
thermal monitor reads CPU temp from psutil and the kernel's /sys/class/thermal
zones first (both work in-container); `sensors` is only a last-resort text
parse for hosts whose sysfs doesn't expose a CPU zone. Added to both the
from-venv and from-scratch runtime stages for completeness.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The plaintext <tool> rescue could turn a failing 2-bit model's repetition
loop (<tool>glob</tool><tool>glob</tool>… / bare names, no args) into a flood
of bogus tool calls. Harden it: reject a batch with >6 <tool> blocks (that's
model degeneration, not many real calls) and drop any bare <tool>name</tool>
that carries no key: value argument (the spam signature). Genuine single/few
calls with arguments still parse; combined with the existing trailing-action,
declared-name, and DeepSeek-only scoping.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- ds4 kv janitor: a checkpoint is deleted only when ALL hold — untouched by
  max(mtime, atime) for the age (so a checkpoint ds4 merely READS, which bumps
  atime not mtime, is spared); not currently open (fd/mmap) by a ds4-server;
  and ds4 is not serving any request. New in-flight counter on Ds4Backend
  (any_request_active) gates the sweep.
- settings: "Download a default DeepSeek V4 model" — select + button backed by
  new /admin/api/ds4/default-models catalog (q2-imatrix / q2-q4 / q4 / mtp from
  antirez/deepseek-v4-gguf). Reuses the normal downloader, which flattens the
  gguf into the cache and surfaces it in the model list; live progress.
- parser: rescue the degraded plaintext <tool>name arg: value</tool> form that
  heavy quants (ds4 q2-imatrix) emit when they can't reproduce DSML. Scoped to
  DeepSeekParser only (never the shared ToolCallParser, so other families are
  untouched), requires a DECLARED tool name, plaintext-only inner, and the
  block(s) to be the message's trailing action — so a <tool> example inside a
  prose reply is not misread as a call.
- settings: corrected ds4 perf note (i-quants/Q2_K fail CUDA prefill; use Q4_K+).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The --kv-disk-dir parse loop used `for _i, _t in ...`, making `_t` a local
in main() and shadowing the module-level `import threading as _t`, so the
later `_t.Thread(...)` raised UnboundLocalError and crash-looped both
engines. Renamed the loop var to `_tok`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- ds4: configurable janitor that age-prunes the on-disk KV-checkpoint cache
  (--kv-disk-dir, defaulted to <offload>/ds4-kv). New Ds4Config fields
  kv_cache_cleanup_enabled / kv_cache_max_age_hours (7d) /
  kv_cache_cleanup_interval_minutes (6h); new codai/api/ds4_kv_janitor.py
  reuses the tmp_janitor sweep (newest-mtime, so active sessions are spared),
  started from main.py only when ds4 + cleanup are both on. Settings UI +
  get/save wired.
- ds4: corrected the perf note — i-quants (IQ2/IQ3) and Q2_K load but fail
  ds4's CUDA prefill (gpu layer 0 ffn batch encode failed → empty reply);
  use K-quants Q4_K and up.
- models: pressing Enter in the HuggingFace search field now runs the search.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Auto-compaction can now summarize with a DIFFERENT model than the one
serving the request, with a global default (config.json `compaction`) and
a per-model override (models.json `auto_compact_model`). Empty = the
request's own model, as before.

- config: new CompactionConfig (enabled/pct/strategy/model) + round-trip
- text.py: resolve effective settings (per-model over global), resolve the
  summarizer LAZILY (only when actually over threshold, so a separate model
  isn't loaded on every request); map-reduce the dropped history into chunks
  sized to the CHOSEN summarizer's own context, reducing iteratively until
  it fits; stream status + live per-chunk progress to the client as content
  deltas (queue-bridged from the summarizer's callback)
- admin: global compaction card (settings) + per-model summarizer dropdown
  (models, shown only for the summarize strategy)

Raw two-pass path is skipped (prompt is built from system + last user turn).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add a "tuning ds4 for performance" note to the global ds4 Settings section and a
condensed inline note to the per-model ds4 streaming section on the Models page:
NVMe/SSD placement (≈10× prefill), capping the expert cache by count, sizing the
VRAM reserve, DS4_CUDA_WEIGHT_ARENA_CHUNK_MB=512, avoiding DS4_CUDA_WEIGHT_CACHE,
and that decode of a model larger than VRAM is streaming-bound (smaller quant is
the real fix). Distilled from measured tuning on the 154GB DeepSeek-V4 MoE.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The streaming tool-content gate withheld <tool>/<|tool_call>/call: markers but
not DeepSeek V4's native <｜DSML｜tool_calls>… block (｜ = U+FF5C), so during a
streamed tool call the raw markup reached the client token-by-token as visible
content (even though the post-stream parser extracted the tool_calls correctly).

_gate_tool_content now withholds everything from the first <｜DSML｜ marker to the
end (dropped on final, surfaced as structured tool_calls), and the trailing-
partial hold list includes the DSML open tag so a marker split across chunks
doesn't leak its leading chars.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Per-model ds4 tuning (these vary by quant/size/context, so they belong on the
model, not globally):
- Optional `ds4` block on a model entry overrides the global Ds4Config for
  ssd_streaming / expert_cache_reserve_gb / extra_args / extra_env; unset fields
  inherit the global config (the default/template). Ds4Backend looks up its own
  model entry and applies the overrides via dataclasses.replace.
- admin: api_model_configure accepts + normalizes the per-model `ds4` block,
  dropping it when empty.
- models page: a "ds4 streaming" section shown only when ds4 is enabled globally
  and the model is a deepseek4; n_ctx stays the context knob.

Fix garbled / truncated ds4 replies: the streaming reader used
iter_lines(decode_unicode=True), which decodes each network chunk independently
and corrupts a multibyte UTF-8 char split across chunks ('—' -> 'â'); the broken
JSON then made json.loads fail and the token was silently dropped (truncated
tails). Parse the SSE byte stream and split on the b"\n" byte (never inside a
UTF-8 sequence), decoding whole lines; also flush a final newline-less line.

UI: slow-reply notice reworded to "Waiting for model reply..." with a trailing
newline so the real reply starts on its own line.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

DeepSeek V4 (ds4) emits native tool calls as <｜DSML｜invoke name="…">
<｜DSML｜parameter name="p" string="true">val</｜DSML｜parameter></｜DSML｜invoke>
(the ｜ is U+FF5C). No parser recognised this, so ToolCallParser returned None
and the raw markup leaked to the client as content even though ds4 reported
finish=tool_calls.

- parse_deepseek_dsml_tool_calls(): extract (name, args); string="false" params
  are JSON-decoded, others kept as strings; ASCII | tolerated.
- Wired into DeepSeekParser and ToolCallParser.extract_tool_calls (the live path).
- strip_dsml_tool_calls(): drop the DSML block from displayed content in both
  strip_tool_calls_from_content paths. Guarded by 'DSML' in text -> no effect on
  other models.

Also reword the slow-reply notice from "Waiting for model to load..." to
"Waiting for model reply..." (the model is usually loaded, just slow).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

ds4-server exposes several CUDA tunables only via environment, not CLI flags.
By default ds4 reserves half the card for non-cache use and allocates the model
weight arena in 1792 MiB chunks — both starve / OOM the streaming expert cache
on small-weight MoE models served from SSD.

Pass an explicit env to ds4-server (Popen now sets env=) with:
  - expert_cache_reserve_gb: typed knob -> DS4_CUDA_STREAMING_EXPERT_CACHE_RESERVE_GB
    (0 = leave ds4's default).
  - extra_env: free-form KEY=VALUE passthrough for the rest, e.g.
    DS4_CUDA_WEIGHT_ARENA_CHUNK_MB=512 to shrink the weight-arena chunk so it
    fits a heap fragmented by the expert cache.

Both surfaced in Settings (config + admin GET/POST + UI), default to no-op so
behaviour is unchanged unless set.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add a GitHub-flavored README.md alongside the existing README.txt in the
all-in-one docker distribution bundle, and have make_dist_bundle.sh stage it
so future builds include both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- models page: the multimodal-projector select now defaults to None unless a
  projector is a strong, unambiguous name match. Scores only distinctive tokens
  (drops generic words + quant tokens, keeps size tokens like 14b), requires
  covering at least half the model's tokens, and rejects ties. Stops a lone
  shared family token from pairing the wrong-size projector.
- task page: the per-engine loaded-model hover now lists each model once by its
  canonical id instead of its aliases (auto gguf stem, explicit alias, type
  prefix). engines_list() resolves loaded keys via the pin index's new model_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

By default a per-model engine pin is a hard constraint: if the pinned
engine is down/incompatible the request fails (no duplicate on another
card). Add an `engine_fallback` model-config flag (admin form checkbox +
persisted in models.json) that opts into the old behaviour — fall back to
a compatible engine when the pin can't be honoured. A pinned engine
that's merely busy-but-alive is still routed to (queues) in both modes;
fallback only applies when it's actually down or can't serve the model.

Threads pin_fallback through pick_engine; the front reads engine_fallback
via _load_pins/_model_info.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

An engine mid-generation is GIL-blocked and fails the health poll, so it
reads as unhealthy. pick_engine required e.healthy at every step, so a
second request for a model pinned to that engine fell through to the
least-loaded engine — which loaded a DUPLICATE copy (and ignored the
model's configured n_ctx, e.g. 2048 vs 32000 → "exceeds context window").

Honour the pin (and the assigned owner) when the engine is alive but
transiently busy: route there so the request queues on its gen-lock and
the owner handles serialization/eviction. Only fall back to another engine
when the owner's process is actually dead. Adds Engine.is_alive() (process
liveness) and registry.engine_owning() (health-agnostic owner lookup).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The gemma loose object/array parser could spin forever: when a recursive
value parse can't advance past a stray delimiter (e.g. '}' where ']' was
expected, as in the broken `{"files":[{"path":"x"]}}` a looping Gemma
finetune emits), the array/object loop kept iterating without consuming
input. parse_gemma_native_tool_calls and the new parse_tool_tag_json_calls
both feed model output through this parser, so a malformed tool call would
hang the request (not just be missed). Add a forward-progress guard to
both loops: bail when an iteration consumes no input. Best-effort recovers
the tool name + good fields from malformed JSON; clean input is unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

collect_models unioned only registry.healthy() engines and re-fetched
each engine's /v1/models live. An engine loading a model is GIL-blocked
and misses the 2s health poll, so it goes "unhealthy" and ALL its models
— including a freshly-added one — drop out of the aggregated /v1/models
until the load finishes. A client (e.g. the kilo model script) polling
during a load then sees models vanish. Cache each engine's last-good
/v1/models and, when it's transiently unhealthy/unreachable, serve that
cached list instead of dropping it. The models are still assigned to the
engine and will serve once it's free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Some Gemma finetunes emit tool calls as <tool>{"name":..,"arguments":..}
using <tool> (or <tool_call>) as BOTH delimiters — no closing slash —
and sometimes append a stray quote. Every existing <tool>…</tool>
pattern requires a slashed closer, so GemmaParser found nothing and the
call was returned as plain text: kilocode saw no tool_call and showed no
reply. Add parse_tool_tag_json_calls, which extracts the brace-balanced
object after each marker via the tolerant loose parser (so trailing junk
and stray quotes don't break it) and reads name + arguments/parameters,
restricted to declared tool names and de-duplicated. Wired into
GemmaParser before the generic fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Whether a model rejects the 'system' role is a property of the chat
template baked into the specific GGUF, not the architecture: the gemma-2
template and the official gemma template raise "System role not
supported", while 'heretic' gemma4 quant conversions ship a permissive
template that accepts system. Detect from the embedded
tokenizer.chat_template (raise_exception/"system role") and fold only
when it actually rejects system; fall back to architecture (Gemma) when
no template is readable. Avoids needlessly folding permissive Gemma
models while still covering gemma-2-9b and strict non-Gemma templates.
The runtime "System role not supported" retry remains as a safety net.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Gemma's chat template has no 'system' role; llama.cpp raises "System
role not supported" and the generation fails (the Kilo client always
sends a system prompt). On that specific error, retry with the system
message(s) folded into the first user turn — Gemma's own convention,
and a no-op for models that accept system. Handles both streaming and
non-streaming paths and preserves multimodal (list) content.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

VulkanBackend.load_model treated any id that isn't a '.gguf' path / file
/ URL as a HuggingFace repo to download. A configured gguf addressed by
its automatic alias ('coe-…-q4_k_m', no extension) thus 404'd against
the Hub instead of loading the local file. Resolve the alias via
_resolve_local_gguf (configured-entry + cache-dir match) first; only fall
back to the HF path when no local gguf is found.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

pick_engine honours the front's assignment (radeon) only if the engine
can_serve the request's required capability. But _required_cap derived
that capability from the bare alias 'coe-…-q4_k_m' — no literal 'gguf' —
so required_capability returned 'transformers' (CUDA-only). radeon is
gguf-only, failed can_serve, and the request fell through to the default
engine (nvidia), even though compute_assignment had correctly placed the
model on radeon (it sees the full '…-q4_k_m.gguf' path).

Resolve the model's configured path in _load_pins (now indexed by the
.gguf-stripped stem too) and, when the name heuristic yields
'transformers' but that path is a .gguf, correct the capability to
'gguf'. whisper/ds4 precedence is unchanged. Combined with the registry
stem-matching, a bare-alias request now lands on the owning Vulkan/AMD
engine.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>