- 26 Jun, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authoredThe 0.1.24 nesting blindly appended the bundled sub-app name to the incoming X-Forwarded-Prefix, which double-counted it when the outer proxy mounts the app under the SAME name and proxies to our /<app>/ path (topology B): outer sends "/township" -> container emitted "/township/township", so every township fetch() 404'd ("Not found" -> JSON parse error in the UI). Per-app maps now only append our name when the incoming prefix doesn't already end in it, so both topologies work: outer-wraps-everything ("/ai" -> "/ai/town ship") and outer-mounts-same-name ("/township" -> "/township"). Direct (no outer proxy) still yields "/township". Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
-
Stefy Lanza (nextime / spora ) authored
Internal container nginx is now chain-aware: it prefers the outer proxy's X-Forwarded-Proto/Host/Prefix and nests bundled sub-app prefixes under any outer prefix (outer /ai + /township -> /ai/township). Fixes characters/ environments thumbnails and other absolute/sub-path URLs 404ing when the all-in-one container runs behind a second reverse proxy. Documented the required outer-proxy headers in docs/reverse-proxy-nginx.md. LoRA training loads its base pipeline outside the model manager (and unloads all manager models first), so the engine reported 0 loaded models mid-train. Surface the active training base model via active_training_model() so the engines card reflects the busy GPU. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- Show which engine each task runs on (badge on active AND history rows; backend already tags t.engine) and an "● processing" badge on the engines card (new inflight/processing fields in /admin/api/engines). - Image-gen advancement showed only "working…": poll() short-circuited to the synthesized task list whenever the primary had any in-flight request, dropping the engine's real step/total. Now it quick-polls the engine first (image/diffusers gen releases the GIL so it answers with real progress) and only falls back to the synthesized list when a GIL-bound text gen times out. - Surface thermal cooldown on the task entry itself: _merge_engine_tasks marks tasks on a cooling engine with cooling/cooling_message (the row already renders it). Co-Authored-By:
-
↔ Stefy Lanza (nextime / spora ) authoredLoRA add_adapter() crashed: peft's dispatch_awq does `from gptqmodel.nn_modules.qlinear.gemm_awq import AwqGEMMQuantLinear`, but gptqmodel 7.1.0 renamed that class to AwqGEMMLinear. peft calls dispatch_awq for ANY non-bnb target when gptqmodel is installed, so the failed import broke every add_adapter (SDXL/Wan/Z-Image), not just AWQ models. Add _ensure_peft_awq_compat(): alias AwqGEMMQuantLinear -> AwqGEMMLinear so peft's import succeeds and falls through to the correct dispatcher. Called before every add_adapter() (sd15/sdxl/dit/wan). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
QLoRA training reached VAE encode then failed: the ZImagePipeline loads the VAE in bf16, but the image tensor was fed as float32 -> conv2d "Input type (float) and bias type (BFloat16) should be the same". Feed the VAE its own weight dtype. Co-Authored-By:
-
- 25 Jun, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authored
QLoRA training crashed in the Qwen3 TEXT ENCODER: the unsloth build quantizes it too, and loading components piecemeal (AutoModel/AutoencoderKL by subfolder) left bitsandbytes' 4-bit quant-state unreconstructed -> "FP4 quantization state not initialized" / AssertionError in Linear4bit.forward. - Load all components via ZImagePipeline.from_pretrained (the proven inference loader, which loads each 4-bit component correctly), then LoRA-train only the transformer. VAE/text-encoder are encoded once then freed (4-bit modules can't .to('cpu'), so drop refs + free). - Disable gradient checkpointing for the 4-bit path: the recompute can desync bnb's quant-state. At 512/batch-1 the 4-bit base + activations fit without it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Training on the full bf16 Tongyi-MAI/Z-Image-Turbo was extremely slow — a ~10-min download plus a heavy bf16 model that doesn't fit cleanly on 24 GB. Switch _train_dit to QLoRA: load the transformer in 4-bit (frozen, ~4 GB, no CPU offload) and train the LoRA on top. This trains directly on the already-cached 4-bit (e.g. unsloth) build — no redirect to the full model, no download. - Load transformer with diffusers BitsAndBytesConfig (nf4); an already-4-bit checkpoint (embedded quant config, e.g. unsloth) is loaded as-is via fallback. - Enable gradient checkpointing and force the input-embedding (all_x_embedder) output to require grad so QLoRA grads reach the attention LoRA layers; hooks removed at job end. - Drop the quantized-base -> full-model redirect added earlier. LoRA still applies to the quantized model at inference (identical architecture). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Env/fighter LoRA training for Z-Image misrouted to _train_sdxl (CLIPTokenizer crash: "NoneType cannot be interpreted as an integer"). Cause: _resolve_base_model_path returns the HF id verbatim (e.g. unsloth/Z-Image-Turbo-unsloth-bnb-4bit) when the model has no local `path`, so the isdir(base_path/transformer) DiT check was False and it fell through to the SDXL/SD15 trainer. - Detect Z-Image by NAME (id contains z-image/zimage) in addition to the diffusers-config class name, so an HF-id base routes to _train_dit. - _train_dit: when the base is a pre-quantized (bnb/nf4/4bit) build, train against the full Tongyi-MAI/Z-Image-Turbo instead (a 4-bit checkpoint isn't a clean full-precision LoRA base); the LoRA still applies to the quantized model at inference. Overridable via lora_train_base_model. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
is_available() reported false in the running engine even though gptqmodel + fast kernels import fine in a fresh process. Two causes: 1. capabilities() cached a DEGRADED result (gptqmodel imported but the inner gptqmodel.utils.backend BACKEND import transiently came up empty, e.g. when the first call landed mid model-load). That empty-backends result stuck for the whole process life, so the settings page said "GPTQModel not installed" until restart. Now a degraded (available-but-no-backends) result is NOT cached — re-detect next call; only a clean positive or a genuine ImportError is cached. 2. is_available() gated on a SPECIFIC fast kernel being detected. GPTQModel always has a Triton/torch fallback and picks the kernel at load, so availability now gates only on gptqmodel importing; backends stay informational. Also: /admin/api/quantize-capabilities re-detects live (capabilities(refresh=True)) so the settings page never serves a stale cache. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
1. Model load/unload (and every is_admin-gated front action) returned 401: is_admin probed prim.url + /admin/api/status on the engine, but that route is front-only (removed from the engine in def78c18), so it 404'd → never 200 → Unauthorized. Add an engine-side admin-gated /admin/api/whoami and point is_admin at it. 2. Township reference generation (characters/environments pages — _run_regen_job and _run_create_profile_job) hardcoded 768x512/512x512 + 28 steps, ignoring the Run page. Add _ref_gen_res_steps(args) (honors keyframe_size/keyframe_steps, re-read from the saved config so edits apply without restart) and use it at the reference generators; generate_character/generate_environment now forward `steps` (the server already accepts it). Keyframes/match videos already honored the config. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai's trainer only targeted SDXL/SD1.x U-Nets, producing unet.-prefixed LoRAs that silently no-op on Z-Image's DiT (ZImageTransformer2DModel, transformer. keys, Qwen3 text encoder) — "No LoRA keys associated to ZImageTransformer2DModel found with prefix='transformer'". The lora_train_base_model=SDXL workaround can't make a LoRA that loads on Z-Image. Add _train_dit: a native flow-matching DiT LoRA trainer for Z-Image, reusing the existing job/progress/checkpoint/queue infra (modeled on the Wan video DiT trainer, which is video-only). All deps (ZImageTransformer2DModel, ZImagePipeline, PEFT, Qwen3) are already in the main venv — no ai-toolkit, no separate venv. Reverse-engineered from diffusers pipeline_z_image.py so training matches inference: chat-template + Qwen3 hidden_states[-2] masked per-sample embed list; AutoencoderKL latents scaled (lat-shift)*scale; list-based transformer I/O with normalized timestep (1000-t)/1000; RAW target = x0-noise (Z-Image negates the model output); timesteps sampled from the discrete turbo schedule (set_timesteps mu-shift) to keep the distilled model sharp; save via ZImagePipeline.save_lora_weights (transformer. keys) so _apply_loras' load_lora_weights applies it. _train_lora_sync routes a Z-Image DiT base to _train_dit (detected via model_index/transformer config _class_name); other DiTs (Flux/SD3) still raise with guidance. v1 — needs one validation train; recipe + knobs in docs/zimage-lora-training.md. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
After the GGUF-isolation split, a torch engine and a gguf engine share one NVIDIA card but each can only evict its OWN models. So loading an image model on the torch engine while the gguf engine still holds a resident GGUF text model failed: local eviction "freed 0.0 GB ... VRAM held elsewhere" → CUDA OOM. Fix — co-located VRAM release: - front passes each engine its same-GPU siblings' internal URLs via CODERAI_COSITED_URLS (matched by identical CODERAI_ENGINE_GPUS selectors). - engine registers an external_vram_releaser that POSTs to each sibling's new /internal/evict-vram when local eviction can't free enough. - /internal/evict-vram → manager.release_idle_vram(): evicts all idle (non-busy) models and returns GB freed; busy/actively-serving models are left alone. Symmetric: the gguf engine can likewise reclaim VRAM from the torch engine's idle diffusers models. Driver-level free VRAM (cross-process) is re-checked after each releaser, so the loader proceeds once the sibling has freed enough. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two fixes after the GGUF process-isolation split: 1. Routing: a model with engine=nvidia in its config (a hard pin) but a GGUF body 503'd, because the torch nvidia engine dropped the `gguf` capability. pick_engine now falls back to the co-located "<name>-gguf" sibling (same card) when the pinned engine can't serve the required capability (and symmetrically for a gguf-pinned model needing transformers). The pin's intent — that physical card — is honoured. 2. Config-save visibility: the cached-models scan is stale-while-revalidate, so the first read after an invalidation returned stale and refreshed in the background — a saved config didn't appear until a second refresh. _invalidate_cache_scan now sets force_sync so the next read recomputes synchronously (fresh immediately). The system worker invalidates on reload-config, so a save shows on the next page load. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The post-delete redirect used `(window.ROOT_PATH||'') + '/matches'`, but window.ROOT_PATH was never defined, so behind the nginx /township mount it redirected to the bare /matches — which hits the coderai front, not the township tool → 404. The mount shim wraps fetch()/EventSource() to add the prefix but not location.href assignments. Define window.ROOT_PATH=P in the shim so the redirect (and any future ROOT_PATH use) resolves to /township/matches; with no prefix (direct access) it stays unset and the bare /matches path is correct. Co-Authored-By:
-
- 24 Jun, 2026 16 commits
-
-
Stefy Lanza (nextime / spora ) authored
llama.cpp's CUDA backend and PyTorch sharing one process corrupt the CUDA context: after a GGUF model runs on an NVIDIA card, the next torch kernel dies with "CUDA error: invalid argument". Seen in a township match — a gemma GGUF served a chat completion on the nvidia engine, then the next Z-Image (diffusers) image gen crashed in its Qwen3 text-encoder's sdpa mask (padding_mask.all()). Every Z-Image run before llama.cpp touched CUDA succeeded; the first one after crashed. Evict+swap in one process can't fix it — torch holds the (now-corrupted) context for the process lifetime. Fix: when GPUs are auto-detected and server.isolate_gguf_engine is True (default), each NVIDIA torch engine gets a co-located sibling gguf engine on the SAME card (own process -> own CUDA context). The torch engine drops the `gguf` capability and serves transformers/diffusers; the sibling (backend=nvidia so GGUF takes the proven CUDA-llama path, capabilities={gguf}) serves llama.cpp. Routing is already capability-based, so GGUF goes to the gguf engine and HF/diffusers to the torch engine. Both are real engine subprocesses, so the front's routing, VRAM/eviction and thermal (cooperative pause + SIGSTOP on the process group) apply unchanged — and SIGSTOPping the gguf engine no longer freezes the torch engine. Ignored when engine_specs is set (declare the split yourself). Disable with server.isolate_gguf_engine=false. Design + trade-offs (extra torch context, no cross-engine eviction on a shared card) in docs/gguf-process-isolation.md. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Trained LoRAs, characters, environments and voices are resolved from the legacy HOME-based dir ~/.coderai, not from --config. In the container the user's home (e.g. /home/ubuntu) isn't where config is mounted ($CONFIG_DIR/coderai), so those lookups missed the mounted data and image-gen failed with "LoRA '<name>' not found on server". Symlink ~/.coderai -> $CODERAI_CONFIG_DIR/coderai so HOME-based and --config lookups agree, removing the need for an extra per-deployment --map. Acts only when safe (a symlink, missing, or an empty dir) so a user-mounted volume or real data is never clobbered. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The models page reads /admin/api/cached-models (served by the system worker), whose _scan_caches() merges each model's saved config (precision/quant/ctx) into the cards. The scan freshness check _scan_signature() only fingerprinted the HF/GGUF cache directories, never models.json, and model-configure never invalidated the scan. A config save rewrites models.json but touches no cache file, so the signature stayed unchanged and the scan was served stale for the full TTL (600s) -- the engine reloaded but the front showed old config until a full restart. - _scan_signature(): fold models.json mtime into the signature so any process serving cached-models re-scans on the next read after a save. - system worker /internal/reload-config: call _invalidate_cache_scan() after rebinding config, so the front's reload-push refreshes the scan immediately. - engine /internal/reload-config: same invalidation for single-process / system-worker-down fallback. Also adds docs/dtype-auto-selection.md (planned design: model-native dtype default read from the checkpoint, precision as explicit override, FA2 fp32 guard) and ignores build artifacts (venv_build.log, .build.pid). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Audit of the nvidia text loader (cuda.py) vs the shared hf_loading.py found real config drift — the same model config behaved differently on the text path: 1. precision IGNORED — cuda.py hardcoded torch_dtype=float16. Now the manager passes the per-model and cuda.py resolves it via resolve_dtype() (default fp16 on CUDA / fp32 on CPU when unset, preserving current behaviour). 2. 4-bit compute dtype hardcoded float16 → now follows the resolved precision (_make_bnb_config takes a compute_dtype). 3. key was ignored — the manager only read . Now it honours flash_attn / flash_attention (per-model) and the global global_args.flash_attn (offload.flash_attention). 4. offload_buffers was only set on the disk-spill path → now also on the GPU+CPU device_map ladder (with offload_folder), so CPU offload doesn't pin activation buffers on the GPU and OOM the forward pass. 5. global max_ram_gb now clamps the CPU offload budget (central _get_gpu_memory_map_with_limit + the disk fallback), matching hf_loading. Diffusers-only items (component_quantization, GGUF components, sdcpp flash flags) are correctly N/A to the AutoModelForCausalLM text path. Bump version to 0.1.9. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Loading an HF model (e.g. Qwen3.5-9B, 4-bit) failed with 'Permission denied: ./offload'. Cause: the model's per-model offload_dir was the relative './offload' (a stale auto-saved default), which _cfg_or_global lets win over the global config; './offload' resolves to the CWD = the READ-ONLY /opt/coderai/app tree in the image. The config WAS respected — a relative offload path is just meaningless where the CWD isn't writable. * hf_loading.resolve_offload_dir(): an absolute offload_dir is respected as-is; a relative/empty one INHERITS the configured GLOBAL offload directory (global_args.offload_dir) when absolute, then CODERAI_OFFLOAD_DIR, then the user cache — never the CWD. Applied in the manager (both load sites, always passed), hf_loading, and defensively in the cuda backend. * main.py + entrypoint: a container-writable CODERAI_OFFLOAD_DIR (=/cache/offload, created by the entrypoint) is used when the GLOBAL config is still the bare './offload' default; explicit config wins. * run_oci.sh: forward HF_TOKEN / HUGGING_FACE_HUB_TOKEN from the host env so the engines authenticate to the HF Hub (the 'unauthenticated requests' warning) for higher rate limits + gated models. HF_HOME/cache dir was already honoured (main.py from config.models.hf_cache_dir). Bump version to 0.1.8. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The point of the keepalive is to hold the client connection open while the engine is stuck loading — so 'silent' must NOT fall back to the dead legacy path (which sends nothing until the engine's first byte and lets short-timeout clients/proxies disconnect). Now ALL streaming inference goes through the front keepalive path: * silent — SSE comment lines (': …') only: keeps the socket alive, emits no chunk/content/status (invisible to event parsers). * invisible — empty-content chunk + x_queue_info (default). * visible — visible status text. * thinking — reasoning channel (when mode != silent). The front stays responsive even when the engine is GIL-blocked, so it can drive these regardless of engine state. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Clients hitting the public API directly (e.g. township via nginx) disconnected during long waits: the direct proxy() path acquired the front queue slot silently and awaited the engine's first byte (model load) with no output. The broker path already kept alive; the direct path now does too. For a STREAMING inference request, commit to a 200 text/event-stream up front and emit keepalive while acquiring the queue slot and during the engine's model load / not-ready retries, then relay the real stream (token-counting for the Tasks page), ending cleanly if the engine dies mid-flight. Configurable mode (models.wait_status_mode, global default 'invisible'; per-model override via the models.json entry): * invisible — empty-content SSE chunk + x_queue_info (holds the connection; no content pollution) * visible — short visible status text (appears in the content) * silent — nothing (legacy path) When thinking is enabled the keepalive goes on the reasoning channel instead (no pollution), unless mode is silent. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
manager (record_vram_delta): record the FULL footprint after a load — including offloaded loads (no longer skipped) — so the next load skips the OOM→move-to-RAM retry dance: * measured_vram_gb — GPU VRAM delta + reserve (GPU portion; eviction uses it) * measured_ram_gb — host-RAM delta (the CPU-offloaded layers) [new] * measured_n_gpu_layers — the layer split llama.cpp settled on [new] total need = vram + ram. All gated by force_vram_update (else only when used_vram_gb unset). New _learned_n_gpu_layers(): when n_gpu_layers is auto/-1, reuse the learned split so a reload jumps straight to the config that fit. Both load sites snapshot host RAM and read the backend's settled n_gpu_layers. frontproxy: * graceful streaming proxy: when an engine dies mid-stream (e.g. a CUDA ggml_abort SIGABRT on a VRAM-OOM decode), end the relayed stream cleanly instead of throwing an unhandled ASGI exception (noisy traceback). * task actions (cancel/interrupt/pause/resume/restart/DELETE) now route to the engine that OWNS the task — fan out to every live engine (each validates the same session cookie), first success wins — fixing 'Task not found' for tasks on a non-primary engine; front-only synthetic ids handled locally. Bump version to 0.1.7. -
Stefy Lanza (nextime / spora ) authored
* coderai-docker (run_oci.sh): new --tool-arg TOOL VAL (one token) and --tool-args TOOL STR (whitespace string), repeatable, for video-editor | videogen | township | parler. They accumulate into CODERAI_<TOOL>_ARGS env vars that supervisord appends to each tool's command line. Documented in --help (plus the --map dir-or-file note and bring-your-config examples), and shown on the startup banner (tool-args:). * entrypoint: default+export the four CODERAI_*_ARGS (empty) so supervisord's %(ENV_...)s never fails on an undefined var; pre-create /cache/video_editor/sessions. * supervisord: append %(ENV_CODERAI_*_ARGS)s to all four tool launchers; the video editor now runs with --session --session-dir /cache/video_editor/sessions so editor-state recovery is ON by default and persists on the cache volume. * video_editor.py: new --session-dir to place session state/assets at a persistent/mapped path (defaults to ~/.cache/... when omitted). Bump version to 0.1.6. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai always runs as front + engines, so drop the single-process special-casing in execute_api_request: always hand the sub-request to the front (the single API), which routes it to the engine that owns the target model. Critically, remove the on-error fallback to in-process dispatch — that would silently run the model on the WRONG engine and hide a routing failure. If the front can't be located/reached it now returns a 502 the caller surfaces. Drops the now-unused os import. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
* pipelines.py: /v1/pipelines/* (image-to-video, story, video-dub, audio-dub) chained their steps by calling the image/video/text/TTS handlers DIRECTLY in-process, forcing every modality onto whichever engine received the pipeline request. In a multi-engine deployment each model may live on a different engine. Route every sub-step through the front via execute_api_request (same fix as characters/environments) so each lands on the engine that owns that model; in-process fallback for single-process mode. Drops the unused TestClient import. * supervisord.conf: the bundled tools (video_editor, videogen, township) now use --base-url http://127.0.0.1:8776 — the PUBLIC nginx port — so they talk to coderai exactly like any external client (nginx already proxies the full API with 4G bodies / 1h timeouts). The front's own bind stays CODERAI_PORT=18776. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/characters/generate and /v1/environments/generate produced their reference images by calling execute_internal_request(request.app, '/v1/images/generations') — an IN-PROCESS self-dispatch on whatever engine handled the request. Two bugs: 1. It re-entered the engine's own _InternalAuthMiddleware without the internal token, so in multi-engine mode it 403'd ('engines are reachable only through the front proxy') — image gen failed while plain text worked (text never self-dispatches). Township (a normal API client) hit exactly this. 2. Even with a token it would run the image model on the WRONG engine: the image model may be assigned to / pinned on a different engine than the one serving the characters/environments request. Add execute_api_request(): behind a front (CODERAI_INTERNAL_TOKEN set) it routes the sub-request THROUGH THE FRONT (the single public API) at 127.0.0.1:<front port>, so the front picks the engine that owns the image model and stamps the internal token itself; the caller's Authorization is forwarded so that engine authorises it like any client call. Single-process mode still dispatches in-process (middleware is a no-op with no token). The client never sends internal headers — it just talks to the one API endpoint. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Mirror the township config fix for the bundled video editor: * supervisord: pass --config /cache/video_editor/video_editor.config.json so a bind-mounted (or cache-persisted) config auto-loads — previously it only auto-loaded video_editor.config.json from the baked cwd, which isn't mappable. * video_editor.py: a MISSING --config file is no longer fatal (was raise SystemExit) — start with defaults; the web UI Save creates it. * entrypoint: pre-create /cache/video_editor so the config dir exists for auto-load + web-UI Save (persists on the standard /cache volume by default). * run_oci.sh (coderai-docker): --map now accepts a single FILE, not just a dir, so a tool config that lives loose on the host can be linked directly, e.g. --map /host/video_editor.config.json:/cache/video_editor/video_editor.config.json Bump version to 0.1.5. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The bundled township tool (supervisord [program:township]) launched without -c/--config, so a township_config.json bind-mounted via --map into /cache/township_output was never read — the web UI came up with blank API key / league upload credentials / options every restart. * supervisord: pass --config /cache/township_output/township_config.json so a mapped config auto-loads. Map the bare-metal dir with coderai-docker --map /path/to/township_output:/cache/township_output and the saved settings + credentials come up automatically. * gen_township_fighters.py: a MISSING --config file is no longer fatal (fresh install with no mapped dir) — fall back to defaults and start normally; a malformed config still errors. The web UI's Save already writes back to <out-dir>/township_config.json, so first-run saves persist into the mapped dir. Bump version to 0.1.4. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The thermal guard was checkpoint-based and ran inside each engine, so it went blind during a single long native call (llama.cpp prefill / image encode): the engine can't reach a between-token checkpoint, and the [thermal][debug] heartbeat stops exactly when the GPU is hottest. Move the AUTHORITATIVE monitor to the front, which stays responsive regardless of what an engine is doing: * Front (engine_supervisor): a thermal thread reads per-card temps via gpu_stats() + CPU temp, maps each card to the owning engine by its CODERAI_ENGINE_GPUS selectors, and drives pause/resume with hysteresis (pause at *_high, resume only back at *_resume). A hot GPU pauses just its engine; a hot CPU pauses all. * Engines stop cooperatively, as before, but triggered remotely: the front POSTs /internal/thermal-pause; thermal.set_external_pause() makes wait_until_safe()/ checkpoint() block at the next safe point (publishing cooldown state so the Tasks page shows it), until /internal/thermal-resume. * Escalation: if a paused engine keeps generating (inflight > 0) — stuck in a native call it can't interrupt — for stop_escalate_checks (default 3) consecutive checks, the front SIGSTOPs the engine's process group; SIGCONT on cooldown. Both signals target the session group so children freeze too. * stop_all()/restart_engine() SIGCONT a frozen engine first (a stopped process ignores SIGTERM until continued); _spawn() resets the thermal flags. * Config: thermal.supervisor_enabled (default on), thermal.stop_escalate_checks. * UI: per-engine temp + pause/frozen state in engines_list and the Tasks cooldown banner (covers a SIGSTOPped engine that can't report its own cooldown). Also: exclude coderai-runtime/*-runtime from the Docker build context (.dockerignore) — a root-owned runtime temp file was breaking the image build. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
record_vram_delta() measured free VRAM via _free_vram_snapshot(), which only read the default CUDA device (device 0). A gpu_split model (e.g. gemma on two cards) lands most weights on the other card, so the device-0 delta came out ~0, tripped the 'delta <= 0' guard, and never persisted measured_vram_gb. The estimate stayed 0 and every load OOM-retried from scratch even with force_vram_update set. Sum free VRAM across every visible CUDA device (scoped per engine via CUDA_VISIBLE_DEVICES), plus amdgpu sysfs when cross-pooling is on or no CUDA device is visible -- mirroring _get_free_vram_gb(). Before/after snapshots are now symmetric, so the delta captures the full multi-card footprint and the real value is persisted; subsequent loads size correctly and stop OOM-looping. Bump version to 0.1.3. Co-Authored-By:
-
- 23 Jun, 2026 9 commits
-
-
Stefy Lanza (nextime / spora ) authored
The loop-guard change added a module-level _GEMMA_CALL_RE = _re.compile(...) at the top of the file, but 'import re as _re' sat far below (line 2266), so import crashed with NameError: name '_re' is not defined and every engine failed to start. Move the import up with the other stdlib imports (removing the late duplicate). Verified codai.api.text now imports cleanly. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
`docker load -i` shows no useful progress and the installer captured its output into a variable, so a ~12G load looked like a multi-minute hang. Stream the tarball through a meter into `docker load` instead — the file read tracks load progress closely. Prefer `pv` (bar + ETA), fall back to GNU `dd status=progress` (bytes + throughput), else plain load with a tip to install pv. The meter draws on stderr so the "Loaded image:" line is still captured. Pre-authenticate sudo first so its prompt doesn't collide with the bar. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The streaming content gate only ran when the request carried a tools array. But the AISBF relay can drop tools (as it does max_tokens), so a gemma model still emitting <|tool_call>call:NAME{…} from its system prompt had that markup streamed straight to the client as content — the spill that poisoned history and fed the tool-call loop. _gate_tool_content now: * ALWAYS withholds the unambiguous native special-token markers (<|tool_call>, <|tool_response>, DeepSeek DSML) and their cross-chunk partials, even with no tools declared; * gates the ambiguous <tool>…</tool> XML form only when tools are present; * gates the gemma-4 call:NAME{…} form by the per-model gemma_tool_parser mode (off: never; full: always; restricted: only declared tool names), so legit call:foo{…} prose/code is streamed instead of withheld+dropped. The gate is now invoked for every stream (not just tools-present), and the final flush likewise, with the mode/tool context resolved once per request. Bump version to 0.1.2. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The gemma-4 native call:NAME{…} heuristic could eat legitimate text: the content stripper removed ANY call:/response:{…} span with no tool-name restriction, so a coding reply containing e.g. call:foo{…} in prose or a snippet got silently deleted; and with no tools declared the parser matched any call:word{. Add a 3-way mode, resolved per-model (models.json "gemma_tool_parser") over the global models.gemma_tool_parser (default "restricted"): * full - parse & strip every call:/response: span (old behavior) * restricted - only when NAME is a declared tool; legit call:foo{…} prose/code is preserved, real declared-tool calls still parse & strip * off - disable the gemma heuristic entirely (bigger models that emit standard structured tool calls) resolve_gemma_tool_mode() + a centralized strip_gemma_native() now gate both the parse (GemmaParser, via the dispatcher) and the strip (ModelParserAdapter and the streaming ToolCallParser, which now stashes the declared tool names from extract_tool_calls so 'restricted' works on the streaming finalizer too). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A model can get stuck re-issuing the same tool call with the same arguments when each attempt fails (e.g. memory replace old_text=X → "No entry matched" on the wrong memory tier) — or when the call spills as un-parsed call:NAME{…} markup into assistant content. Each brokered request carries the full history, so coderai can see the repetition the agent's own tool-runner didn't break. _detect_tool_loop scans the recent history for a (tool, arguments) signature repeated >= threshold times where those attempts failed (per the tool result) or spilled as markup, and injects one system reminder before generation telling the model to stop repeating the call. Covers both structured tool_calls and spilled gemma-style markup. Configurable on every model: per-model models.json tool_loop_guard / tool_loop_repeats override the global models.tool_loop_* (default on, repeats=3); repeats<=0 disables. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
ChatMessage.convert_content_array_to_string ran at parse time (mode=before) and unconditionally joined any multipart content list into a string, replacing each image_url part with the literal text "[image_url content]". That destroyed the image before text.py's vision pipeline (_vision_ok / _normalize_vision_content, which the backend's mmproj/MTMDChatHandler consume) ever saw it — so a vision model (e.g. lisa = Gemma-4 + mmproj) received only text and answered "no image attached", and agents looped retrying. text.py's end-to-end vision handling was effectively dead code because content was always pre-stringified. Now flatten only TEXT-ONLY multipart arrays (the KiloCode case); preserve the list whenever it carries an image_url (or any non-text part) so the multimodal backend receives the image. Non-vision models still degrade to the placeholder downstream (text.py:1439), and text-only/plain-string paths are unchanged. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The broker streaming relay counted tokens into engine.active["step"] but never set ["rate"], so _merge_engine_tasks (which overlays both onto the running task) showed token progression with a frozen 0 speed. Compute tok/s from the first streamed token (so the model-load/queue wait doesn't drag the average down) and publish it as m["rate"], mirroring the engine-side text path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ships the broker load-status / max_tokens fixes, the run-as-invoking-user default (+ --root opt-out), nginx console silencing, and the expanded installer help. --versioned builds now tag coderai:full_all_0.1.1. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three operational fixes for the distributable image: run_oci.sh: default the container to the invoking user (uid:gid, SUDO_UID-aware) instead of the image's root default, with a new --root opt-out. The image has no USER directive and supervisord sets no user=, so a run without --user created root-owned dirs (logs, coderai-tmp, hf cache) in the bind-mounted /cache; a later --user run then couldn't write them ("cannot write /cache/logs ... logging to stdout only"). Running as the user by default makes a fresh install safe regardless of run order. --root restores the old behaviour (and throwaway config copy) for shared root-managed data dirs. nginx.conf: error_log crit (was info) + access_log off. nginx's [notice] startup/worker lines and per-request access logs were piped to the container console by supervisord and buried coderai's own output on attach / docker logs. Real failures (crit/alert/emerg) still surface. install.sh: print an extensive post-install guide (quick start, --local + --map, where data lives, the run-as-you default, file logging, docker logs/stop, --help). Co-Authored-By:
-