- 29 Jun, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authoredvideo: fix generation-OOM VRAM leak (traceback pin) + use sequential offload for injected components Two issues from the latest debug.log on the cached VACE model: 1. Generation-OOM VRAM leak (the cascade driver). The OOM/retry handler reloaded INSIDE `except Exception as e:`, so sys.exc_info() kept the failed forward pass's traceback alive — pinning its GPU activations (and the resident expert). _free_pipeline_vram() then reclaimed nothing, the retry reload saw the card still ~21 GB full, and every fallback (sequential/disk) OOM'd in turn -> 500. Same traceback-pinning bug fixed earlier for the load ladder. Restructured: capture the error, drop e.__traceback__, and do the free + gc + empty_cache + retry OUTSIDE the except block, where the activations are finally collectable. 2. model CPU offload OOMs this dual-expert forward. With injected cached components we forced 'model' offload, which keeps a whole ~7 GB expert resident; the A14B forward then spiked past 24 GB (loaded at 11 GB, OOM mid-generate). Force 'sequential' instead — minimal footprint that reliably fits; the cache still makes the LOAD fast. (offload_strategy=group remains available for more speed.) Retry policy unified: OOM -> sequential; device-mismatch -> incremental cache off (plain device_map build hooks everything itself). Both retry once, then 500. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
-
- 28 Jun, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authored
The incremental per-component cache built and cached all components correctly, but generation then failed: RuntimeError: Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu (... wrapper_CUDA__index_select) Cause: the load used the `balanced` (device_map) strategy. accelerate's device_map only dispatches components it LOADS — injected pre-loaded components keep whatever device they're on (CPU) with NO offload hook, so a forward pass mixes cuda inputs with cpu weights. Fix: when incremental cached components are injected, force a HOOK-based offload (model → group → sequential), which add offload hooks to EVERY component in the pipeline (injected included). model CPU offload keeps only the active ~7 GB 4-bit expert resident, so it fits easily and is fast. device_map strategies (balanced/disk) are skipped in this case. Safety net: a non-OOM generation failure that looks like a device mismatch now retries ONCE with the incremental cache disabled (plain build = pre-cache behaviour), so generation self-heals even if the injection path is imperfect instead of wedging the queue. (Load-time failures already fall back this way.) Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A Wan2.2 A14B pipeline that only loads via offload could never be cached — diffusers can't save_pretrained an offloaded pipe, so every load re-quantized from bf16 shards (~14 min) and risked OOM. Cache it per-component instead: - pipeline_cache: per-component subdirs each with their own completion marker + signature (component_dir / component_valid / save_component), atomic via .building + sweep_stale(). A load evicted before all components are cached keeps the finished ones; the next load mixes cached + freshly-built (now-cached) ones, converging to a full cache. (Plus the earlier sweep_stale / _unsavable_reason robustness.) - hf_loading.build_cached_components: for each bnb-quantizable heavy component, load from the per-component cache if present, else build fresh from the model — each ONE AT A TIME on the GPU (where bnb quantizes it to uniform 4-bit), saved, then moved to CPU + empty_cache. Peak VRAM is a single component even when the whole model doesn't fit; uniform 4-bit by construction (no bf16/4-bit mix). Verified the bnb round-trip (load 4-bit on GPU -> save_pretrained -> to('cpu') -> reload). - video._load_video_pipeline: inject the cached components like GGUF components so the existing offload ladder assembles them. Kill-switch CODERAI_INCREMENTAL_CACHE=0. Fully fallback-safe: on any load failure the request handler retries once with use_incremental_cache=False (plain build = today's behaviour). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Three fixes for the Wan2.2-VACE-Fun OOM cascade: 1. Offload retry death-spiral (the OOM driver). When from_pretrained OOMs MID-LOAD, `pipe` is never assigned, so the ~20 GB already placed on the GPU is pinned by the EXCEPTION'S TRACEBACK frames (their locals hold the partial model), not by `pipe`. _clear_mem() frees via `pipe` (None) so it reclaims nothing, and the next balanced step recomputes its GPU budget from a card still ~20 GB full (70%->17.5, 60%->3.4, 40%->2.3 GiB — all doomed). Drop `e.__traceback__` and run _clear_mem() OUTSIDE the except block (where sys.exc_info no longer pins the frames) so the stranded tensors are collectable before the next attempt. Applied to the balanced chain and the full-GPU and model-offload handlers. 2. Observability. The load-failure and generation-failure handlers raised HTTPException(500) with the cause only in the HTTP detail — debug.log showed a bare "Response status: 500". Log the full traceback in all three handlers. 3. Pipeline cache robustness. diffusers refuses save_pretrained on a CPU/seq offloaded pipeline, and a device_map/disk-offloaded one has meta tensors — either way the save half-writes a .building dir then a killed process leaves it orphaned (tens of GB). Add _unsavable_reason() to skip those cleanly (no junk), and sweep_stale() to delete orphaned *.building dirs. A full-GPU load stays savable and still caches the 4-bit pipeline. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two eviction fixes folded into 0.1.30: - VRAM eviction: a diffusers pipeline under device_map/accelerate offload (balanced/disk/model/sequential) or bitsandbytes quantization REJECTS .to('cpu') — the naive move silently left the weights resident and stranded VRAM, feeding the OOM death-spiral where the next load OOMs on a near-full card. Route pipelines (those exposing `components`) through the thorough _free_pipeline_vram (remove accelerate hooks -> drop component refs -> empty_cache); plain non-pipeline models still use a simple .to('cpu'). - RAM eviction: never unload the LIVE model (active_in_vram OR current_model_key) that a request loop keeps reusing. An offloaded pipeline's host RAM IS the live model, so evicting it between same-model requests just forces an immediate reload — churning VRAM/RAM and re-stranding device_map weights while freeing nothing lasting. The last-resort active-model eviction now fires only for a STALE active model (no longer the current one). Co-Authored-By:
-
- 27 Jun, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authored
Two bugs let a torch video pipeline run on the nvidia-gguf engine (caps={"gguf"}) when the nvidia engine was briefly down mid-restart: 1. _INFERENCE_PATHS listed "/v1/videos/generations" (plural) but the endpoint is "/v1/video/generations" (singular, video.py:3104). So video requests failed is_inference_path(), skipped all capability-aware routing, and fell through to registry.primary() — which returns the first HEALTHY engine when the primary (nvidia) is unhealthy, i.e. the gguf sibling. Fixed the path. 2. pick_engine() step 5 fell back to a capability-BLIND least_loaded(None) pick, so a typed request could still land on an engine lacking the capability. Now a typed request (transformers/gguf/whisper) only ever picks a capable engine — preferring the primary when it can serve the cap (request queues / caller retries), else 503 — instead of mis-routing to an incompatible engine. Net: video generation (cap=transformers) routes to the nvidia engine only; if it is busy it queues there, if it is restarting the caller retries — it never runs a torch pipeline on the gguf-only engine. Bumps to 0.1.30. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The context-free VRAM query used gpu_memory(), which (via NVML/nvidia-smi) reports every physical NVIDIA card regardless of CUDA_VISIBLE_DEVICES. So the Vulkan/Radeon engine (CUDA_VISIBLE_DEVICES="") showed the NVIDIA 3090's VRAM in its status instead of its own AMD card. Switch the per-engine status poll (api/app.py) and the capability probe (broker/capabilities.py) to visible_gpu_memory(), which honours CUDA_VISIBLE_DEVICES (index OR UUID; empty -> no CUDA cards). Add gpu_query.amd_gpu_memory() (amdgpu sysfs, driver-free) as the fallback so the Radeon engine reports its real card; capabilities keeps its torch-loaded guard so the torch-free front still enumerates the whole node. Bumps to 0.1.29. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Reporting VRAM via torch.cuda.mem_get_info lazily creates the CUDA primary context (~256 MiB on an RTX 3090). An engine that never loads a torch model (the GGUF/llama.cpp engine) therefore pinned ~256 MiB just to answer health polls, the capability probe and the load-path eviction check — and that stray context was enough to tip a borderline 4-bit Wan2.2 A14B video load into OOM. New codai/models/gpu_query.py queries VRAM without a context: pynvml first, nvidia-smi fallback, torch only if a context already exists. visible_gpu_memory() scopes to the engine's cards via CUDA_VISIBLE_DEVICES (matched by index OR UUID; empty value -> no CUDA cards, e.g. the Vulkan/Radeon engine). Wired into the idle health poll (api/app.py), the capability probe (broker/capabilities.py) and the load-path free-VRAM checks (models/manager.py: _get_free_vram_gb / _free_vram_snapshot). Adds nvidia-ml-py to requirements-nvidia.txt and the update overlay. Bumps to 0.1.28. Net: the GGUF engine sits at 0 MiB while idle (its real context comes from llama.cpp on model load and is freed on unload), returning the headroom that makes the A14B load fit instead of OOM-ing. Co-Authored-By:
-
- 26 Jun, 2026 7 commits
-
-
Stefy Lanza (nextime / spora ) authored
The diffusers pipeline is cached/reused across image requests, so the adapters _apply_loras added last time linger; re-loading the same fighter then raised "Adapter name <x> already in use", and stale adapters from a different request accumulated. Track the request adapters on the pipeline and delete them (plus a defensive per-name delete) before re-loading, leaving acceleration adapters (__accel__/__accel_2__) untouched. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Loading a trained LoRA at inference (e.g. a fighter LoRA on an image pipeline) crashed with "cannot import name AwqGEMMQuantLinear" because peft dispatches AWQ for any non-bnb target when gptqmodel is installed, and gptqmodel 7.1.0 renamed that class to AwqGEMMLinear. The alias shim existed only in the training path. Extract it to codai/models/peft_compat.ensure_peft_awq_compat() (cached) and call it before load_lora_weights in codai/api/images._apply_loras and the acceleration fuse path; the trainer now delegates to the shared shim. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The 0.1.24 nesting blindly appended the bundled sub-app name to the incoming X-Forwarded-Prefix, which double-counted it when the outer proxy mounts the app under the SAME name and proxies to our /<app>/ path (topology B): outer sends "/township" -> container emitted "/township/township", so every township fetch() 404'd ("Not found" -> JSON parse error in the UI). Per-app maps now only append our name when the incoming prefix doesn't already end in it, so both topologies work: outer-wraps-everything ("/ai" -> "/ai/town ship") and outer-mounts-same-name ("/township" -> "/township"). Direct (no outer proxy) still yields "/township". Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Internal container nginx is now chain-aware: it prefers the outer proxy's X-Forwarded-Proto/Host/Prefix and nests bundled sub-app prefixes under any outer prefix (outer /ai + /township -> /ai/township). Fixes characters/ environments thumbnails and other absolute/sub-path URLs 404ing when the all-in-one container runs behind a second reverse proxy. Documented the required outer-proxy headers in docs/reverse-proxy-nginx.md. LoRA training loads its base pipeline outside the model manager (and unloads all manager models first), so the engine reported 0 loaded models mid-train. Surface the active training base model via active_training_model() so the engines card reflects the busy GPU. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- Show which engine each task runs on (badge on active AND history rows; backend already tags t.engine) and an "● processing" badge on the engines card (new inflight/processing fields in /admin/api/engines). - Image-gen advancement showed only "working…": poll() short-circuited to the synthesized task list whenever the primary had any in-flight request, dropping the engine's real step/total. Now it quick-polls the engine first (image/diffusers gen releases the GIL so it answers with real progress) and only falls back to the synthesized list when a GIL-bound text gen times out. - Surface thermal cooldown on the task entry itself: _merge_engine_tasks marks tasks on a cooling engine with cooling/cooling_message (the row already renders it). Co-Authored-By:
-
↔ Stefy Lanza (nextime / spora ) authoredLoRA add_adapter() crashed: peft's dispatch_awq does `from gptqmodel.nn_modules.qlinear.gemm_awq import AwqGEMMQuantLinear`, but gptqmodel 7.1.0 renamed that class to AwqGEMMLinear. peft calls dispatch_awq for ANY non-bnb target when gptqmodel is installed, so the failed import broke every add_adapter (SDXL/Wan/Z-Image), not just AWQ models. Add _ensure_peft_awq_compat(): alias AwqGEMMQuantLinear -> AwqGEMMLinear so peft's import succeeds and falls through to the correct dispatcher. Called before every add_adapter() (sd15/sdxl/dit/wan). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
QLoRA training reached VAE encode then failed: the ZImagePipeline loads the VAE in bf16, but the image tensor was fed as float32 -> conv2d "Input type (float) and bias type (BFloat16) should be the same". Feed the VAE its own weight dtype. Co-Authored-By:
-
- 25 Jun, 2026 10 commits
-
-
Stefy Lanza (nextime / spora ) authored
QLoRA training crashed in the Qwen3 TEXT ENCODER: the unsloth build quantizes it too, and loading components piecemeal (AutoModel/AutoencoderKL by subfolder) left bitsandbytes' 4-bit quant-state unreconstructed -> "FP4 quantization state not initialized" / AssertionError in Linear4bit.forward. - Load all components via ZImagePipeline.from_pretrained (the proven inference loader, which loads each 4-bit component correctly), then LoRA-train only the transformer. VAE/text-encoder are encoded once then freed (4-bit modules can't .to('cpu'), so drop refs + free). - Disable gradient checkpointing for the 4-bit path: the recompute can desync bnb's quant-state. At 512/batch-1 the 4-bit base + activations fit without it. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Training on the full bf16 Tongyi-MAI/Z-Image-Turbo was extremely slow — a ~10-min download plus a heavy bf16 model that doesn't fit cleanly on 24 GB. Switch _train_dit to QLoRA: load the transformer in 4-bit (frozen, ~4 GB, no CPU offload) and train the LoRA on top. This trains directly on the already-cached 4-bit (e.g. unsloth) build — no redirect to the full model, no download. - Load transformer with diffusers BitsAndBytesConfig (nf4); an already-4-bit checkpoint (embedded quant config, e.g. unsloth) is loaded as-is via fallback. - Enable gradient checkpointing and force the input-embedding (all_x_embedder) output to require grad so QLoRA grads reach the attention LoRA layers; hooks removed at job end. - Drop the quantized-base -> full-model redirect added earlier. LoRA still applies to the quantized model at inference (identical architecture). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Env/fighter LoRA training for Z-Image misrouted to _train_sdxl (CLIPTokenizer crash: "NoneType cannot be interpreted as an integer"). Cause: _resolve_base_model_path returns the HF id verbatim (e.g. unsloth/Z-Image-Turbo-unsloth-bnb-4bit) when the model has no local `path`, so the isdir(base_path/transformer) DiT check was False and it fell through to the SDXL/SD15 trainer. - Detect Z-Image by NAME (id contains z-image/zimage) in addition to the diffusers-config class name, so an HF-id base routes to _train_dit. - _train_dit: when the base is a pre-quantized (bnb/nf4/4bit) build, train against the full Tongyi-MAI/Z-Image-Turbo instead (a 4-bit checkpoint isn't a clean full-precision LoRA base); the LoRA still applies to the quantized model at inference. Overridable via lora_train_base_model. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
is_available() reported false in the running engine even though gptqmodel + fast kernels import fine in a fresh process. Two causes: 1. capabilities() cached a DEGRADED result (gptqmodel imported but the inner gptqmodel.utils.backend BACKEND import transiently came up empty, e.g. when the first call landed mid model-load). That empty-backends result stuck for the whole process life, so the settings page said "GPTQModel not installed" until restart. Now a degraded (available-but-no-backends) result is NOT cached — re-detect next call; only a clean positive or a genuine ImportError is cached. 2. is_available() gated on a SPECIFIC fast kernel being detected. GPTQModel always has a Triton/torch fallback and picks the kernel at load, so availability now gates only on gptqmodel importing; backends stay informational. Also: /admin/api/quantize-capabilities re-detects live (capabilities(refresh=True)) so the settings page never serves a stale cache. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
1. Model load/unload (and every is_admin-gated front action) returned 401: is_admin probed prim.url + /admin/api/status on the engine, but that route is front-only (removed from the engine in def78c18), so it 404'd → never 200 → Unauthorized. Add an engine-side admin-gated /admin/api/whoami and point is_admin at it. 2. Township reference generation (characters/environments pages — _run_regen_job and _run_create_profile_job) hardcoded 768x512/512x512 + 28 steps, ignoring the Run page. Add _ref_gen_res_steps(args) (honors keyframe_size/keyframe_steps, re-read from the saved config so edits apply without restart) and use it at the reference generators; generate_character/generate_environment now forward `steps` (the server already accepts it). Keyframes/match videos already honored the config. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai's trainer only targeted SDXL/SD1.x U-Nets, producing unet.-prefixed LoRAs that silently no-op on Z-Image's DiT (ZImageTransformer2DModel, transformer. keys, Qwen3 text encoder) — "No LoRA keys associated to ZImageTransformer2DModel found with prefix='transformer'". The lora_train_base_model=SDXL workaround can't make a LoRA that loads on Z-Image. Add _train_dit: a native flow-matching DiT LoRA trainer for Z-Image, reusing the existing job/progress/checkpoint/queue infra (modeled on the Wan video DiT trainer, which is video-only). All deps (ZImageTransformer2DModel, ZImagePipeline, PEFT, Qwen3) are already in the main venv — no ai-toolkit, no separate venv. Reverse-engineered from diffusers pipeline_z_image.py so training matches inference: chat-template + Qwen3 hidden_states[-2] masked per-sample embed list; AutoencoderKL latents scaled (lat-shift)*scale; list-based transformer I/O with normalized timestep (1000-t)/1000; RAW target = x0-noise (Z-Image negates the model output); timesteps sampled from the discrete turbo schedule (set_timesteps mu-shift) to keep the distilled model sharp; save via ZImagePipeline.save_lora_weights (transformer. keys) so _apply_loras' load_lora_weights applies it. _train_lora_sync routes a Z-Image DiT base to _train_dit (detected via model_index/transformer config _class_name); other DiTs (Flux/SD3) still raise with guidance. v1 — needs one validation train; recipe + knobs in docs/zimage-lora-training.md. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
After the GGUF-isolation split, a torch engine and a gguf engine share one NVIDIA card but each can only evict its OWN models. So loading an image model on the torch engine while the gguf engine still holds a resident GGUF text model failed: local eviction "freed 0.0 GB ... VRAM held elsewhere" → CUDA OOM. Fix — co-located VRAM release: - front passes each engine its same-GPU siblings' internal URLs via CODERAI_COSITED_URLS (matched by identical CODERAI_ENGINE_GPUS selectors). - engine registers an external_vram_releaser that POSTs to each sibling's new /internal/evict-vram when local eviction can't free enough. - /internal/evict-vram → manager.release_idle_vram(): evicts all idle (non-busy) models and returns GB freed; busy/actively-serving models are left alone. Symmetric: the gguf engine can likewise reclaim VRAM from the torch engine's idle diffusers models. Driver-level free VRAM (cross-process) is re-checked after each releaser, so the loader proceeds once the sibling has freed enough. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two fixes after the GGUF process-isolation split: 1. Routing: a model with engine=nvidia in its config (a hard pin) but a GGUF body 503'd, because the torch nvidia engine dropped the `gguf` capability. pick_engine now falls back to the co-located "<name>-gguf" sibling (same card) when the pinned engine can't serve the required capability (and symmetrically for a gguf-pinned model needing transformers). The pin's intent — that physical card — is honoured. 2. Config-save visibility: the cached-models scan is stale-while-revalidate, so the first read after an invalidation returned stale and refreshed in the background — a saved config didn't appear until a second refresh. _invalidate_cache_scan now sets force_sync so the next read recomputes synchronously (fresh immediately). The system worker invalidates on reload-config, so a save shows on the next page load. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The post-delete redirect used `(window.ROOT_PATH||'') + '/matches'`, but window.ROOT_PATH was never defined, so behind the nginx /township mount it redirected to the bare /matches — which hits the coderai front, not the township tool → 404. The mount shim wraps fetch()/EventSource() to add the prefix but not location.href assignments. Define window.ROOT_PATH=P in the shim so the redirect (and any future ROOT_PATH use) resolves to /township/matches; with no prefix (direct access) it stays unset and the bare /matches path is correct. Co-Authored-By:
-
- 24 Jun, 2026 15 commits
-
-
Stefy Lanza (nextime / spora ) authored
llama.cpp's CUDA backend and PyTorch sharing one process corrupt the CUDA context: after a GGUF model runs on an NVIDIA card, the next torch kernel dies with "CUDA error: invalid argument". Seen in a township match — a gemma GGUF served a chat completion on the nvidia engine, then the next Z-Image (diffusers) image gen crashed in its Qwen3 text-encoder's sdpa mask (padding_mask.all()). Every Z-Image run before llama.cpp touched CUDA succeeded; the first one after crashed. Evict+swap in one process can't fix it — torch holds the (now-corrupted) context for the process lifetime. Fix: when GPUs are auto-detected and server.isolate_gguf_engine is True (default), each NVIDIA torch engine gets a co-located sibling gguf engine on the SAME card (own process -> own CUDA context). The torch engine drops the `gguf` capability and serves transformers/diffusers; the sibling (backend=nvidia so GGUF takes the proven CUDA-llama path, capabilities={gguf}) serves llama.cpp. Routing is already capability-based, so GGUF goes to the gguf engine and HF/diffusers to the torch engine. Both are real engine subprocesses, so the front's routing, VRAM/eviction and thermal (cooperative pause + SIGSTOP on the process group) apply unchanged — and SIGSTOPping the gguf engine no longer freezes the torch engine. Ignored when engine_specs is set (declare the split yourself). Disable with server.isolate_gguf_engine=false. Design + trade-offs (extra torch context, no cross-engine eviction on a shared card) in docs/gguf-process-isolation.md. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Trained LoRAs, characters, environments and voices are resolved from the legacy HOME-based dir ~/.coderai, not from --config. In the container the user's home (e.g. /home/ubuntu) isn't where config is mounted ($CONFIG_DIR/coderai), so those lookups missed the mounted data and image-gen failed with "LoRA '<name>' not found on server". Symlink ~/.coderai -> $CODERAI_CONFIG_DIR/coderai so HOME-based and --config lookups agree, removing the need for an extra per-deployment --map. Acts only when safe (a symlink, missing, or an empty dir) so a user-mounted volume or real data is never clobbered. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The models page reads /admin/api/cached-models (served by the system worker), whose _scan_caches() merges each model's saved config (precision/quant/ctx) into the cards. The scan freshness check _scan_signature() only fingerprinted the HF/GGUF cache directories, never models.json, and model-configure never invalidated the scan. A config save rewrites models.json but touches no cache file, so the signature stayed unchanged and the scan was served stale for the full TTL (600s) -- the engine reloaded but the front showed old config until a full restart. - _scan_signature(): fold models.json mtime into the signature so any process serving cached-models re-scans on the next read after a save. - system worker /internal/reload-config: call _invalidate_cache_scan() after rebinding config, so the front's reload-push refreshes the scan immediately. - engine /internal/reload-config: same invalidation for single-process / system-worker-down fallback. Also adds docs/dtype-auto-selection.md (planned design: model-native dtype default read from the checkpoint, precision as explicit override, FA2 fp32 guard) and ignores build artifacts (venv_build.log, .build.pid). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Audit of the nvidia text loader (cuda.py) vs the shared hf_loading.py found real config drift — the same model config behaved differently on the text path: 1. precision IGNORED — cuda.py hardcoded torch_dtype=float16. Now the manager passes the per-model and cuda.py resolves it via resolve_dtype() (default fp16 on CUDA / fp32 on CPU when unset, preserving current behaviour). 2. 4-bit compute dtype hardcoded float16 → now follows the resolved precision (_make_bnb_config takes a compute_dtype). 3. key was ignored — the manager only read . Now it honours flash_attn / flash_attention (per-model) and the global global_args.flash_attn (offload.flash_attention). 4. offload_buffers was only set on the disk-spill path → now also on the GPU+CPU device_map ladder (with offload_folder), so CPU offload doesn't pin activation buffers on the GPU and OOM the forward pass. 5. global max_ram_gb now clamps the CPU offload budget (central _get_gpu_memory_map_with_limit + the disk fallback), matching hf_loading. Diffusers-only items (component_quantization, GGUF components, sdcpp flash flags) are correctly N/A to the AutoModelForCausalLM text path. Bump version to 0.1.9. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Loading an HF model (e.g. Qwen3.5-9B, 4-bit) failed with 'Permission denied: ./offload'. Cause: the model's per-model offload_dir was the relative './offload' (a stale auto-saved default), which _cfg_or_global lets win over the global config; './offload' resolves to the CWD = the READ-ONLY /opt/coderai/app tree in the image. The config WAS respected — a relative offload path is just meaningless where the CWD isn't writable. * hf_loading.resolve_offload_dir(): an absolute offload_dir is respected as-is; a relative/empty one INHERITS the configured GLOBAL offload directory (global_args.offload_dir) when absolute, then CODERAI_OFFLOAD_DIR, then the user cache — never the CWD. Applied in the manager (both load sites, always passed), hf_loading, and defensively in the cuda backend. * main.py + entrypoint: a container-writable CODERAI_OFFLOAD_DIR (=/cache/offload, created by the entrypoint) is used when the GLOBAL config is still the bare './offload' default; explicit config wins. * run_oci.sh: forward HF_TOKEN / HUGGING_FACE_HUB_TOKEN from the host env so the engines authenticate to the HF Hub (the 'unauthenticated requests' warning) for higher rate limits + gated models. HF_HOME/cache dir was already honoured (main.py from config.models.hf_cache_dir). Bump version to 0.1.8. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The point of the keepalive is to hold the client connection open while the engine is stuck loading — so 'silent' must NOT fall back to the dead legacy path (which sends nothing until the engine's first byte and lets short-timeout clients/proxies disconnect). Now ALL streaming inference goes through the front keepalive path: * silent — SSE comment lines (': …') only: keeps the socket alive, emits no chunk/content/status (invisible to event parsers). * invisible — empty-content chunk + x_queue_info (default). * visible — visible status text. * thinking — reasoning channel (when mode != silent). The front stays responsive even when the engine is GIL-blocked, so it can drive these regardless of engine state. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Clients hitting the public API directly (e.g. township via nginx) disconnected during long waits: the direct proxy() path acquired the front queue slot silently and awaited the engine's first byte (model load) with no output. The broker path already kept alive; the direct path now does too. For a STREAMING inference request, commit to a 200 text/event-stream up front and emit keepalive while acquiring the queue slot and during the engine's model load / not-ready retries, then relay the real stream (token-counting for the Tasks page), ending cleanly if the engine dies mid-flight. Configurable mode (models.wait_status_mode, global default 'invisible'; per-model override via the models.json entry): * invisible — empty-content SSE chunk + x_queue_info (holds the connection; no content pollution) * visible — short visible status text (appears in the content) * silent — nothing (legacy path) When thinking is enabled the keepalive goes on the reasoning channel instead (no pollution), unless mode is silent. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
manager (record_vram_delta): record the FULL footprint after a load — including offloaded loads (no longer skipped) — so the next load skips the OOM→move-to-RAM retry dance: * measured_vram_gb — GPU VRAM delta + reserve (GPU portion; eviction uses it) * measured_ram_gb — host-RAM delta (the CPU-offloaded layers) [new] * measured_n_gpu_layers — the layer split llama.cpp settled on [new] total need = vram + ram. All gated by force_vram_update (else only when used_vram_gb unset). New _learned_n_gpu_layers(): when n_gpu_layers is auto/-1, reuse the learned split so a reload jumps straight to the config that fit. Both load sites snapshot host RAM and read the backend's settled n_gpu_layers. frontproxy: * graceful streaming proxy: when an engine dies mid-stream (e.g. a CUDA ggml_abort SIGABRT on a VRAM-OOM decode), end the relayed stream cleanly instead of throwing an unhandled ASGI exception (noisy traceback). * task actions (cancel/interrupt/pause/resume/restart/DELETE) now route to the engine that OWNS the task — fan out to every live engine (each validates the same session cookie), first success wins — fixing 'Task not found' for tasks on a non-primary engine; front-only synthetic ids handled locally. Bump version to 0.1.7. -
Stefy Lanza (nextime / spora ) authored
* coderai-docker (run_oci.sh): new --tool-arg TOOL VAL (one token) and --tool-args TOOL STR (whitespace string), repeatable, for video-editor | videogen | township | parler. They accumulate into CODERAI_<TOOL>_ARGS env vars that supervisord appends to each tool's command line. Documented in --help (plus the --map dir-or-file note and bring-your-config examples), and shown on the startup banner (tool-args:). * entrypoint: default+export the four CODERAI_*_ARGS (empty) so supervisord's %(ENV_...)s never fails on an undefined var; pre-create /cache/video_editor/sessions. * supervisord: append %(ENV_CODERAI_*_ARGS)s to all four tool launchers; the video editor now runs with --session --session-dir /cache/video_editor/sessions so editor-state recovery is ON by default and persists on the cache volume. * video_editor.py: new --session-dir to place session state/assets at a persistent/mapped path (defaults to ~/.cache/... when omitted). Bump version to 0.1.6. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai always runs as front + engines, so drop the single-process special-casing in execute_api_request: always hand the sub-request to the front (the single API), which routes it to the engine that owns the target model. Critically, remove the on-error fallback to in-process dispatch — that would silently run the model on the WRONG engine and hide a routing failure. If the front can't be located/reached it now returns a 502 the caller surfaces. Drops the now-unused os import. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
* pipelines.py: /v1/pipelines/* (image-to-video, story, video-dub, audio-dub) chained their steps by calling the image/video/text/TTS handlers DIRECTLY in-process, forcing every modality onto whichever engine received the pipeline request. In a multi-engine deployment each model may live on a different engine. Route every sub-step through the front via execute_api_request (same fix as characters/environments) so each lands on the engine that owns that model; in-process fallback for single-process mode. Drops the unused TestClient import. * supervisord.conf: the bundled tools (video_editor, videogen, township) now use --base-url http://127.0.0.1:8776 — the PUBLIC nginx port — so they talk to coderai exactly like any external client (nginx already proxies the full API with 4G bodies / 1h timeouts). The front's own bind stays CODERAI_PORT=18776. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
/v1/characters/generate and /v1/environments/generate produced their reference images by calling execute_internal_request(request.app, '/v1/images/generations') — an IN-PROCESS self-dispatch on whatever engine handled the request. Two bugs: 1. It re-entered the engine's own _InternalAuthMiddleware without the internal token, so in multi-engine mode it 403'd ('engines are reachable only through the front proxy') — image gen failed while plain text worked (text never self-dispatches). Township (a normal API client) hit exactly this. 2. Even with a token it would run the image model on the WRONG engine: the image model may be assigned to / pinned on a different engine than the one serving the characters/environments request. Add execute_api_request(): behind a front (CODERAI_INTERNAL_TOKEN set) it routes the sub-request THROUGH THE FRONT (the single public API) at 127.0.0.1:<front port>, so the front picks the engine that owns the image model and stamps the internal token itself; the caller's Authorization is forwarded so that engine authorises it like any client call. Single-process mode still dispatches in-process (middleware is a no-op with no token). The client never sends internal headers — it just talks to the one API endpoint. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Mirror the township config fix for the bundled video editor: * supervisord: pass --config /cache/video_editor/video_editor.config.json so a bind-mounted (or cache-persisted) config auto-loads — previously it only auto-loaded video_editor.config.json from the baked cwd, which isn't mappable. * video_editor.py: a MISSING --config file is no longer fatal (was raise SystemExit) — start with defaults; the web UI Save creates it. * entrypoint: pre-create /cache/video_editor so the config dir exists for auto-load + web-UI Save (persists on the standard /cache volume by default). * run_oci.sh (coderai-docker): --map now accepts a single FILE, not just a dir, so a tool config that lives loose on the host can be linked directly, e.g. --map /host/video_editor.config.json:/cache/video_editor/video_editor.config.json Bump version to 0.1.5. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The bundled township tool (supervisord [program:township]) launched without -c/--config, so a township_config.json bind-mounted via --map into /cache/township_output was never read — the web UI came up with blank API key / league upload credentials / options every restart. * supervisord: pass --config /cache/township_output/township_config.json so a mapped config auto-loads. Map the bare-metal dir with coderai-docker --map /path/to/township_output:/cache/township_output and the saved settings + credentials come up automatically. * gen_township_fighters.py: a MISSING --config file is no longer fatal (fresh install with no mapped dir) — fall back to defaults and start normally; a malformed config still errors. The web UI's Save already writes back to <out-dir>/township_config.json, so first-run saves persist into the mapped dir. Bump version to 0.1.4. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The thermal guard was checkpoint-based and ran inside each engine, so it went blind during a single long native call (llama.cpp prefill / image encode): the engine can't reach a between-token checkpoint, and the [thermal][debug] heartbeat stops exactly when the GPU is hottest. Move the AUTHORITATIVE monitor to the front, which stays responsive regardless of what an engine is doing: * Front (engine_supervisor): a thermal thread reads per-card temps via gpu_stats() + CPU temp, maps each card to the owning engine by its CODERAI_ENGINE_GPUS selectors, and drives pause/resume with hysteresis (pause at *_high, resume only back at *_resume). A hot GPU pauses just its engine; a hot CPU pauses all. * Engines stop cooperatively, as before, but triggered remotely: the front POSTs /internal/thermal-pause; thermal.set_external_pause() makes wait_until_safe()/ checkpoint() block at the next safe point (publishing cooldown state so the Tasks page shows it), until /internal/thermal-resume. * Escalation: if a paused engine keeps generating (inflight > 0) — stuck in a native call it can't interrupt — for stop_escalate_checks (default 3) consecutive checks, the front SIGSTOPs the engine's process group; SIGCONT on cooldown. Both signals target the session group so children freeze too. * stop_all()/restart_engine() SIGCONT a frozen engine first (a stopped process ignores SIGTERM until continued); _spawn() resets the thermal flags. * Config: thermal.supervisor_enabled (default on), thermal.stop_escalate_checks. * UI: per-engine temp + pause/frozen state in engines_list and the Tasks cooldown banner (covers a SIGSTOPped engine that can't report its own cooldown). Also: exclude coderai-runtime/*-runtime from the Docker build context (.dockerignore) — a root-owned runtime temp file was breaking the image build. Co-Authored-By:
-