- 23 Jul, 2026 5 commits
-
-
Stefy Lanza (nextime / spora ) authoredThe tooltip matched loaded_info entries (raw engine keys) against canonicalized display names client-side and mostly missed — footprints now canonicalize server-side with the same mapping as the model list, keyed by canonical id, so every loaded model shows its VRAM (+RAM when offloading). llama-vl: bound mtmd image_max_tokens (config `image_max_tokens`, default 1024) — large photos otherwise expand to thousands of vision tokens whose transient compute buffer evicts co-resident models on a small card. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
-
Stefy Lanza (nextime / spora ) authored
Eviction cleanup() closed the llama ctx / killed the dinov2 subprocess while another thread was mid-encode (embeddings hold no pool ref, so _is_key_busy saw idle) — use-after-free segfaults that crash-looped the radeon engine. cleanup() now takes the same per-model lock the embed paths hold, so it waits for the in-flight call. Engines card: the loaded-models hover now lists each model with its measured VRAM footprint and, separately, the host-RAM slice when CPU-offloading. Plumbed engine-state loaded_info → supervisor → registry → engines_list → tooltip. dinov2cpp: build the quantize tool too (dinov2-large-Q8_0 is lossless at cos 0.9973 and drops 612→329 MB). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
dinov2: packaging/dinov2cpp/ builds dinov2.cpp with ggml's Vulkan backend plus a coderai `embed` stdin/stdout server (HF-matching preprocessing: short-side 256 bicubic, center-crop 224; CLS read via ggml_backend_tensor_get so GPU backends work; DINOV2_FORCE_CPU escape hatch). embeddings.py 'dinov2cpp' backend drives one persistent subprocess per model (evict = terminate, reload = respawn); dinov2 GGUF entries route there. Verified on the RX 580: cos 0.9973 vs the HF dinov2-large backend. Tasks page: every task now shows the compute device it runs on and whether the model is CPU-offloading. record_vram_delta records per-model device+offload ("GPU RTX 3090 · CPU offload 12.4 GB"); api_tasks tags each task from that map (engine default as fallback); tasks.html renders a green/amber/gray badge (GPU / offload / CPU). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A GGUF embedder with a configured `mmproj` now loads as a vision-capable 'llama-vl' backend: the mtmd image tower feeds an embeddings context (pooling LAST) under the GME chat prompt — the same scheme as the HF qwenvl backend, so both produce the same vector space (~0.89 cosine agreement at Q4). Runs natively on whatever the llama.cpp build targets (Vulkan on radeon). Text goes through the same prompt+last-token path so both modalities share the GME space; PIL-decoded images are passed as raw RGB bitmaps (mtmd's stb can't read AVIF). Serialized on the per-model lock; cleanup frees the mtmd ctx. Reverts the front's image-reroute-to-HF-sibling: a model pinned to an engine must be served there, not silently moved. The gme mmproj was generated from the local HF checkpoint with llama.cpp's convert_hf_to_gguf.py --mmproj (F16, 1.3 GB). Co-Authored-By:
-
- 22 Jul, 2026 20 commits
-
-
Stefy Lanza (nextime / spora ) authored
llama.cpp's embedding API has no image tower, so `image` requests to a GGUF embedder 400'd. When the same model is also registered as a non-GGUF HF entry (e.g. gme-Qwen2-VL GGUF on radeon + HF on nvidia), the front now rewrites the image request to that sibling — one model name for clients; text stays pinned to the GGUF's engine, images go to the engine that has the vision tower. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two remaining GGUF crashers: (1) detokenize→retokenize is not token-count-stable, so a chunk cut at exactly n_ctx could re-inflate past it inside embed() and trip the GGML_ASSERT — chunks are now shrunk until their re-tokenized length verifiably fits (n_ctx-64 margin); (2) llama.cpp contexts are not thread-safe and parallel indexer requests segfaulted the engine (-11) — embed calls now hold a per-model lock. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Inputs beyond the context window are split into context-sized token windows, each embedded, and combined with a token-count-weighted mean (then normalized) — no content dropped. Single-chunk inputs keep the direct path. Pairs with raising the GGUF embedders' n_ctx in config. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
llama.cpp embedding mode SIGABRTs (GGML_ASSERT out_ids == n_outputs) when an input exceeds the batch size — with the default n_batch=512 any long listing killed the radeon engine (front kept respawning it). Size n_batch/n_ubatch to n_ctx at load, and truncate each input to the context window before embed so no accepted input can trip the assert. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A .gguf path in the embedding registry now loads through llama_cpp (embedding=True, n_ctx/n_gpu_layers from the model config) instead of failing in SentenceTransformer. Per-token outputs are mean-pooled, vectors normalized, matryoshka `dimensions` truncation reuses the shared path. cleanup() closes the llama ctx so eviction frees VRAM. Text-only: image requests against a GGUF embedder keep returning 400. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A bulk indexer's burst of first-requests all saw model_obj None and EACH loaded its own copy of the embedder (observed pool_instances=3 for the 4B model + 6 for DINOv2 ≈ the whole card) — this duplicate stacking, not a single load, was what exhausted VRAM and poisoned the measured footprint (26.8 GB recorded for an 8 GB model, measured across overlapping loads). Loads now take a per-model asyncio lock with a re-check, so one request loads and the rest reuse the loaded model. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Teardown leak: _free_pipeline_vram now records the pipeline's CUDA storage pointers up front and, after the normal teardown+gc, walks gc.get_objects() and nulls any surviving tensors out of their list/dict referrers (the proven pass from the CUDA text backend's cleanup) — the "~22 GB untracked (teardown leak; referenced elsewhere)" VRAM is now reclaimed instead of poisoning every later load until a restart. Names the holders when it fires so future leak sources identify themselves. Tracking: a video pipeline only registers in manager.models AFTER its multi-minute load, so concurrent loads raced the free-VRAM check into mutual OOM. note_loading()/clear_loading() reserve the model's estimate for the whole load window; _get_free_vram_gb subtracts reservations and the orphan-VRAM check counts them (no more false "teardown leak" for an in-progress load). The embeddings OOM-retry waits out active reservations (bounded 300s) before evicting and retrying. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
When the GPU is held by another workload (video job, concurrent load racing request_model's free-VRAM check), the embedding load/encode hit CUDA OOM and 500'd. Now both paths behave like other model loads: run the manager's eviction (which frees idle models and waits for busy ones to go idle) and retry once. No CPU fallback — the request participates in the same VRAM queue as everything else. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
This reverts commit 7908670c.
-
Stefy Lanza (nextime / spora ) authored
Bulk video jobs (township) occupy the whole card while active, so the embedding model 500'd with CUDA OOM on load and CUBLAS alloc failures on encode. Both paths now fall back to CPU — degraded throughput beats hard-failing every request until the GPU frees up. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The final usage chunk joined message contents with str.join, which raised "expected str instance, list found" for multipart (vision) content and surfaced as an error event at the end of otherwise-successful streams. Flatten via _content_to_text instead (all three token-estimate sites). Embeddings limit 120→1200 req/min per IP: bulk indexers legitimately embed at several req/s; keep the limit an abuse guard, not a throttle. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The plain-Jinja fast path in generate_chat_stream renders the template itself and generates via create_completion — which never invokes the multimodal (mmproj) chat handler, so image_url parts were silently dropped and vision models answered as if no image was sent. Streaming vision requests now fall through to create_chat_completion. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Declare turboquant-py in the OCI requirements so the image (and the upgrade delta-sync) installs it — the admin UI offered the 'library' backend but the package was never present. The upstream 0.1.x package only accepts bit_width 1-4, so an explicitly selected library backend with turbo6/turbo8 now raises a precise "supports 1-4 bit" error instead of the misleading "unavailable or failed"; 'auto' still falls back to the builtin quantizer silently. Verified: library turbo4 cos>0.994 vs originals, auto turbo8 builtin fallback intact. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The `dimensions` request param truncated AFTER normalizing, so truncated vectors were no longer unit-norm and downstream cosine-via-dot-product math broke. Truncate then re-normalize (OpenAI semantics), shared by all backends. Verified on MRL-trained Qwen3-Embedding-4B: at 1024/2560 dims retrieval quality is unchanged (0.900 vs 0.181 relevant/unrelated; full-dim was 0.898 vs 0.189). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
docker commit snapshots the container's RUNTIME config, so the upgrade flow's --entrypoint coderai-upgrade override and -e CODERAI_UPGRADE_* vars were being committed into the image — leaving coderai:base with the upgrader as its default ENTRYPOINT, so the server "failed to start" (every new container ran the upgrade script, which as --user also dies on the root-owned app tree). Capture the image's original ENTRYPOINT/CMD before the upgrade run and re-apply them via docker commit --change, and blank the injected env vars. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Support the three planned embedding models end-to-end (all verified on GPU): - 'vision' backend (DINOv2/ViT family): image-only encoders have no text tower or tokenizer, so none of the existing paths could load them. AutoImageProcessor + AutoModel, CLS/pooler embedding, normalized. Text input returns a clear 400. Verified: facebook/dinov2-large, 1024-d, similar-image 0.966 vs 0.750 cross. - 'qwenvl' backend (GME-Qwen2-VL…): the repo's remote code (auto_map + custom_st) pins transformers<4.52 and refuses to load on this stack, so load the plain Qwen2-VL weights through the CONCRETE native classes (which never consult auto_map) and reproduce GME's embedding recipe: their chat prompt format and last-token pooling, fp16, normalized. Shared 1536-d text+image space verified cross-modally (OCR-style test). No trust_remote_code needed. - ST image path: custom multimodal ST modules take {"image": …} dicts, not bare PIL (kept PIL-first for ST's native CLIP wrapper). - Tighten _supports_images: ST's plain Transformer module also exposes a `processor` attribute, which made text-only models (Qwen3-Embedding) look image-capable; now require an actual image-capable processor. Verified: Qwen/Qwen3-Embedding-4B (ST, 2560-d), facebook/dinov2-large (vision, 1024-d), Alibaba-NLP/gme-Qwen2-VL-2B-Instruct (qwenvl, 1536-d cross-modal). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A full `pip install -r requirements.txt` on upgrade is fragile: the repo's requirements list can contain entries that don't cleanly reinstall in the image env (e.g. kokoro-tts has no py3.13 wheel — the image provisions it another way), so reprocessing the whole file fails on a pre-existing line and aborts the upgrade. Instead, diff the normalized requirement specs (old tree vs fetched) and pip-install only the new/changed lines; satisfied/unclean existing lines are left untouched. Empty delta => no pip run. --force no longer implies a full reinstall (it forces the code refresh; deps still follow the delta). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- Add openssh-client to the image (Dockerfile.oci + the update overlay): git ships in the base but the CUDA runtime has no ssh, so `--upgrade --ssh-key` failed with `ssh: not found`. HTTPS upgrades already worked without it. - run_oci.sh: fix the --upgrade banner — `(force)` showed even when force was 0 (`${VAR:+…}` fires on the string "0"), and the auth line printed the ssh key path twice. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
Make `coderai-docker --upgrade` a complete in-image update, not just an app-code swap: - Dependency sync: fingerprint the declared deps (requirements.txt + the OCI extras) before/after fetch; when they change (or with --force) re-run pip into /opt/coderai/python using the same effective set as the image build. Native CUDA builds pinned `>=` stay satisfied so pip never clobbers them. A pip failure aborts before commit so a half-upgraded image is never produced. --no-pip (CODERAI_UPGRADE_SKIP_PIP) refreshes code only. - System-file refresh: the launchers + nginx/supervisord configs live outside /opt/coderai/app, so the code swap alone never updated them (including this upgrade script itself). Mirror Dockerfile.update's COPY set, writing each via temp+atomic-rename so replacing the running coderai-upgrade is safe. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Add a runtime code-upgrade path so shipped images can pick up new code without rebuilding or shipping an overlay image. - launcher/coderai-upgrade: in-image upgrader. Shallow-clones the configured ref (default: production branch of the nexlab repo), compares its codai/__init__.py __version__ against the installed one, and rsyncs the fetched tree over /opt/coderai/app when newer (or with --force). Mirrors the build's path exclusions (.git/venv*/dist/…). Exit 0=updated, 10=up-to-date, else error. Auth over HTTPS by default, or SSH when a key path is provided. - run_oci.sh: add --upgrade/--force/--upgrade-ref/--upgrade-repo/--ssh-key. Runs the in-image upgrader in a throwaway container (as root, --entrypoint) and, only when it reports an update, docker-commits the result back onto the SAME image tag — no rebuild, no new overlay. SSH key is bind-mounted read-only. - Dockerfile.update / Dockerfile.oci: install /usr/local/bin/coderai-upgrade. Co-Authored-By:
-
- 21 Jul, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
The session cookie is named session_<port> so two instances on one host don't clobber each other's cookie, but the lightweight front telemetry auth only checked the literal "session" name — 401'ing every front-served status call. Match the engine's get_current_user and accept any session / session_* cookie. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Wire up real image embeddings and make embedding models first-class in the VRAM lifecycle. - api/embeddings.py: detect CLIP/SigLIP dual encoders and drive them through transformers get_text_features/get_image_features so text and images share one projected space (ST path kept for repos shipping a native recipe). request.image is now actually read (URL/data-URI/path/base64), vectors are appended after the text ones, and text-only models return a clear 400. Handle the transformers 5.x pooled-output return shape. - Wrap the loaded model in _EmbeddingModel (unpacks as (backend, model) but exposes cleanup()) and register it via add_model() + record_vram_delta() on the request path, so it is measured, LRU-tracked, and cleanly evicted like every other model type instead of leaking as a bare tuple. - admin: fix the model-load button for embeddings (was routed to the diffusers loader) to use _load_embedding_model, matching the request path. - admin: backfill used_vram_gb after a download completes, since the entry is saved before the weights exist on disk; factor the estimate into a shared _estimate_used_vram_gb helper. A freshly-downloaded CLIP/SigLIP entry now always carries a reasonable estimate so pre-load eviction sizes correctly. Co-Authored-By:
-
- 03 Jul, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authored
Two changes to the Run-page character/environment generation: 1. Generate FROM SCRATCH via the text model instead of the built-in static pool. stage_characters/stage_environments always iterated FIGHTER_POOL/ ENVIRONMENT_POOL and used their hardcoded (pre-fallback) prompts — the LLM was never invoked for a full run. New _invent_profiles() calls the text model (_autogen_profile_payload) to invent fresh profiles; the static pool is only a fallback when no text model is configured (with a clear warning). 2. Phase the pipeline: invent ALL prompts first, then render ALL reference images, then train image LoRAs, then video LoRAs (prompts → images → image-LoRA → video-LoRA). New _render_profile_images() does the image phase from the saved prompts; the reuse/skip paths are unchanged. num_fighters/num_environments set how many to invent (default: the pool size). (CLI main() still uses the pool-based stage_characters/stage_environments; the web Run page is the phased-from-scratch path.) Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
evict_cosited_siblings() frees the co-located sibling's VRAM ONCE at training start, but training runs for minutes as an in-engine background job that the front swap-gate can't cover (its POST returns a job_id immediately). So a concurrent LLM request reloaded the gguf text model mid-training and OOM'd the trainer (one fighter LoRA failed while others succeeded). Add codai/models/gpu_lock.py: a cross-engine GPU reservation. Training reserves the card for its whole duration — locally AND on co-located siblings via new /internal/gpu-reserve + /internal/gpu-release endpoints — and every ordinary model-load path (manager.request_model, video _load_video_pipeline, image _load_diffusers_pipeline) calls wait_until_free() first. A request that needs a load during training now BLOCKS until training releases, then loads and serves — the same queue-behind-the-owner behaviour the swap-gate gives request-level work, now extended to cover training. The training thread is exempt from its own reservation, so loading the base model never self-deadlocks; waits are bounded (900s) so a stuck reservation can't hang forever. Validated: sibling reservation blocks a loader thread until release; the reserving thread never waits on itself. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
diffusers' Model.set_attention_backend() doesn't just set per-processor backends — it ALSO flips a process-wide active backend (attention_dispatch's _active_backend). The video path sets that to flash-attn for the Wan transformer; image and video share the nvidia-engine process, so the global stayed flash and leaked to the next image model. Z-Image's transformer sets no backend of its own (passes backend=None → uses the global) and its attention is masked, so it crashed with "`attn_mask` is not supported for flash-attn 2" → image/environment generation 400. reset_attention_backend() clears per-processor backends but NOT the global, so it didn't help. Fix: restore the diffusers global backend to the env default (native/SDPA) (a) before every image generation — bulletproof against a leaked flash backend — and (b) in the video pipeline teardown (_free_pipeline_vram), so it can't persist after a video pipe is freed. Masked image attention (SDPA) now always works; the video transformer keeps its own per-processor backend. Co-Authored-By:
-
- 02 Jul, 2026 2 commits
-
-
Stefy Lanza (nextime / spora ) authored
LoRA training freed VRAM with unload_all_models(), which only unloads THIS engine's models. On the GGUF-isolation split the co-located gguf (text) engine kept its model resident (~7.4 GB), so fp32 training (~16 GB) + the sibling exceeded the 24 GB card → "CUDA out of memory. Tried to allocate 32 MiB … 26 MiB free … Process 226 has 7.36 GiB" — every fighter LoRA (dlaba, zigo, zlo, …) failed. Training also isn't covered by the front swap-gate, so nothing else cleared the sibling. Add multi_model_manager.evict_cosited_siblings(): invoke the registered cross-engine VRAM releasers (the cosite releaser posts wait=True, so it waits for a busy sibling to reach a safe point). Call it right after unload_all_models() in both training paths (image + video/Wan), so training gets the whole card. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Z-Image-Turbo-unsloth-bnb-4bit (and any pre-quantized bnb/fp8/nf4/gptq/awq checkpoint, or a runtime-quantized model) dequantizes to a HALF compute dtype and its transformer uses FlashAttention, which only supports fp16/bf16. The per-model image loader defaults precision to f32, so such a model loaded in float32 and crashed with "FlashAttention only support fp16 and bf16 data type" (image/character generation → 400/500), besides wasting VRAM. When precision is left at the f32 default AND the model is quantized (name contains bnb/4bit/8bit/fp8/nf4/gptq/awq, or config sets load_in_4bit/8bit/ component_quantization), load in bf16 instead. Non-quantized models keep the f32 default. --no-ram already forced fp16, so it's unaffected. Co-Authored-By:
-
- 01 Jul, 2026 4 commits
-
-
Stefy Lanza (nextime / spora ) authored
A cache dir's completion marker only proves the save FINISHED, not that every file landed intact. A transient truncated write — repeatedly a 0-byte tokenizer/tokenizer_config.json — slipped into an otherwise "valid" cache and then threw JSONDecodeError on EVERY subsequent video load, knocking the pipeline off its fast path into the offload fallback ladder (balanced→sequential→disk), which churned for hours, leaked ~22 GB VRAM, and died on a meta-tensor error. Add _first_bad_json(dir): walk the (small) JSONs and flag the first that is empty or unparseable — the big weights are .safetensors and aren't scanned, so it's cheap. Wire it in on BOTH sides: - load: valid() and component_valid() now invalidate + return False when any cached JSON is corrupt, so a poisoned cache becomes a clean rebuild instead of a death spiral. - save: save()/save_component() verify the temp dir before committing, and mark_monolithic_complete() refuses to finalize a dir with a corrupt JSON — so a truncated write is never cached in the first place. Added invalidate_path(p) helper. Verified: _first_bad_json flags 0-byte and garbage JSONs, passes clean ones. The already-poisoned Wan2.2-VACE cache was deleted out-of-band to unblock. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The GpuSwapGate.release() was async and awaited in the dispatch `finally` blocks. When a request was cancelled/interrupted mid-flight (client disconnect, an interrupted text generation), `await self._swap_release(...)` in the finally could itself be cancelled BEFORE it decremented the running counter — stranding a gate slot. With `running` stuck > 0, `_pump()` never ran, so a video request queued behind the interrupted text request was never granted: the GPU never swapped even though the text engine had gone idle (observed: video stuck ~37min while the text engine was idle for 11). Fix: make release()/_pump() SYNCHRONOUS and drop the asyncio.Lock. Every critical section is straight-line (no await between read and write), so under asyncio's single thread they're already atomic — and a synchronous release from a `finally` always completes even while the coroutine is being cancelled. acquire() keeps its one `await` (the event wait) with synchronous cancel-cleanup. All release call sites are now non-awaited. Added `[gpu-swap] queued/swapping` logging so the owner/queue/swap transitions are visible in debug.log. Validated: cancelling a text request whose slot a queued video is waiting behind now frees the slot and grants the video; a cancelled queued waiter leaks nothing. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Builds on the cross-engine clean-swap eviction: instead of two engines on one shared card ever running forwards concurrently (→ VRAM contention → OOM → disk-thrash), the front now serializes model OWNERSHIP of a shared GPU while batching to avoid per-request thrash. New GpuSwapGate (frontproxy/reqqueue.py), one per shared-GPU group (keyed by the co-located engines' CODERAI_ENGINE_GPUS selector, created only when an engine has a sibling on its card): * A request for the model that currently OWNS the GPU runs immediately — a swap isn't needed (a lone stream never stalls). Concurrency stays capped downstream by the existing per-model FrontQueue. * A request for a DIFFERENT model queues. The owner keeps being served up to `cap` requests (server.gpu_swap_batch, default 10) while another model waits, then — once the owner is fully idle (never mid-request) — the GPU SWAPS to the waiting model (which evicts + loads), serves it, and round-robins BACK if the original has requests queued. No thrash (batch), no starvation (cap). Wired into all four dispatch paths (broker, broker-stream, direct stream with keepalive, direct non-stream) for every GPU-inference kind (text/image/video): acquire the swap slot before the per-model queue, release in the finalizer; cancelling a pending acquire (client disconnect) drops the waiter with no leak. The text-stream path emits keepalives while waiting out a swap so the client doesn't time out. Scheduler validated by async unit tests: cap engages at exactly N with a competitor waiting; a lone same-model stream runs unbounded; round-robin alternates; cancelled waiters leak no slot. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
On the GGUF-isolation split, a torch (video/image) engine and a gguf (text) engine share one NVIDIA card. When one needed VRAM it asked the co-located sibling to release via /internal/evict-vram, but that only evicted the sibling's IDLE models and SKIPPED busy ones — so a text-model load would proceed into the VRAM an in-flight video clip still needed for its forward, and BOTH OOM'd. Recovery then laddered the video load down to disk offload and thrashed for ~1h. Give the cross-engine path the same wait-then-evict the local eviction already has: release_idle_vram(needed_gb, wait_for_busy, wait_timeout) first evicts idle models, then — only if still short — WAITS for each busy model to reach a safe idle point (between requests, e.g. between video clip parts) and evicts it. This converts contention into a CLEAN SWAP: the render's current unit finishes, its model is evicted, the sibling loads alone, and the render reloads + resumes on its next unit. Bounded by wait_timeout (180s) so two mutually-waiting busy engines can't deadlock — one gives up and falls back to its own CPU/disk offload. /internal/evict-vram now reads needed_gb + wait + wait_timeout from the body and forwards them; _cosite_vram_releaser sends wait=True with an HTTP timeout that exceeds the sibling's wait budget so the swap isn't cut short. Symmetric: both engines register the releaser at each other, so either direction swaps cleanly. Co-Authored-By:
-
- 30 Jun, 2026 3 commits
-
-
Stefy Lanza (nextime / spora ) authored
Two fixes for the township video render failing with CUDA OOM and a ~17 GB "untracked teardown leak" that survived gc + empty_cache. 1. Resident-experts regression. video_resident_experts now defaults to OFF. A dual-expert 14B model (Wan2.2-VACE-Fun: transformer + transformer_2, ~10 GB each at 4-bit) cannot hold both experts + text encoder + VAE + the activation peak in 24 GB; the resident load left transformer_2 on the CPU yet reported success, so the denoise loop (which needs both experts) OOM'd at step 0. 'model' CPU offload keeps only the active ~7 GB expert resident and swaps, so it fits. Also: when resident leaves ANY component off-GPU it is now treated as a failed load — the partial pipe is torn down and it falls through to model offload, instead of returning a half-loaded pipe that pins ~10 GB. 2. Teardown leak. _free_pipeline_vram now breaks the references that outlived a plain component-null: reset any non-default attention backend, unload LoRA/PEFT adapters, run the pipe's own maybe_free_model_hooks()/reset_device_map() (frees accelerate offload hooks + their staging buffers), and drop the _coderai_* stamped attrs, before nulling components + gc + empty_cache. Verified live (image 0.1.33): 31 video units rendered, 0 OOM, 0 leak diagnostics, idle GPU back to ~6 GB. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Add a @media (max-width:640px) block to the shared _CSS injected into every township page via _page(). On narrow screens: stack the .row/.row3/.modal .row2 form grids to one column, wrap the nav bar, shrink the modal to fit a 320px viewport (min-width:0; width:94%), make the fixed-width 215/230px tile cards full-width, render inputs at 16px to stop iOS Safari focus-zoom, and give buttons roomier wrap-friendly tap targets. Desktop layout is unchanged. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The diffusers video path uses SageAttention (INT8 attention) when available for faster Wan2.2 rendering. Like flash-attn it is CUDA-arch-sensitive, so it is built from source in the devel/builder stage against the just-installed torch, gated by BUILD_SAGEATTENTION (default 1) and SAGEATTENTION_ARCH (default 8.6 = RTX 3090). The build is non-fatal: on failure the image still works and the runtime attention-backend resolver falls back to flash/SDPA. build_oci_image.sh passes the three new args through (overridable via env: BUILD_SAGEATTENTION / SAGEATTENTION_REF / SAGEATTENTION_ARCH). Note: the fast update_oci_image.sh overlay is based on the runtime image (no nvcc), so it cannot build SageAttention — a full build_oci_image.sh is needed to bake it in. Co-Authored-By:
-
- 29 Jun, 2026 1 commit
-
-
Stefy Lanza (nextime / spora ) authored
The 'auto' attention-backend resolver picked 'sage' whenever the sageattention package was merely importable. But diffusers requires a specific version with compiled CUDA kernels (0.38 needs >=2.1.1); an old v1 wheel is importable yet rejected, which would silently drop to plain SDPA instead of Flash. Gate on diffusers' own _CAN_USE_SAGE_ATTN (with an is_sageattention_version fallback) so 'auto' chooses sage only when it can actually dispatch, else flash, else SDPA. SageAttention 2.2.0 is now built from source in the venv (CUDA kernels, TORCH_CUDA_ARCH_LIST=8.6), so 'auto' resolves to sage on this host. Co-Authored-By:
-