• Stefy Lanza (nextime / spora )'s avatar
    video: resident all-GPU experts + faster attention backend (flash/sage) · fe2cdee1
    Stefy Lanza (nextime / spora ) authored
    Two opt-out speedups for the diffusers video path (both default ON), aimed
    at phase-3 render time on the dual-expert Wan2.2-VACE-Fun-A14B.
    
    1) Resident experts (video_resident_experts, default on)
       The injected bnb-4bit components previously forced hook-based 'model'
       CPU offload, which re-shuffles modules GPU<->CPU on every clip part
       (text-encode, VACE-encode, VAE-decode all pay a transfer). Now the
       whole pipeline (both 4-bit experts + text encoder + VAE, ~20 GB) is
       loaded RESIDENT on the 24 GB card via a new 'resident' load strategy:
       build with the injected components, then .to('cuda') every component,
       no offload hooks. The denoise loop and en/decode run on resident
       weights with zero per-part shuffling.
       Fully fallback-safe: on an activation-peak OOM the loader degrades to
       'model' offload, and the generation ladder gains a ('model', True)
       rung as the first fallback when rung 0 was resident — i.e. exactly the
       previous behaviour. Set video_resident_experts=false to force it.
       record_vram_delta now treats 'resident' as on-GPU (not offloaded) so
       the measured footprint is the true GPU delta.
    
    2) Attention backend (video_attention_backend, default 'auto')
       Switch the transformer(s) to a faster attention backend via diffusers
       0.38's set_attention_backend, applied at the single _report_loaded
       chokepoint (covers initial + every fallback reload). 'auto' prefers
       SageAttention (INT8) if installed, else FlashAttention (flash_attn is
       installed), else leaves the default SDPA. Per-component try/except so
       an unavailable backend is a logged no-op.
       Self-heal: if a non-OOM generation failure occurs while a non-default
       backend is active (a flash/sage shape incompat on this torch build),
       the transformers are reset to default attention and the SAME pipe is
       retried once before any costly reload rung.
    
    Config: both read per-model (models.json) first, then global_args, then
    the default — no config change needed to get the speedups. Untested on a
    live render (no GPU run this session); behaviour falls back to the prior
    path on any failure.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    fe2cdee1
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...