• Stefy Lanza (nextime / spora )'s avatar
    video: resident all-GPU experts + faster attention backend (flash/sage) · fe2cdee1
    Stefy Lanza (nextime / spora ) authored
    Two opt-out speedups for the diffusers video path (both default ON), aimed
    at phase-3 render time on the dual-expert Wan2.2-VACE-Fun-A14B.
    
    1) Resident experts (video_resident_experts, default on)
       The injected bnb-4bit components previously forced hook-based 'model'
       CPU offload, which re-shuffles modules GPU<->CPU on every clip part
       (text-encode, VACE-encode, VAE-decode all pay a transfer). Now the
       whole pipeline (both 4-bit experts + text encoder + VAE, ~20 GB) is
       loaded RESIDENT on the 24 GB card via a new 'resident' load strategy:
       build with the injected components, then .to('cuda') every component,
       no offload hooks. The denoise loop and en/decode run on resident
       weights with zero per-part shuffling.
       Fully fallback-safe: on an activation-peak OOM the loader degrades to
       'model' offload, and the generation ladder gains a ('model', True)
       rung as the first fallback when rung 0 was resident — i.e. exactly the
       previous behaviour. Set video_resident_experts=false to force it.
       record_vram_delta now treats 'resident' as on-GPU (not offloaded) so
       the measured footprint is the true GPU delta.
    
    2) Attention backend (video_attention_backend, default 'auto')
       Switch the transformer(s) to a faster attention backend via diffusers
       0.38's set_attention_backend, applied at the single _report_loaded
       chokepoint (covers initial + every fallback reload). 'auto' prefers
       SageAttention (INT8) if installed, else FlashAttention (flash_attn is
       installed), else leaves the default SDPA. Per-component try/except so
       an unavailable backend is a logged no-op.
       Self-heal: if a non-OOM generation failure occurs while a non-default
       backend is active (a flash/sage shape incompat on this torch build),
       the transformers are reset to default attention and the SAME pipe is
       retried once before any costly reload rung.
    
    Config: both read per-model (models.json) first, then global_args, then
    the default — no config change needed to get the speedups. Untested on a
    live render (no GPU run this session); behaviour falls back to the prior
    path on any failure.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    fe2cdee1
video.py 174 KB