video: resident all-GPU experts + faster attention backend (flash/sage)

Two opt-out speedups for the diffusers video path (both default ON), aimed
at phase-3 render time on the dual-expert Wan2.2-VACE-Fun-A14B.

1) Resident experts (video_resident_experts, default on)
   The injected bnb-4bit components previously forced hook-based 'model'
   CPU offload, which re-shuffles modules GPU<->CPU on every clip part
   (text-encode, VACE-encode, VAE-decode all pay a transfer). Now the
   whole pipeline (both 4-bit experts + text encoder + VAE, ~20 GB) is
   loaded RESIDENT on the 24 GB card via a new 'resident' load strategy:
   build with the injected components, then .to('cuda') every component,
   no offload hooks. The denoise loop and en/decode run on resident
   weights with zero per-part shuffling.
   Fully fallback-safe: on an activation-peak OOM the loader degrades to
   'model' offload, and the generation ladder gains a ('model', True)
   rung as the first fallback when rung 0 was resident — i.e. exactly the
   previous behaviour. Set video_resident_experts=false to force it.
   record_vram_delta now treats 'resident' as on-GPU (not offloaded) so
   the measured footprint is the true GPU delta.

2) Attention backend (video_attention_backend, default 'auto')
   Switch the transformer(s) to a faster attention backend via diffusers
   0.38's set_attention_backend, applied at the single _report_loaded
   chokepoint (covers initial + every fallback reload). 'auto' prefers
   SageAttention (INT8) if installed, else FlashAttention (flash_attn is
   installed), else leaves the default SDPA. Per-component try/except so
   an unavailable backend is a logged no-op.
   Self-heal: if a non-OOM generation failure occurs while a non-default
   backend is active (a flash/sage shape incompat on this torch build),
   the transformers are reset to default attention and the SAME pipe is
   retried once before any costly reload rung.

Config: both read per-model (models.json) first, then global_args, then
the default — no config change needed to get the speedups. Untested on a
live render (no GPU run this session); behaviour falls back to the prior
path on any failure.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
parent f4ce8539
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment