video: resident all-GPU experts + faster attention backend (flash/sage)
Two opt-out speedups for the diffusers video path (both default ON), aimed
at phase-3 render time on the dual-expert Wan2.2-VACE-Fun-A14B.
1) Resident experts (video_resident_experts, default on)
The injected bnb-4bit components previously forced hook-based 'model'
CPU offload, which re-shuffles modules GPU<->CPU on every clip part
(text-encode, VACE-encode, VAE-decode all pay a transfer). Now the
whole pipeline (both 4-bit experts + text encoder + VAE, ~20 GB) is
loaded RESIDENT on the 24 GB card via a new 'resident' load strategy:
build with the injected components, then .to('cuda') every component,
no offload hooks. The denoise loop and en/decode run on resident
weights with zero per-part shuffling.
Fully fallback-safe: on an activation-peak OOM the loader degrades to
'model' offload, and the generation ladder gains a ('model', True)
rung as the first fallback when rung 0 was resident — i.e. exactly the
previous behaviour. Set video_resident_experts=false to force it.
record_vram_delta now treats 'resident' as on-GPU (not offloaded) so
the measured footprint is the true GPU delta.
2) Attention backend (video_attention_backend, default 'auto')
Switch the transformer(s) to a faster attention backend via diffusers
0.38's set_attention_backend, applied at the single _report_loaded
chokepoint (covers initial + every fallback reload). 'auto' prefers
SageAttention (INT8) if installed, else FlashAttention (flash_attn is
installed), else leaves the default SDPA. Per-component try/except so
an unavailable backend is a logged no-op.
Self-heal: if a non-OOM generation failure occurs while a non-default
backend is active (a flash/sage shape incompat on this torch build),
the transformers are reset to default attention and the SAME pipe is
retried once before any costly reload rung.
Config: both read per-model (models.json) first, then global_args, then
the default — no config change needed to get the speedups. Untested on a
live render (no GPU run this session); behaviour falls back to the prior
path on any failure.
Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
Showing
This diff is collapsed.
Please
register
or
sign in
to comment