video: revert to model offload for injected components (the leak, not model offload, was the OOM)

Follow-up to the generation-OOM leak fix: forcing 'sequential' was treating the
symptom. The real cause of "model offload OOMs generation" was that the card
already had ~11 GB LEAKED from the prior clip's failed forward (its exception
traceback pinned the activations). model offload itself adds ~0 to the GPU (weights
stay on CPU); subtract the leak and the forward fits in 24 GB.

With the leak fixed (free + retry now happen outside the except, so the pinned
activations are collectable), loads start from a clean card, so force the faster
'model' CPU offload for injected components instead of sequential. The leak-free
retry still degrades to sequential only if a genuinely clean-card forward OOMs.

Why load-time cleanup couldn't fix it on its own: the leaked VRAM was neither a
tracked model (already popped from the registry) nor free-able (still referenced by
the traceback), so manager eviction had nothing to evict and empty_cache() — which
only reclaims unreferenced memory — was a no-op. The only fix is dropping the
reference at its source, which the traceback fix does.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
parent bc31e2d5
...@@ -946,15 +946,16 @@ def _load_video_pipeline(model_name: str, device: str, mode: str, offload: str = ...@@ -946,15 +946,16 @@ def _load_video_pipeline(model_name: str, device: str, mode: str, offload: str =
# injected ones keep whatever device they're on with NO offload hook, and a # injected ones keep whatever device they're on with NO offload hook, and a
# forward pass then hits "index on cuda:0, weights on cpu". The HOOK-based # forward pass then hits "index on cuda:0, weights on cpu". The HOOK-based
# strategies add hooks to EVERY component (injected included), so force one. # strategies add hooks to EVERY component (injected included), so force one.
# Use SEQUENTIAL, not model: model CPU offload keeps a whole expert resident # Use 'model' (keeps only the active ~7 GB 4-bit expert resident) — it fits the
# and OOMs this dual-expert A14B forward (verified — loads at 11 GB then the # forward on a CLEAN card. An earlier "model offload OOMs generation" was a
# forward spikes past 24 GB). Sequential has the minimal footprint that # red herring: the card had ~11 GB LEAKED from the prior clip's failed forward
# reliably fits; the cache still makes the LOAD fast even if generation is # (exception traceback pinning its activations — now fixed), so the forward had
# slower. (A user wanting speed can set offload_strategy=group explicitly.) # only ~13 GB to work in. With clean VRAM model offload fits; the leak-free
if _have_injected_components and offload in (None, 'auto', 'balanced', 'disk', 'model'): # retry still degrades to sequential if a genuinely clean forward ever OOMs.
if _have_injected_components and offload in (None, 'auto', 'balanced', 'disk'):
print(f" [pipeline-cache] injected cached components → forcing hook-based " print(f" [pipeline-cache] injected cached components → forcing hook-based "
f"'sequential' CPU offload (device_map can't place pre-loaded components)") f"'model' CPU offload (device_map can't place pre-loaded components)")
offload = 'sequential' offload = 'model'
# Resolve offload directory for disk-offload fallback. # Resolve offload directory for disk-offload fallback.
_offload_dir = ( _offload_dir = (
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment