From the latest render log: the per-component cache + same-model reuse worked
(clip 02 ran on clip 01's resident pipe, LoRA-synced, no reload), but BIG clips
(46-50 frames) OOM'd in the dual-expert forward under 'model' CPU offload, and the
single retry then crashed two ways:
  - NameError: loading_task was imported only in the load-only block, so on a
    REUSED pipe the retry's `with loading_task(...)` was unbound.
  - enable_sequential_cpu_offload on injected bnb-4bit components hits
    "Params4bit.__new__() got an unexpected keyword argument '_is_hf_initialized'".
So every big clip returned 500 instead of recovering.

Fixes:
1. Generation fallback LADDER. On a recoverable failure (OOM / device-mismatch) we
   no longer return 5xx — we free the pipe (outside the except, so the failed
   forward's traceback can't pin its VRAM) and reload with the next strategy, then
   retry generation. Return on the FIRST success, or 500 only after EVERY rung
   fails. Rungs: [already-loaded] -> balanced -> disk, all PLAIN device_map builds
   (incremental=False) so device_map places every component itself — sidestepping
   the injected-component failure modes ('model' OOM, 'sequential' Params4bit,
   device-map-can't-place-injected). loading_task imported up-front.
2. Monolithic cache finalize. After the per-component heavy build, complete the
   cache DIR (save light components + model_index.json + marker) so later loads use
   the monolithic HIT path: from_pretrained(dir, device_map=balanced), NO injection
   — big-clip generation fits AND loads pre-quantized from cache (fast, no
   re-quantize). Conservative: only marked complete when every component is present.

Net: first clip builds+finalizes the cache; thereafter loads are balanced-from-cache
(fits any clip size, no injection bugs), same-model clips reuse, and a transient OOM
walks the ladder instead of failing the clip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce