-
Stefy Lanza (nextime / spora ) authored
A Wan2.2 A14B pipeline that only loads via offload could never be cached — diffusers can't save_pretrained an offloaded pipe, so every load re-quantized from bf16 shards (~14 min) and risked OOM. Cache it per-component instead: - pipeline_cache: per-component subdirs each with their own completion marker + signature (component_dir / component_valid / save_component), atomic via .building + sweep_stale(). A load evicted before all components are cached keeps the finished ones; the next load mixes cached + freshly-built (now-cached) ones, converging to a full cache. (Plus the earlier sweep_stale / _unsavable_reason robustness.) - hf_loading.build_cached_components: for each bnb-quantizable heavy component, load from the per-component cache if present, else build fresh from the model — each ONE AT A TIME on the GPU (where bnb quantizes it to uniform 4-bit), saved, then moved to CPU + empty_cache. Peak VRAM is a single component even when the whole model doesn't fit; uniform 4-bit by construction (no bf16/4-bit mix). Verified the bnb round-trip (load 4-bit on GPU -> save_pretrained -> to('cpu') -> reload). - video._load_video_pipeline: inject the cached components like GGUF components so the existing offload ladder assembles them. Kill-switch CODERAI_INCREMENTAL_CACHE=0. Fully fallback-safe: on any load failure the request handler retries once with use_incremental_cache=False (plain build = today's behaviour). Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
93e2a4b9