• Stefy Lanza (nextime / spora )'s avatar
    video: incremental per-component pipeline cache (load/build/cache one component at a time) · 93e2a4b9
    Stefy Lanza (nextime / spora ) authored
    A Wan2.2 A14B pipeline that only loads via offload could never be cached — diffusers
    can't save_pretrained an offloaded pipe, so every load re-quantized from bf16 shards
    (~14 min) and risked OOM. Cache it per-component instead:
    
    - pipeline_cache: per-component subdirs each with their own completion marker +
      signature (component_dir / component_valid / save_component), atomic via .building
      + sweep_stale(). A load evicted before all components are cached keeps the finished
      ones; the next load mixes cached + freshly-built (now-cached) ones, converging to a
      full cache. (Plus the earlier sweep_stale / _unsavable_reason robustness.)
    
    - hf_loading.build_cached_components: for each bnb-quantizable heavy component, load
      from the per-component cache if present, else build fresh from the model — each ONE
      AT A TIME on the GPU (where bnb quantizes it to uniform 4-bit), saved, then moved to
      CPU + empty_cache. Peak VRAM is a single component even when the whole model doesn't
      fit; uniform 4-bit by construction (no bf16/4-bit mix). Verified the bnb round-trip
      (load 4-bit on GPU -> save_pretrained -> to('cpu') -> reload).
    
    - video._load_video_pipeline: inject the cached components like GGUF components so the
      existing offload ladder assembles them. Kill-switch CODERAI_INCREMENTAL_CACHE=0.
      Fully fallback-safe: on any load failure the request handler retries once with
      use_incremental_cache=False (plain build = today's behaviour).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    93e2a4b9
Name
Last commit
Last update
..
cache Loading commit data...
__init__.py Loading commit data...
acceleration.py Loading commit data...
capabilities.py Loading commit data...
gpu_query.py Loading commit data...
grammar.py Loading commit data...
hf_loading.py Loading commit data...
manager.py Loading commit data...
parser.py Loading commit data...
peft_compat.py Loading commit data...
pipeline_cache.py Loading commit data...
quant.py Loading commit data...
ram_monitor.py Loading commit data...
templates.py Loading commit data...
thermal.py Loading commit data...
tmp_janitor.py Loading commit data...
tool_call_grammar.gbnf Loading commit data...
turboquant.py Loading commit data...
utils.py Loading commit data...