• Stefy Lanza (nextime / spora )'s avatar
    video: incremental per-component pipeline cache (load/build/cache one component at a time) · 93e2a4b9
    Stefy Lanza (nextime / spora ) authored
    A Wan2.2 A14B pipeline that only loads via offload could never be cached — diffusers
    can't save_pretrained an offloaded pipe, so every load re-quantized from bf16 shards
    (~14 min) and risked OOM. Cache it per-component instead:
    
    - pipeline_cache: per-component subdirs each with their own completion marker +
      signature (component_dir / component_valid / save_component), atomic via .building
      + sweep_stale(). A load evicted before all components are cached keeps the finished
      ones; the next load mixes cached + freshly-built (now-cached) ones, converging to a
      full cache. (Plus the earlier sweep_stale / _unsavable_reason robustness.)
    
    - hf_loading.build_cached_components: for each bnb-quantizable heavy component, load
      from the per-component cache if present, else build fresh from the model — each ONE
      AT A TIME on the GPU (where bnb quantizes it to uniform 4-bit), saved, then moved to
      CPU + empty_cache. Peak VRAM is a single component even when the whole model doesn't
      fit; uniform 4-bit by construction (no bf16/4-bit mix). Verified the bnb round-trip
      (load 4-bit on GPU -> save_pretrained -> to('cpu') -> reload).
    
    - video._load_video_pipeline: inject the cached components like GGUF components so the
      existing offload ladder assembles them. Kill-switch CODERAI_INCREMENTAL_CACHE=0.
      Fully fallback-safe: on any load failure the request handler retries once with
      use_incremental_cache=False (plain build = today's behaviour).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    93e2a4b9
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...