• Stefy Lanza (nextime / spora )'s avatar
    manager: don't strand VRAM or churn the live model on eviction · bb95ab1d
    Stefy Lanza (nextime / spora ) authored
    Two eviction fixes folded into 0.1.30:
    
    - VRAM eviction: a diffusers pipeline under device_map/accelerate offload
      (balanced/disk/model/sequential) or bitsandbytes quantization REJECTS
      .to('cpu') — the naive move silently left the weights resident and stranded
      VRAM, feeding the OOM death-spiral where the next load OOMs on a near-full
      card. Route pipelines (those exposing `components`) through the thorough
      _free_pipeline_vram (remove accelerate hooks -> drop component refs ->
      empty_cache); plain non-pipeline models still use a simple .to('cpu').
    
    - RAM eviction: never unload the LIVE model (active_in_vram OR current_model_key)
      that a request loop keeps reusing. An offloaded pipeline's host RAM IS the live
      model, so evicting it between same-model requests just forces an immediate
      reload — churning VRAM/RAM and re-stranding device_map weights while freeing
      nothing lasting. The last-resort active-model eviction now fires only for a
      STALE active model (no longer the current one).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    bb95ab1d
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...