• Stefy Lanza (nextime / spora )'s avatar
    manager: don't strand VRAM or churn the live model on eviction · bb95ab1d
    Stefy Lanza (nextime / spora ) authored
    Two eviction fixes folded into 0.1.30:
    
    - VRAM eviction: a diffusers pipeline under device_map/accelerate offload
      (balanced/disk/model/sequential) or bitsandbytes quantization REJECTS
      .to('cpu') — the naive move silently left the weights resident and stranded
      VRAM, feeding the OOM death-spiral where the next load OOMs on a near-full
      card. Route pipelines (those exposing `components`) through the thorough
      _free_pipeline_vram (remove accelerate hooks -> drop component refs ->
      empty_cache); plain non-pipeline models still use a simple .to('cpu').
    
    - RAM eviction: never unload the LIVE model (active_in_vram OR current_model_key)
      that a request loop keeps reusing. An offloaded pipeline's host RAM IS the live
      model, so evicting it between same-model requests just forces an immediate
      reload — churning VRAM/RAM and re-stranding device_map weights while freeing
      nothing lasting. The last-resort active-model eviction now fires only for a
      STALE active model (no longer the current one).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    bb95ab1d
Name
Last commit
Last update
..
cache Loading commit data...
__init__.py Loading commit data...
acceleration.py Loading commit data...
capabilities.py Loading commit data...
gpu_query.py Loading commit data...
grammar.py Loading commit data...
hf_loading.py Loading commit data...
manager.py Loading commit data...
parser.py Loading commit data...
peft_compat.py Loading commit data...
pipeline_cache.py Loading commit data...
quant.py Loading commit data...
ram_monitor.py Loading commit data...
templates.py Loading commit data...
thermal.py Loading commit data...
tmp_janitor.py Loading commit data...
tool_call_grammar.gbnf Loading commit data...
turboquant.py Loading commit data...
utils.py Loading commit data...