• Stefy Lanza (nextime / spora )'s avatar
    manager: don't strand VRAM or churn the live model on eviction · bb95ab1d
    Stefy Lanza (nextime / spora ) authored
    Two eviction fixes folded into 0.1.30:
    
    - VRAM eviction: a diffusers pipeline under device_map/accelerate offload
      (balanced/disk/model/sequential) or bitsandbytes quantization REJECTS
      .to('cpu') — the naive move silently left the weights resident and stranded
      VRAM, feeding the OOM death-spiral where the next load OOMs on a near-full
      card. Route pipelines (those exposing `components`) through the thorough
      _free_pipeline_vram (remove accelerate hooks -> drop component refs ->
      empty_cache); plain non-pipeline models still use a simple .to('cpu').
    
    - RAM eviction: never unload the LIVE model (active_in_vram OR current_model_key)
      that a request loop keeps reusing. An offloaded pipeline's host RAM IS the live
      model, so evicting it between same-model requests just forces an immediate
      reload — churning VRAM/RAM and re-stranding device_map weights while freeing
      nothing lasting. The last-resort active-model eviction now fires only for a
      STALE active model (no longer the current one).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    bb95ab1d
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...