-
Stefy Lanza (nextime / spora ) authored
Two eviction fixes folded into 0.1.30: - VRAM eviction: a diffusers pipeline under device_map/accelerate offload (balanced/disk/model/sequential) or bitsandbytes quantization REJECTS .to('cpu') — the naive move silently left the weights resident and stranded VRAM, feeding the OOM death-spiral where the next load OOMs on a near-full card. Route pipelines (those exposing `components`) through the thorough _free_pipeline_vram (remove accelerate hooks -> drop component refs -> empty_cache); plain non-pipeline models still use a simple .to('cpu'). - RAM eviction: never unload the LIVE model (active_in_vram OR current_model_key) that a request loop keeps reusing. An offloaded pipeline's host RAM IS the live model, so evicting it between same-model requests just forces an immediate reload — churning VRAM/RAM and re-stranding device_map weights while freeing nothing lasting. The last-resort active-model eviction now fires only for a STALE active model (no longer the current one). Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
bb95ab1d
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| admin | ||
| api | ||
| backends | ||
| broker | ||
| frontproxy | ||
| models | ||
| openai | ||
| pydantic | ||
| queue | ||
| tasks | ||
| __init__.py | ||
| cli.py | ||
| config.py | ||
| main.py | ||
| platform_paths.py | ||
| system_app.py |