-
Stefy Lanza (nextime / spora ) authored
After the GGUF-isolation split, a torch engine and a gguf engine share one NVIDIA card but each can only evict its OWN models. So loading an image model on the torch engine while the gguf engine still holds a resident GGUF text model failed: local eviction "freed 0.0 GB ... VRAM held elsewhere" → CUDA OOM. Fix — co-located VRAM release: - front passes each engine its same-GPU siblings' internal URLs via CODERAI_COSITED_URLS (matched by identical CODERAI_ENGINE_GPUS selectors). - engine registers an external_vram_releaser that POSTs to each sibling's new /internal/evict-vram when local eviction can't free enough. - /internal/evict-vram → manager.release_idle_vram(): evicts all idle (non-busy) models and returns GB freed; busy/actively-serving models are left alone. Symmetric: the gguf engine can likewise reclaim VRAM from the torch engine's idle diffusers models. Driver-level free VRAM (cross-process) is re-checked after each releaser, so the loader proceeds once the sibling has freed enough. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
88063940
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| admin | ||
| api | ||
| backends | ||
| broker | ||
| frontproxy | ||
| models | ||
| openai | ||
| pydantic | ||
| queue | ||
| tasks | ||
| __init__.py | ||
| cli.py | ||
| config.py | ||
| main.py | ||
| platform_paths.py | ||
| system_app.py |