Skip to content
Commit 88063940 authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files

engines: cross-engine VRAM eviction for co-located GGUF/torch engines



After the GGUF-isolation split, a torch engine and a gguf engine share one NVIDIA
card but each can only evict its OWN models. So loading an image model on the torch
engine while the gguf engine still holds a resident GGUF text model failed: local
eviction "freed 0.0 GB ... VRAM held elsewhere" → CUDA OOM.

Fix — co-located VRAM release:
- front passes each engine its same-GPU siblings' internal URLs via
  CODERAI_COSITED_URLS (matched by identical CODERAI_ENGINE_GPUS selectors).
- engine registers an external_vram_releaser that POSTs to each sibling's new
  /internal/evict-vram when local eviction can't free enough.
- /internal/evict-vram → manager.release_idle_vram(): evicts all idle (non-busy)
  models and returns GB freed; busy/actively-serving models are left alone.

Symmetric: the gguf engine can likewise reclaim VRAM from the torch engine's idle
diffusers models. Driver-level free VRAM (cross-process) is re-checked after each
releaser, so the loader proceeds once the sibling has freed enough.

Co-Authored-By: default avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
parent ec1f7218
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment