• Stefy Lanza (nextime / spora )'s avatar
    engines: cross-engine VRAM eviction for co-located GGUF/torch engines · 88063940
    Stefy Lanza (nextime / spora ) authored
    After the GGUF-isolation split, a torch engine and a gguf engine share one NVIDIA
    card but each can only evict its OWN models. So loading an image model on the torch
    engine while the gguf engine still holds a resident GGUF text model failed: local
    eviction "freed 0.0 GB ... VRAM held elsewhere" → CUDA OOM.
    
    Fix — co-located VRAM release:
    - front passes each engine its same-GPU siblings' internal URLs via
      CODERAI_COSITED_URLS (matched by identical CODERAI_ENGINE_GPUS selectors).
    - engine registers an external_vram_releaser that POSTs to each sibling's new
      /internal/evict-vram when local eviction can't free enough.
    - /internal/evict-vram → manager.release_idle_vram(): evicts all idle (non-busy)
      models and returns GB freed; busy/actively-serving models are left alone.
    
    Symmetric: the gguf engine can likewise reclaim VRAM from the torch engine's idle
    diffusers models. Driver-level free VRAM (cross-process) is re-checked after each
    releaser, so the loader proceeds once the sibling has freed enough.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    88063940
Name
Last commit
Last update
..
__init__.py Loading commit data...
_film_net.py Loading commit data...
_rife_ifnet.py Loading commit data...
app.py Loading commit data...
archive.py Loading commit data...
audio_backends.py Loading commit data...
audio_clean.py Loading commit data...
audio_gen.py Loading commit data...
audio_stems.py Loading commit data...
characters.py Loading commit data...
custom_pipelines.py Loading commit data...
ds4_kv_janitor.py Loading commit data...
ds4_worker.py Loading commit data...
embeddings.py Loading commit data...
environments.py Loading commit data...
faceswap.py Loading commit data...
fastlog.py Loading commit data...
images.py Loading commit data...
log.py Loading commit data...
loras.py Loading commit data...
parler_worker.py Loading commit data...
pipelines.py Loading commit data...
prompt_cache.py Loading commit data...
ratelimit.py Loading commit data...
spatial.py Loading commit data...
state.py Loading commit data...
text.py Loading commit data...
transcriptions.py Loading commit data...
tts.py Loading commit data...
tts_backends.py Loading commit data...
urlutils.py Loading commit data...
video.py Loading commit data...
voice_clone.py Loading commit data...
voice_convert.py Loading commit data...