• Stefy Lanza (nextime / spora )'s avatar
    engines: cross-engine VRAM eviction for co-located GGUF/torch engines · 88063940
    Stefy Lanza (nextime / spora ) authored
    After the GGUF-isolation split, a torch engine and a gguf engine share one NVIDIA
    card but each can only evict its OWN models. So loading an image model on the torch
    engine while the gguf engine still holds a resident GGUF text model failed: local
    eviction "freed 0.0 GB ... VRAM held elsewhere" → CUDA OOM.
    
    Fix — co-located VRAM release:
    - front passes each engine its same-GPU siblings' internal URLs via
      CODERAI_COSITED_URLS (matched by identical CODERAI_ENGINE_GPUS selectors).
    - engine registers an external_vram_releaser that POSTs to each sibling's new
      /internal/evict-vram when local eviction can't free enough.
    - /internal/evict-vram → manager.release_idle_vram(): evicts all idle (non-busy)
      models and returns GB freed; busy/actively-serving models are left alone.
    
    Symmetric: the gguf engine can likewise reclaim VRAM from the torch engine's idle
    diffusers models. Driver-level free VRAM (cross-process) is re-checked after each
    releaser, so the loader proceeds once the sibling has freed enough.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    88063940
main.py 68.7 KB