• Stefy Lanza (nextime / spora )'s avatar
    gpu: make VRAM accounting/eviction ALWAYS multi-device, not just under split · 067a797e
    Stefy Lanza (nextime / spora ) authored
    Previously pooling was gated on gpu_split, so a 2-card same-backend engine (e.g.
    2× 3090, or 2× Radeon) still measured only one device for the fit/eviction math.
    
    Now both manager._get_free_vram_gb() and vulkan._pooled_free_vram_gb():
    - ALWAYS sum every visible CUDA device (torch honours CUDA_VISIBLE_DEVICES, so it
      is scoped to this engine's NVIDIA cards) → same-backend split is accounted for
      with no flag.
    - add AMD card(s) (amdgpu sysfs) only when cross-backend split is on OR no CUDA
      device is visible (a Radeon/Vulkan engine), so a Radeon engine counts its own
      cards and an NVIDIA engine only reaches across to Radeon when split is enabled.
    
    So: 2× NVIDIA → summed across both NVIDIAs; 2× Radeon → summed across both Radeons;
    split on → summed across all NVIDIA + Radeon.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    067a797e
Name
Last commit
Last update
..
cache Loading commit data...
__init__.py Loading commit data...
acceleration.py Loading commit data...
capabilities.py Loading commit data...
grammar.py Loading commit data...
hf_loading.py Loading commit data...
manager.py Loading commit data...
parser.py Loading commit data...
pipeline_cache.py Loading commit data...
quant.py Loading commit data...
ram_monitor.py Loading commit data...
templates.py Loading commit data...
thermal.py Loading commit data...
tmp_janitor.py Loading commit data...
tool_call_grammar.gbnf Loading commit data...
turboquant.py Loading commit data...
utils.py Loading commit data...