• Stefy Lanza (nextime / spora )'s avatar
    gpu: cross-GPU VRAM accounting for split models (Phase 3) · 32f20536
    Stefy Lanza (nextime / spora ) authored
    When cross-backend pooling (offload.gpu_split) is on a model can span several
    cards, so VRAM math must consider the pool, not one device:
    
    - vulkan.py: new _pooled_free_vram_gb() sums free VRAM across every visible card
      (all CUDA devices + AMD via amdgpu sysfs; no double count). The GGUF auto-offload
      fit decision uses the pooled figure when the model is gpu_split, so layers aren't
      needlessly pushed to CPU just because one card's free VRAM looks small.
    - manager._get_free_vram_gb(): when gpu_split is enabled, report pooled free across
      all GPUs (used by the eviction loop + budget checks), so eviction frees/measures
      capacity across both cards. Off → unchanged single-device behavior.
    
    Pooling is aggregate (sum); strict per-device fitting/targeted eviction can be a
    later refinement.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    32f20536
Name
Last commit
Last update
..
cache Loading commit data...
__init__.py Loading commit data...
acceleration.py Loading commit data...
capabilities.py Loading commit data...
grammar.py Loading commit data...
hf_loading.py Loading commit data...
manager.py Loading commit data...
parser.py Loading commit data...
pipeline_cache.py Loading commit data...
quant.py Loading commit data...
ram_monitor.py Loading commit data...
templates.py Loading commit data...
thermal.py Loading commit data...
tmp_janitor.py Loading commit data...
tool_call_grammar.gbnf Loading commit data...
turboquant.py Loading commit data...
utils.py Loading commit data...