• Stefy Lanza (nextime / spora )'s avatar
    gpu: cross-GPU VRAM accounting for split models (Phase 3) · 32f20536
    Stefy Lanza (nextime / spora ) authored
    When cross-backend pooling (offload.gpu_split) is on a model can span several
    cards, so VRAM math must consider the pool, not one device:
    
    - vulkan.py: new _pooled_free_vram_gb() sums free VRAM across every visible card
      (all CUDA devices + AMD via amdgpu sysfs; no double count). The GGUF auto-offload
      fit decision uses the pooled figure when the model is gpu_split, so layers aren't
      needlessly pushed to CPU just because one card's free VRAM looks small.
    - manager._get_free_vram_gb(): when gpu_split is enabled, report pooled free across
      all GPUs (used by the eviction loop + budget checks), so eviction frees/measures
      capacity across both cards. Off → unchanged single-device behavior.
    
    Pooling is aggregate (sum); strict per-device fitting/targeted eviction can be a
    later refinement.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    32f20536
manager.py 193 KB