-
Stefy Lanza (nextime / spora ) authored
When cross-backend pooling (offload.gpu_split) is on a model can span several cards, so VRAM math must consider the pool, not one device: - vulkan.py: new _pooled_free_vram_gb() sums free VRAM across every visible card (all CUDA devices + AMD via amdgpu sysfs; no double count). The GGUF auto-offload fit decision uses the pooled figure when the model is gpu_split, so layers aren't needlessly pushed to CPU just because one card's free VRAM looks small. - manager._get_free_vram_gb(): when gpu_split is enabled, report pooled free across all GPUs (used by the eviction loop + budget checks), so eviction frees/measures capacity across both cards. Off → unchanged single-device behavior. Pooling is aggregate (sum); strict per-device fitting/targeted eviction can be a later refinement. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com>
32f20536