• Stefy Lanza (nextime / spora )'s avatar
    gpu: cross-GPU VRAM accounting for split models (Phase 3) · 32f20536
    Stefy Lanza (nextime / spora ) authored
    When cross-backend pooling (offload.gpu_split) is on a model can span several
    cards, so VRAM math must consider the pool, not one device:
    
    - vulkan.py: new _pooled_free_vram_gb() sums free VRAM across every visible card
      (all CUDA devices + AMD via amdgpu sysfs; no double count). The GGUF auto-offload
      fit decision uses the pooled figure when the model is gpu_split, so layers aren't
      needlessly pushed to CPU just because one card's free VRAM looks small.
    - manager._get_free_vram_gb(): when gpu_split is enabled, report pooled free across
      all GPUs (used by the eviction loop + budget checks), so eviction frees/measures
      capacity across both cards. Off → unchanged single-device behavior.
    
    Pooling is aggregate (sum); strict per-device fitting/targeted eviction can be a
    later refinement.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    32f20536
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...