• Stefy Lanza (nextime / spora )'s avatar
    gpu: exclude llvmpipe from split, mmproj-aware headroom + CPU spill when pool is full · b5acbc2c
    Stefy Lanza (nextime / spora ) authored
    Three related fixes for cross-GPU split of a vision model at large context:
    
    1) gpu_detect: when exposing GPUs for cross-split (allow_cross), include only REAL
       hardware Vulkan devices (nvidia/amd/intel) and EXCLUDE software rasterizers
       (llvmpipe/lavapipe/virtio = vendor "other"). They're CPU-backed with no real
       VRAM, slower than native CPU offload, and their presence skewed the device list
       so the tensor_split ratio didn't line up with the actual cards (and layers could
       land on a fake GPU).
    
    2) vulkan auto tensor_split now reserves the mmproj's size + compute margin on
       main_gpu (the projector always loads there), so the proportional split doesn't
       fill main_gpu to the brim and abort when CLIP can't allocate
       (GGML_ASSERT(buffer) failed).
    
    3) auto-offload (n_gpu_layers sizing) subtracts the same mmproj reserve from the
       POOLED free VRAM, so when model+KV+projector exceed BOTH cards combined it
       reduces GPU layers and spills to CPU gracefully instead of crashing.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    b5acbc2c
gpu_detect.py 16.3 KB