• Stefy Lanza (nextime / spora )'s avatar
    gpu: exclude llvmpipe from split, mmproj-aware headroom + CPU spill when pool is full · b5acbc2c
    Stefy Lanza (nextime / spora ) authored
    Three related fixes for cross-GPU split of a vision model at large context:
    
    1) gpu_detect: when exposing GPUs for cross-split (allow_cross), include only REAL
       hardware Vulkan devices (nvidia/amd/intel) and EXCLUDE software rasterizers
       (llvmpipe/lavapipe/virtio = vendor "other"). They're CPU-backed with no real
       VRAM, slower than native CPU offload, and their presence skewed the device list
       so the tensor_split ratio didn't line up with the actual cards (and layers could
       land on a fake GPU).
    
    2) vulkan auto tensor_split now reserves the mmproj's size + compute margin on
       main_gpu (the projector always loads there), so the proportional split doesn't
       fill main_gpu to the brim and abort when CLIP can't allocate
       (GGML_ASSERT(buffer) failed).
    
    3) auto-offload (n_gpu_layers sizing) subtracts the same mmproj reserve from the
       POOLED free VRAM, so when model+KV+projector exceed BOTH cards combined it
       reduces GPU layers and spills to CPU gracefully instead of crashing.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    b5acbc2c
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...