• Stefy Lanza (nextime / spora )'s avatar
    gpu: exclude llvmpipe from split, mmproj-aware headroom + CPU spill when pool is full · b5acbc2c
    Stefy Lanza (nextime / spora ) authored
    Three related fixes for cross-GPU split of a vision model at large context:
    
    1) gpu_detect: when exposing GPUs for cross-split (allow_cross), include only REAL
       hardware Vulkan devices (nvidia/amd/intel) and EXCLUDE software rasterizers
       (llvmpipe/lavapipe/virtio = vendor "other"). They're CPU-backed with no real
       VRAM, slower than native CPU offload, and their presence skewed the device list
       so the tensor_split ratio didn't line up with the actual cards (and layers could
       land on a fake GPU).
    
    2) vulkan auto tensor_split now reserves the mmproj's size + compute margin on
       main_gpu (the projector always loads there), so the proportional split doesn't
       fill main_gpu to the brim and abort when CLIP can't allocate
       (GGML_ASSERT(buffer) failed).
    
    3) auto-offload (n_gpu_layers sizing) subtracts the same mmproj reserve from the
       POOLED free VRAM, so when model+KV+projector exceed BOTH cards combined it
       reduces GPU layers and spills to CPU gracefully instead of crashing.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    b5acbc2c
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...