• Stefy Lanza (nextime / spora )'s avatar
    gpu: per-engine backend isolation (fix docker cross-GPU split) + opt-in... · 018396c1
    Stefy Lanza (nextime / spora ) authored
    gpu: per-engine backend isolation (fix docker cross-GPU split) + opt-in per-model cross-backend pooling
    
    Phase 1 — fix the docker-only "model loads on both GPUs":
    - gpu_detect.vendor_env detects each Vulkan device's vendor and pins each engine
      to ONLY its own backend's cards by real indices (not assumed 0..n-1). When a
      vendor has no Vulkan device (e.g. NVIDIA in a container that lacks nvidia_icd.json
      because the toolkit only injects it with the graphics capability), the engine
      gets ZERO Vulkan and runs CUDA-only instead of falling back to all ICDs and
      grabbing the Radeon via RADV. Same-backend split (e.g. 2x 3090) is preserved.
    
    Phase 2 — opt-in cross-backend GPU pooling, per model:
    - OffloadConfig.gpu_split (default off) + tensor_split ("0.8,0.2", llama.cpp
      device order: CUDA first then Vulkan); global default + per-model override.
    - vendor_env(allow_cross=…) exposes the foreign card when enabled; the engine
      supervisor passes it from config.
    - manager threads gpu_split/tensor_split (per-model via _raw_cfg, else global via
      global_args) into the GGUF loader; vulkan.py sets llama.cpp tensor_split when on
      and otherwise leaves split_mode=LAYER so same-backend split still works.
    - admin model-configure accepts gpu_split + tensor_split per model.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    018396c1
Name
Last commit
Last update
..
static Loading commit data...
templates Loading commit data...
__init__.py Loading commit data...
auth.py Loading commit data...
download_worker.py Loading commit data...
routes.py Loading commit data...