• Stefy Lanza (nextime / spora )'s avatar
    gpu: per-engine backend isolation (fix docker cross-GPU split) + opt-in... · 018396c1
    Stefy Lanza (nextime / spora ) authored
    gpu: per-engine backend isolation (fix docker cross-GPU split) + opt-in per-model cross-backend pooling
    
    Phase 1 — fix the docker-only "model loads on both GPUs":
    - gpu_detect.vendor_env detects each Vulkan device's vendor and pins each engine
      to ONLY its own backend's cards by real indices (not assumed 0..n-1). When a
      vendor has no Vulkan device (e.g. NVIDIA in a container that lacks nvidia_icd.json
      because the toolkit only injects it with the graphics capability), the engine
      gets ZERO Vulkan and runs CUDA-only instead of falling back to all ICDs and
      grabbing the Radeon via RADV. Same-backend split (e.g. 2x 3090) is preserved.
    
    Phase 2 — opt-in cross-backend GPU pooling, per model:
    - OffloadConfig.gpu_split (default off) + tensor_split ("0.8,0.2", llama.cpp
      device order: CUDA first then Vulkan); global default + per-model override.
    - vendor_env(allow_cross=…) exposes the foreign card when enabled; the engine
      supervisor passes it from config.
    - manager threads gpu_split/tensor_split (per-model via _raw_cfg, else global via
      global_args) into the GGUF loader; vulkan.py sets llama.cpp tensor_split when on
      and otherwise leaves split_mode=LAYER so same-backend split still works.
    - admin model-configure accepts gpu_split + tensor_split per model.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    018396c1
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...