-
Stefy Lanza (nextime / spora ) authored
gpu: per-engine backend isolation (fix docker cross-GPU split) + opt-in per-model cross-backend pooling Phase 1 — fix the docker-only "model loads on both GPUs": - gpu_detect.vendor_env detects each Vulkan device's vendor and pins each engine to ONLY its own backend's cards by real indices (not assumed 0..n-1). When a vendor has no Vulkan device (e.g. NVIDIA in a container that lacks nvidia_icd.json because the toolkit only injects it with the graphics capability), the engine gets ZERO Vulkan and runs CUDA-only instead of falling back to all ICDs and grabbing the Radeon via RADV. Same-backend split (e.g. 2x 3090) is preserved. Phase 2 — opt-in cross-backend GPU pooling, per model: - OffloadConfig.gpu_split (default off) + tensor_split ("0.8,0.2", llama.cpp device order: CUDA first then Vulkan); global default + per-model override. - vendor_env(allow_cross=…) exposes the foreign card when enabled; the engine supervisor passes it from config. - manager threads gpu_split/tensor_split (per-model via _raw_cfg, else global via global_args) into the GGUF loader; vulkan.py sets llama.cpp tensor_split when on and otherwise leaves split_mode=LAYER so same-backend split still works. - admin model-configure accepts gpu_split + tensor_split per model. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com>
018396c1
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| static | ||
| templates | ||
| __init__.py | ||
| auth.py | ||
| download_worker.py | ||
| routes.py |