-
Stefy Lanza (nextime / spora ) authored
Add a cap on how much VRAM the auto-split may place on EACH secondary (non-main) card, so a slow second GPU (e.g. RX 580) stays lightly loaded and bottlenecks throughput less — the remainder stays on the fast card or spills to CPU. - config offload.split_secondary_cap_gb: global default, persisted + pushed live to every engine's global_args (each engine reads config at startup, so it's global). - per-model split_secondary_cap_gb overrides — but only to TIGHTEN: the effective cap is min(global, per-model) of whichever are set; a per-model value higher than the global is ignored. - applied in vulkan auto-split: caps each non-main device's free VRAM in the ratio (both vram & performance strategies), and in the auto-offload pool so the excess proactively spills to CPU instead of only reacting to an OOM. - UI: model config modal "Secondary card VRAM cap" + global Settings field. - added to _LOAD_AFFECTING so changes re-apply live on the next request. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com>
82aec0f9