• Stefy Lanza (nextime / spora )'s avatar
    gpu: secondary-card VRAM cap (per-model + global), effective = lowest of the two · 82aec0f9
    Stefy Lanza (nextime / spora ) authored
    Add a cap on how much VRAM the auto-split may place on EACH secondary (non-main)
    card, so a slow second GPU (e.g. RX 580) stays lightly loaded and bottlenecks
    throughput less — the remainder stays on the fast card or spills to CPU.
    
    - config offload.split_secondary_cap_gb: global default, persisted + pushed live to
      every engine's global_args (each engine reads config at startup, so it's global).
    - per-model split_secondary_cap_gb overrides — but only to TIGHTEN: the effective
      cap is min(global, per-model) of whichever are set; a per-model value higher than
      the global is ignored.
    - applied in vulkan auto-split: caps each non-main device's free VRAM in the ratio
      (both vram & performance strategies), and in the auto-offload pool so the excess
      proactively spills to CPU instead of only reacting to an OOM.
    - UI: model config modal "Secondary card VRAM cap" + global Settings field.
    - added to _LOAD_AFFECTING so changes re-apply live on the next request.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    82aec0f9
main.py 64.9 KB