• Stefy Lanza (nextime / spora )'s avatar
    gpu: secondary-card VRAM cap (per-model + global), effective = lowest of the two · 82aec0f9
    Stefy Lanza (nextime / spora ) authored
    Add a cap on how much VRAM the auto-split may place on EACH secondary (non-main)
    card, so a slow second GPU (e.g. RX 580) stays lightly loaded and bottlenecks
    throughput less — the remainder stays on the fast card or spills to CPU.
    
    - config offload.split_secondary_cap_gb: global default, persisted + pushed live to
      every engine's global_args (each engine reads config at startup, so it's global).
    - per-model split_secondary_cap_gb overrides — but only to TIGHTEN: the effective
      cap is min(global, per-model) of whichever are set; a per-model value higher than
      the global is ignored.
    - applied in vulkan auto-split: caps each non-main device's free VRAM in the ratio
      (both vram & performance strategies), and in the auto-offload pool so the excess
      proactively spills to CPU instead of only reacting to an OOM.
    - UI: model config modal "Secondary card VRAM cap" + global Settings field.
    - added to _LOAD_AFFECTING so changes re-apply live on the next request.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    82aec0f9
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...