• Stefy Lanza (nextime / spora )'s avatar
    gpu: performance split strategy (fast card first) — per-model + global, UI-configurable · 64187292
    Stefy Lanza (nextime / spora ) authored
    The auto split is VRAM-proportional, which hands a big share to a slow second GPU
    (e.g. RX 580) so it bottlenecks throughput (Radeon ~100%, 3090 ~30%). Add a
    "performance" strategy that fills the fast lead card (main_gpu) first and spills
    only the overflow to the slower card(s), using the model's expected size — so the
    weak GPU holds the fewest layers (and none at all if the model fits on the lead
    card). "vram" (default) keeps the capacity-maximizing behavior.
    
    - config offload.split_strategy ("vram"|"performance") global default, persisted +
      pushed to live global_args; per-model split_strategy overrides it.
    - plumbed via build_runtime_kwargs + manager _cfg_or_global into vulkan auto-split.
    - model config modal: "Split strategy" select shown with the split options; global
      Settings: default split strategy select.
    - split_strategy added to the live-reload (_LOAD_AFFECTING) set so changing it
      re-applies on the next request without a restart.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    64187292
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...