• Stefy Lanza (nextime / spora )'s avatar
    gpu/live: honor per-model gpu_split (was offloading to CPU); apply model... · b3e9d1bb
    Stefy Lanza (nextime / spora ) authored
    gpu/live: honor per-model gpu_split (was offloading to CPU); apply model config live on any load-affecting change
    
    Two fixes:
    
    1) build_runtime_kwargs never promoted gpu_split/tensor_split to top-level kwargs
       (only into _raw_cfg), but the manager reads config['gpu_split'] via _cfg_or_global
       (which doesn't consult _raw_cfg). So a model set to "Split — <card> first" loaded
       gpu_split=False → confined to one card → its big context spilled to CPU instead
       of distributing to the second GPU. Now promoted, so the split is honored and the
       GGUF auto-offload sizes against the POOLED VRAM across both cards.
    
    2) apply_model_entry_live only evicted a loaded model when *acceleration* changed.
       Generalize to ANY load-affecting field (n_ctx, n_gpu_layers, gpu_split,
       tensor_split, cache types, kv_offload, n_batch/ubatch/seq_max, flash, quant,
       engine pin, vae/precision, …): if it changed and the model is loaded, evict it so
       the next request reloads with the new config — config changes apply immediately,
       no server restart.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    b3e9d1bb
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...