• Stefy Lanza (nextime / spora )'s avatar
    gpu/live: honor per-model gpu_split (was offloading to CPU); apply model... · b3e9d1bb
    Stefy Lanza (nextime / spora ) authored
    gpu/live: honor per-model gpu_split (was offloading to CPU); apply model config live on any load-affecting change
    
    Two fixes:
    
    1) build_runtime_kwargs never promoted gpu_split/tensor_split to top-level kwargs
       (only into _raw_cfg), but the manager reads config['gpu_split'] via _cfg_or_global
       (which doesn't consult _raw_cfg). So a model set to "Split — <card> first" loaded
       gpu_split=False → confined to one card → its big context spilled to CPU instead
       of distributing to the second GPU. Now promoted, so the split is honored and the
       GGUF auto-offload sizes against the POOLED VRAM across both cards.
    
    2) apply_model_entry_live only evicted a loaded model when *acceleration* changed.
       Generalize to ANY load-affecting field (n_ctx, n_gpu_layers, gpu_split,
       tensor_split, cache types, kv_offload, n_batch/ubatch/seq_max, flash, quant,
       engine pin, vae/precision, …): if it changed and the model is loaded, evict it so
       the next request reloads with the new config — config changes apply immediately,
       no server restart.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    b3e9d1bb
main.py 64.5 KB