-
Stefy Lanza (nextime / spora ) authored
gpu/live: honor per-model gpu_split (was offloading to CPU); apply model config live on any load-affecting change Two fixes: 1) build_runtime_kwargs never promoted gpu_split/tensor_split to top-level kwargs (only into _raw_cfg), but the manager reads config['gpu_split'] via _cfg_or_global (which doesn't consult _raw_cfg). So a model set to "Split — <card> first" loaded gpu_split=False → confined to one card → its big context spilled to CPU instead of distributing to the second GPU. Now promoted, so the split is honored and the GGUF auto-offload sizes against the POOLED VRAM across both cards. 2) apply_model_entry_live only evicted a loaded model when *acceleration* changed. Generalize to ANY load-affecting field (n_ctx, n_gpu_layers, gpu_split, tensor_split, cache types, kv_offload, n_batch/ubatch/seq_max, flash, quant, engine pin, vae/precision, …): if it changed and the model is loaded, evict it so the next request reloads with the new config — config changes apply immediately, no server restart. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com>
b3e9d1bb