• Stefy Lanza (nextime / spora )'s avatar
    gpu/live: honor per-model gpu_split (was offloading to CPU); apply model... · b3e9d1bb
    Stefy Lanza (nextime / spora ) authored
    gpu/live: honor per-model gpu_split (was offloading to CPU); apply model config live on any load-affecting change
    
    Two fixes:
    
    1) build_runtime_kwargs never promoted gpu_split/tensor_split to top-level kwargs
       (only into _raw_cfg), but the manager reads config['gpu_split'] via _cfg_or_global
       (which doesn't consult _raw_cfg). So a model set to "Split — <card> first" loaded
       gpu_split=False → confined to one card → its big context spilled to CPU instead
       of distributing to the second GPU. Now promoted, so the split is honored and the
       GGUF auto-offload sizes against the POOLED VRAM across both cards.
    
    2) apply_model_entry_live only evicted a loaded model when *acceleration* changed.
       Generalize to ANY load-affecting field (n_ctx, n_gpu_layers, gpu_split,
       tensor_split, cache types, kv_offload, n_batch/ubatch/seq_max, flash, quant,
       engine pin, vae/precision, …): if it changed and the model is loaded, evict it so
       the next request reloads with the new config — config changes apply immediately,
       no server restart.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    b3e9d1bb
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...