• Stefy Lanza (nextime / spora )'s avatar
    vram: don't ratchet down a model's learned GPU-layer split under contention · f54428cf
    Stefy Lanza (nextime / spora ) authored
    record_vram_delta persisted measured_n_gpu_layers from every load, including
    VRAM-CONTENDED ones (another model resident → llama.cpp fit fewer layers). Since
    _learned_n_gpu_layers reuses that as a hard cap for auto (-1) configs, one unlucky
    contended load pinned a model half on CPU permanently — it never recovered even
    when the card was later free.
    
    Observed on the gemma-4-26B-A4B 32k config (alias lisa-32k): a contended load
    ratcheted 31→20 GPU layers and stuck, spilling ~5.8 GB of weights + KV to CPU on
    a 3090 with 11 GB free.
    
    Fix: in record_vram_delta, never overwrite a HIGHER known-good measured_n_gpu_layers
    with a LOWER value from an offloaded (CPU-spilled) load. A genuinely-too-big model
    still settles at its free-card maximum on an uncontended load, so keeping the higher
    value is safe; worst case is one offload-retry on a later truly-contended load.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    f54428cf
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...