• Stefy Lanza (nextime / spora )'s avatar
    vram: don't ratchet down a model's learned GPU-layer split under contention · f54428cf
    Stefy Lanza (nextime / spora ) authored
    record_vram_delta persisted measured_n_gpu_layers from every load, including
    VRAM-CONTENDED ones (another model resident → llama.cpp fit fewer layers). Since
    _learned_n_gpu_layers reuses that as a hard cap for auto (-1) configs, one unlucky
    contended load pinned a model half on CPU permanently — it never recovered even
    when the card was later free.
    
    Observed on the gemma-4-26B-A4B 32k config (alias lisa-32k): a contended load
    ratcheted 31→20 GPU layers and stuck, spilling ~5.8 GB of weights + KV to CPU on
    a 3090 with 11 GB free.
    
    Fix: in record_vram_delta, never overwrite a HIGHER known-good measured_n_gpu_layers
    with a LOWER value from an offloaded (CPU-spilled) load. A genuinely-too-big model
    still settles at its free-card maximum on an uncontended load, so keeping the higher
    value is safe; worst case is one offload-retry on a later truly-contended load.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    f54428cf
Name
Last commit
Last update
..
cache Loading commit data...
__init__.py Loading commit data...
acceleration.py Loading commit data...
capabilities.py Loading commit data...
gpu_lock.py Loading commit data...
gpu_query.py Loading commit data...
grammar.py Loading commit data...
hf_loading.py Loading commit data...
manager.py Loading commit data...
parser.py Loading commit data...
peft_compat.py Loading commit data...
pipeline_cache.py Loading commit data...
quant.py Loading commit data...
ram_monitor.py Loading commit data...
templates.py Loading commit data...
thermal.py Loading commit data...
tmp_janitor.py Loading commit data...
tool_call_grammar.gbnf Loading commit data...
turboquant.py Loading commit data...
utils.py Loading commit data...