• Stefy Lanza (nextime / spora )'s avatar
    vram: don't ratchet down a model's learned GPU-layer split under contention · f54428cf
    Stefy Lanza (nextime / spora ) authored
    record_vram_delta persisted measured_n_gpu_layers from every load, including
    VRAM-CONTENDED ones (another model resident → llama.cpp fit fewer layers). Since
    _learned_n_gpu_layers reuses that as a hard cap for auto (-1) configs, one unlucky
    contended load pinned a model half on CPU permanently — it never recovered even
    when the card was later free.
    
    Observed on the gemma-4-26B-A4B 32k config (alias lisa-32k): a contended load
    ratcheted 31→20 GPU layers and stuck, spilling ~5.8 GB of weights + KV to CPU on
    a 3090 with 11 GB free.
    
    Fix: in record_vram_delta, never overwrite a HIGHER known-good measured_n_gpu_layers
    with a LOWER value from an offloaded (CPU-spilled) load. A genuinely-too-big model
    still settles at its free-card maximum on an uncontended load, so keeping the higher
    value is safe; worst case is one offload-retry on a later truly-contended load.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    f54428cf
manager.py 243 KB