• Stefy Lanza (nextime / spora )'s avatar
    vram: don't ratchet down a model's learned GPU-layer split under contention · f54428cf
    Stefy Lanza (nextime / spora ) authored
    record_vram_delta persisted measured_n_gpu_layers from every load, including
    VRAM-CONTENDED ones (another model resident → llama.cpp fit fewer layers). Since
    _learned_n_gpu_layers reuses that as a hard cap for auto (-1) configs, one unlucky
    contended load pinned a model half on CPU permanently — it never recovered even
    when the card was later free.
    
    Observed on the gemma-4-26B-A4B 32k config (alias lisa-32k): a contended load
    ratcheted 31→20 GPU layers and stuck, spilling ~5.8 GB of weights + KV to CPU on
    a 3090 with 11 GB free.
    
    Fix: in record_vram_delta, never overwrite a HIGHER known-good measured_n_gpu_layers
    with a LOWER value from an offloaded (CPU-spilled) load. A genuinely-too-big model
    still settles at its free-card maximum on an uncontended load, so keeping the higher
    value is safe; worst case is one offload-retry on a later truly-contended load.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    f54428cf
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...