• Stefy Lanza (nextime / spora )'s avatar
    manager: size a model by the config that was requested, not a same-path sibling · c5ae720e
    Stefy Lanza (nextime / spora ) authored
    The second lisa-32k request evicted the model that was already serving it, then
    reloaded the same file at 18 of 65 layers:
    
      # >>> REQUEST  model=lisa-32k
      [vram-est] key='/AI/guffcache/gemma-4-26B-…gguf' -> 51.97 GB
      Loading '/AI/guffcache/gemma-4-26B-…gguf': need 52.0 GB, have 12.6 GB free — evicting
      On-request VRAM eviction: unloading active model 'lisa-32k' (measured 14.33 GB)
      n_gpu_layers : 18  |  n_ctx : 32768
    
    lisa and lisa-32k are the same weights file at two contexts — 178000 and 32768 —
    so they measure 51.97 GB and 14.33 GB. The VRAM estimate was keyed by the PATH,
    which matches whichever sibling registered it, so the small config was sized with
    the big one's footprint: a 52 GB requirement that nothing could satisfy, an
    eviction of the running model, and a crippled reload.
    
    _get_model_used_vram_gb now prefers the config of the name actually requested when
    it names a same-path sibling. Four pairs on this box were affected, including the
    qwen38 / qwen38-128k pair added yesterday.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    c5ae720e
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...