• Stefy Lanza (nextime / spora )'s avatar
    manager: size a model by the config that was requested, not a same-path sibling · c5ae720e
    Stefy Lanza (nextime / spora ) authored
    The second lisa-32k request evicted the model that was already serving it, then
    reloaded the same file at 18 of 65 layers:
    
      # >>> REQUEST  model=lisa-32k
      [vram-est] key='/AI/guffcache/gemma-4-26B-…gguf' -> 51.97 GB
      Loading '/AI/guffcache/gemma-4-26B-…gguf': need 52.0 GB, have 12.6 GB free — evicting
      On-request VRAM eviction: unloading active model 'lisa-32k' (measured 14.33 GB)
      n_gpu_layers : 18  |  n_ctx : 32768
    
    lisa and lisa-32k are the same weights file at two contexts — 178000 and 32768 —
    so they measure 51.97 GB and 14.33 GB. The VRAM estimate was keyed by the PATH,
    which matches whichever sibling registered it, so the small config was sized with
    the big one's footprint: a 52 GB requirement that nothing could satisfy, an
    eviction of the running model, and a crippled reload.
    
    _get_model_used_vram_gb now prefers the config of the name actually requested when
    it names a same-path sibling. Four pairs on this box were affected, including the
    qwen38 / qwen38-128k pair added yesterday.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    c5ae720e
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...