• Stefy Lanza (nextime / spora )'s avatar
    learn once whether a model fits on the card, instead of re-proving it · 6a7be148
    Stefy Lanza (nextime / spora ) authored
    Letting a sized-to-fit cache hit try the GPU (1f42d08c) is right for a model
    that fits and expensive for one that does not. Measured on LTX-2.5:
    
        04:55:07  RESIDENT all-GPU load (sized to fit)
        04:56:53  [resident] components left off-GPU: transformer(CUDA OOM),
                  vocoder(CUDA OOM), prompt_enhancer(CUDA OOM)
        05:11:10  ✓ loaded — strategy: model CPU offload
        05:11:12  23.28 GB total, of which 21.32 GB offloaded to host RAM
    
    Sixteen minutes to reach exactly the placement that would have been chosen
    immediately, and it would have paid that on every load. The sizing was not
    wrong so much as answering a different question: it knows what the weights
    weigh (22.1 GB against 25.0 GB free), while resident needs every component --
    transformer, vocoder, VAE, text encoder, prompt_enhancer -- plus the
    activation peak, all at once. The real footprint turned out to be 23.28 GB.
    
    Only an attempt can answer that, so the answer is recorded as a system-owned
    learned field beside measured_n_gpu_layers, which exists for the same reason:
    stop repeating an expensive discovery every load.
    
    Both outcomes are recorded, not just the failure. Writing only failures would
    re-prove a model that fits on every load; writing only successes would never
    stop the expensive retry. And it is not a one-way door -- a later success
    clears an earlier failure, because a bigger card, a smaller quantisation or
    fewer co-resident models all change the answer.
    
    Unknown still means try. A model never attempted, or a card that has since
    changed, gets its chance at the GPU rather than being written off by default.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    6a7be148
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-acestep.txt Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-latentsync.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-musetalk.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...