• Stefy Lanza (nextime / spora )'s avatar
    learn once whether a model fits on the card, instead of re-proving it · 6a7be148
    Stefy Lanza (nextime / spora ) authored
    Letting a sized-to-fit cache hit try the GPU (1f42d08c) is right for a model
    that fits and expensive for one that does not. Measured on LTX-2.5:
    
        04:55:07  RESIDENT all-GPU load (sized to fit)
        04:56:53  [resident] components left off-GPU: transformer(CUDA OOM),
                  vocoder(CUDA OOM), prompt_enhancer(CUDA OOM)
        05:11:10  ✓ loaded — strategy: model CPU offload
        05:11:12  23.28 GB total, of which 21.32 GB offloaded to host RAM
    
    Sixteen minutes to reach exactly the placement that would have been chosen
    immediately, and it would have paid that on every load. The sizing was not
    wrong so much as answering a different question: it knows what the weights
    weigh (22.1 GB against 25.0 GB free), while resident needs every component --
    transformer, vocoder, VAE, text encoder, prompt_enhancer -- plus the
    activation peak, all at once. The real footprint turned out to be 23.28 GB.
    
    Only an attempt can answer that, so the answer is recorded as a system-owned
    learned field beside measured_n_gpu_layers, which exists for the same reason:
    stop repeating an expensive discovery every load.
    
    Both outcomes are recorded, not just the failure. Writing only failures would
    re-prove a model that fits on every load; writing only successes would never
    stop the expensive retry. And it is not a one-way door -- a later success
    clears an earlier failure, because a bigger card, a smaller quantisation or
    fewer co-resident models all change the answer.
    
    Unknown still means try. A model never attempted, or a card that has since
    changed, gets its chance at the GPU rather than being written off by default.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    6a7be148
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
compose Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pipelines Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
util Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
hf_cache.py Loading commit data...
main.py Loading commit data...
ocr_fleet_workers.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...