-
Stefy Lanza (nextime / spora ) authored
Letting a sized-to-fit cache hit try the GPU (1f42d08c) is right for a model that fits and expensive for one that does not. Measured on LTX-2.5: 04:55:07 RESIDENT all-GPU load (sized to fit) 04:56:53 [resident] components left off-GPU: transformer(CUDA OOM), vocoder(CUDA OOM), prompt_enhancer(CUDA OOM) 05:11:10 ✓ loaded — strategy: model CPU offload 05:11:12 23.28 GB total, of which 21.32 GB offloaded to host RAM Sixteen minutes to reach exactly the placement that would have been chosen immediately, and it would have paid that on every load. The sizing was not wrong so much as answering a different question: it knows what the weights weigh (22.1 GB against 25.0 GB free), while resident needs every component -- transformer, vocoder, VAE, text encoder, prompt_enhancer -- plus the activation peak, all at once. The real footprint turned out to be 23.28 GB. Only an attempt can answer that, so the answer is recorded as a system-owned learned field beside measured_n_gpu_layers, which exists for the same reason: stop repeating an expensive discovery every load. Both outcomes are recorded, not just the failure. Writing only failures would re-prove a model that fits on every load; writing only successes would never stop the expensive retry. And it is not a one-way door -- a later success clears an earlier failure, because a bigger card, a smaller quantisation or fewer co-resident models all change the answer. Unknown still means try. A model never attempted, or a card that has since changed, gets its chance at the GPU rather than being written off by default. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
6a7be148
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| admin | ||
| api | ||
| backends | ||
| broker | ||
| cluster | ||
| compose | ||
| frontproxy | ||
| models | ||
| ocr | ||
| openai | ||
| pipelines | ||
| pydantic | ||
| queue | ||
| tasks | ||
| util | ||
| __init__.py | ||
| cli.py | ||
| config.py | ||
| hf_cache.py | ||
| main.py | ||
| ocr_fleet_workers.py | ||
| platform_paths.py | ||
| system_app.py |