-
Stefy Lanza (nextime / spora ) authored
Follow-up to the generation-OOM leak fix: forcing 'sequential' was treating the symptom. The real cause of "model offload OOMs generation" was that the card already had ~11 GB LEAKED from the prior clip's failed forward (its exception traceback pinned the activations). model offload itself adds ~0 to the GPU (weights stay on CPU); subtract the leak and the forward fits in 24 GB. With the leak fixed (free + retry now happen outside the except, so the pinned activations are collectable), loads start from a clean card, so force the faster 'model' CPU offload for injected components instead of sequential. The leak-free retry still degrades to sequential only if a genuinely clean-card forward OOMs. Why load-time cleanup couldn't fix it on its own: the leaked VRAM was neither a tracked model (already popped from the registry) nor free-able (still referenced by the traceback), so manager eviction had nothing to evict and empty_cache() — which only reclaims unreferenced memory — was a no-op. The only fix is dropping the reference at its source, which the traceback fix does. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
1dbf15ed
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| admin | ||
| api | ||
| backends | ||
| broker | ||
| frontproxy | ||
| models | ||
| openai | ||
| pydantic | ||
| queue | ||
| tasks | ||
| __init__.py | ||
| cli.py | ||
| config.py | ||
| main.py | ||
| platform_paths.py | ||
| system_app.py |