-
Stefy Lanza (nextime / spora ) authored
record_vram_delta() measured free VRAM via _free_vram_snapshot(), which only read the default CUDA device (device 0). A gpu_split model (e.g. gemma on two cards) lands most weights on the other card, so the device-0 delta came out ~0, tripped the 'delta <= 0' guard, and never persisted measured_vram_gb. The estimate stayed 0 and every load OOM-retried from scratch even with force_vram_update set. Sum free VRAM across every visible CUDA device (scoped per engine via CUDA_VISIBLE_DEVICES), plus amdgpu sysfs when cross-pooling is on or no CUDA device is visible -- mirroring _get_free_vram_gb(). Before/after snapshots are now symmetric, so the delta captures the full multi-card footprint and the real value is persisted; subsequent loads size correctly and stop OOM-looping. Bump version to 0.1.3. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
80b21cf5
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| admin | ||
| api | ||
| backends | ||
| broker | ||
| frontproxy | ||
| models | ||
| openai | ||
| pydantic | ||
| queue | ||
| tasks | ||
| __init__.py | ||
| cli.py | ||
| config.py | ||
| main.py | ||
| platform_paths.py | ||
| system_app.py |