• Stefy Lanza (nextime / spora )'s avatar
    vram: measure multi-GPU load delta across all devices (fix split models stuck at 0) · 80b21cf5
    Stefy Lanza (nextime / spora ) authored
    record_vram_delta() measured free VRAM via _free_vram_snapshot(), which only
    read the default CUDA device (device 0). A gpu_split model (e.g. gemma on two
    cards) lands most weights on the other card, so the device-0 delta came out
    ~0, tripped the 'delta <= 0' guard, and never persisted measured_vram_gb. The
    estimate stayed 0 and every load OOM-retried from scratch even with
    force_vram_update set.
    
    Sum free VRAM across every visible CUDA device (scoped per engine via
    CUDA_VISIBLE_DEVICES), plus amdgpu sysfs when cross-pooling is on or no CUDA
    device is visible -- mirroring _get_free_vram_gb(). Before/after snapshots are
    now symmetric, so the delta captures the full multi-card footprint and the
    real value is persisted; subsequent loads size correctly and stop OOM-looping.
    
    Bump version to 0.1.3.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    80b21cf5
__init__.py 4.11 KB