• Stefy Lanza (nextime / spora )'s avatar
    manager: sweep orphan VRAM + untracked-VRAM diagnostic before every video load · 2ab70301
    Stefy Lanza (nextime / spora ) authored
    Belt-and-suspenders for the "untracked in-use VRAM" failure class (a teardown that
    drops a model's registry entry but leaves GPU tensors resident — e.g. activations
    pinned by an exception traceback):
    
    - manager.sweep_orphan_vram(): gc.collect() + empty_cache() reclaims the
      UNREFERENCED kind (a just-dropped-but-not-collected pipeline), and logs a
      diagnostic when the card holds materially more VRAM than tracked models account
      for (the referenced kind, which can't be freed here — must be fixed at source).
    - manager._estimate_tracked_vram_gb(): sums tracked models' footprints for that
      check (offloaded models over-report, so the check is conservative, never a false
      positive on a legitimately offloaded model).
    - video: call it at the start of every load — right after request_model's eviction
      — so a model swap is verified to have actually freed the outgoing model's VRAM,
      and any stray orphan is swept before the new model loads.
    
    Same-model consecutive requests already reuse the resident (offloaded) pipe via
    request_model, so this only runs on a genuine (re)load.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    2ab70301
manager.py 208 KB