manager: sweep orphan VRAM + untracked-VRAM diagnostic before every video load

Belt-and-suspenders for the "untracked in-use VRAM" failure class (a teardown that
drops a model's registry entry but leaves GPU tensors resident — e.g. activations
pinned by an exception traceback):

- manager.sweep_orphan_vram(): gc.collect() + empty_cache() reclaims the
  UNREFERENCED kind (a just-dropped-but-not-collected pipeline), and logs a
  diagnostic when the card holds materially more VRAM than tracked models account
  for (the referenced kind, which can't be freed here — must be fixed at source).
- manager._estimate_tracked_vram_gb(): sums tracked models' footprints for that
  check (offloaded models over-report, so the check is conservative, never a false
  positive on a legitimately offloaded model).
- video: call it at the start of every load — right after request_model's eviction
  — so a model swap is verified to have actually freed the outgoing model's VRAM,
  and any stray orphan is swept before the new model loads.

Same-model consecutive requests already reuse the resident (offloaded) pipe via
request_model, so this only runs on a genuine (re)load.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

codai/api/video.py

@@ -850,6 +850,15 @@ def _load_video_pipeline(model_name: str, device: str, mode: str, offload: str =
         return _load_sdcpp_video_model(model_name, offload, model_cfg)
 
     import sys, time, torch, gc
+    # Start every load from a swept card: reclaim any UNREFERENCED orphaned VRAM a
+    # prior teardown dropped but didn't collect, and surface a diagnostic if the
+    # card holds VRAM no tracked model accounts for (a referenced teardown leak we
+    # can't free here). This runs right after request_model's eviction, so it's also
+    # the "make sure the swapped-out model actually freed its VRAM" check.
+    try:
+        multi_model_manager.sweep_orphan_vram(context="before video load")
+    except Exception:
+        pass
     PClass = _detect_pipeline_class(model_name, mode)
     if PClass is None:
         raise RuntimeError("diffusers not installed: pip install diffusers")

codai/models/manager.py

@@ -2293,6 +2293,57 @@ class MultiModelManager:
         except Exception:
             return False
 
+    def _estimate_tracked_vram_gb(self) -> float:
+        """Sum the estimated VRAM footprint of every currently-tracked model.
+
+        Used to detect UNTRACKED in-use VRAM (a teardown leak): if the card reports
+        far more VRAM in use than this accounts for, something dropped a model's
+        registry entry but left its GPU tensors resident (e.g. activations pinned by
+        an exception traceback). Offloaded models over-report here (their full
+        footprint vs the small GPU-resident slice), which only makes the check
+        CONSERVATIVE — it never false-positives on a legitimately offloaded model."""
+        total = 0.0
+        for key in list(self.models.keys()):
+            try:
+                total += self._get_model_used_vram_gb(key)
+            except Exception:
+                pass
+        return total
+
+    def sweep_orphan_vram(self, context: str = "") -> None:
+        """Reclaim any UNREFERENCED orphaned GPU memory and surface a diagnostic for
+        the referenced kind. gc.collect() makes a just-dropped-but-not-collected
+        pipeline collectable; empty_cache() returns it to the driver. A referenced
+        leak can't be freed here (it must be fixed at its source) but is announced so
+        a mystery OOM is traceable. Cheap; safe to call before every load."""
+        import gc as _gc
+        for _ in range(2):
+            _gc.collect()
+        try:
+            import torch as _t
+            if not _t.cuda.is_available():
+                return
+            _t.cuda.synchronize()
+            _t.cuda.empty_cache()
+        except Exception:
+            return
+        try:
+            from codai.models.gpu_query import visible_gpu_memory
+            gpus = visible_gpu_memory() or []
+            if not gpus:
+                return
+            used_gb = sum(d["total"] - d["free"] for d in gpus) / 1e9
+            tracked_gb = self._estimate_tracked_vram_gb()
+            # ~1 GB driver/context baseline + slack before we call it a leak.
+            orphan_gb = used_gb - tracked_gb - 1.0
+            if orphan_gb > 2.0:
+                print(f"  [vram]{(' ' + context) if context else ''} {used_gb:.1f} GB "
+                      f"in use but tracked models account for only {tracked_gb:.1f} GB "
+                      f"— ~{orphan_gb:.1f} GB untracked (teardown leak; not freeable "
+                      f"here — referenced elsewhere)")
+        except Exception:
+            pass
+
     def _get_free_vram_gb(self) -> float:
         """Return estimated free VRAM in GB across the cards this engine can use.