manager: clean cross-engine VRAM swap (evict a busy sibling at its unit boundary)

On the GGUF-isolation split, a torch (video/image) engine and a gguf (text)
engine share one NVIDIA card. When one needed VRAM it asked the co-located
sibling to release via /internal/evict-vram, but that only evicted the
sibling's IDLE models and SKIPPED busy ones — so a text-model load would
proceed into the VRAM an in-flight video clip still needed for its forward,
and BOTH OOM'd. Recovery then laddered the video load down to disk offload
and thrashed for ~1h.

Give the cross-engine path the same wait-then-evict the local eviction
already has: release_idle_vram(needed_gb, wait_for_busy, wait_timeout) first
evicts idle models, then — only if still short — WAITS for each busy model to
reach a safe idle point (between requests, e.g. between video clip parts) and
evicts it. This converts contention into a CLEAN SWAP: the render's current
unit finishes, its model is evicted, the sibling loads alone, and the render
reloads + resumes on its next unit. Bounded by wait_timeout (180s) so two
mutually-waiting busy engines can't deadlock — one gives up and falls back to
its own CPU/disk offload.

/internal/evict-vram now reads needed_gb + wait + wait_timeout from the body
and forwards them; _cosite_vram_releaser sends wait=True with an HTTP timeout
that exceeds the sibling's wait budget so the swap isn't cut short. Symmetric:
both engines register the releaser at each other, so either direction swaps
cleanly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

codai/api/app.py

@@ -365,12 +365,29 @@ async def internal_engine_state():
 async def internal_evict_vram(request: Request):
     """A co-located sibling engine (same GPU) asks this engine to release VRAM it
     can't evict itself — the GGUF-isolation split runs two engines on one NVIDIA
-    card. Evicts all idle (non-busy) models and reports GB freed. Runs in a thread
-    so eviction's blocking CUDA frees don't stall the event loop."""
-    import asyncio
+    card. Evicts all idle (non-busy) models; when the caller passes needed_gb + wait
+    and idle eviction isn't enough, it WAITS for a busy model (e.g. an in-flight
+    video clip) to finish its current unit and evicts it too — a clean cross-engine
+    swap instead of the sibling loading into the render's VRAM and both OOMing. Runs
+    in a thread so eviction's blocking CUDA frees don't stall the event loop."""
+    import asyncio, json as _json
+    try:
+        data = _json.loads((await request.body()) or b"{}") or {}
+    except Exception:
+        data = {}
+    try:
+        needed_gb = float(data.get("needed_gb") or 0.0)
+    except (TypeError, ValueError):
+        needed_gb = 0.0
+    wait = bool(data.get("wait", True))
+    try:
+        wait_timeout = float(data.get("wait_timeout") or 180.0)
+    except (TypeError, ValueError):
+        wait_timeout = 180.0
     try:
         from codai.models.manager import multi_model_manager
-        freed = await asyncio.to_thread(multi_model_manager.release_idle_vram)
+        freed = await asyncio.to_thread(
+            multi_model_manager.release_idle_vram, needed_gb, wait, wait_timeout)
         return {"ok": True, "freed_gb": float(freed)}
     except Exception as e:
         return {"ok": False, "error": str(e), "freed_gb": 0.0}

codai/main.py

@@ -758,14 +758,23 @@ def main():
         _itok = os.environ.get("CODERAI_INTERNAL_TOKEN")
 
         def _cosite_vram_releaser(needed_gb: float) -> float:
+            # Ask each co-located sibling to free `needed_gb`, WAITING for a busy
+            # model (e.g. an in-flight video clip) to finish its current unit and be
+            # evicted — a clean cross-engine swap rather than loading into the
+            # render's VRAM and both OOMing. The HTTP timeout must exceed the
+            # sibling's wait budget (a video clip part can run ~2.5 min) so the wait
+            # isn't cut short; on true timeout the sibling gives up and we fall back
+            # to our own offload.
             import httpx as _httpx
+            _wait_budget = 180.0
             total = 0.0
             for _u in _cosited:
                 try:
                     _r = _httpx.post(f"{_u}/internal/evict-vram",
-                                     json={"needed_gb": needed_gb},
+                                     json={"needed_gb": needed_gb, "wait": True,
+                                           "wait_timeout": _wait_budget},
                                      headers={"x-coderai-internal": _itok or ""},
-                                     timeout=180.0)
+                                     timeout=_wait_budget + 30.0)
                     if _r.status_code == 200:
                         total += float((_r.json() or {}).get("freed_gb") or 0.0)
                 except Exception as _e:

codai/models/manager.py

@@ -3752,16 +3752,34 @@ class MultiModelManager:
                   f"Remaining models are busy or VRAM is held elsewhere — the new "
                   f"model will load with CPU/disk offload.")
 
-    def release_idle_vram(self) -> float:
-        """Evict every loaded model that isn't actively serving a request, to free
-        VRAM for a CO-LOCATED sibling engine sharing this GPU (the GGUF-isolation
-        split runs a torch engine and a gguf engine on one NVIDIA card; neither can
-        evict the other's models, so the one needing room asks the other to release
-        via /internal/evict-vram). Returns GB freed. Busy models are left alone."""
+    def release_idle_vram(self, needed_gb: float = 0.0,
+                          wait_for_busy: bool = False,
+                          wait_timeout: float = 180.0) -> float:
+        """Evict loaded models to free VRAM for a CO-LOCATED sibling engine sharing
+        this GPU (the GGUF-isolation split runs a torch engine and a gguf engine on
+        one NVIDIA card; neither can evict the other's models, so the one needing
+        room asks the other to release via /internal/evict-vram). Returns GB freed.
+
+        Pass 1 evicts every IDLE model immediately. Pass 2 — only when the caller
+        passes `needed_gb` and `wait_for_busy` and Pass 1 didn't free enough — WAITS
+        for each still-busy model to reach a safe idle point (between requests, e.g.
+        between video clip parts) and then evicts it too. This is the cross-engine
+        analogue of the local `_wait_until_idle` eviction: it turns a would-be
+        CONTENTION (the sibling loading into VRAM an active render still needs → both
+        OOM) into a CLEAN SWAP (the render's current unit finishes, its model is
+        evicted, the sibling loads alone, and the render reloads + resumes on its
+        next unit). Bounded by `wait_timeout` so two mutually-waiting busy engines
+        can't deadlock — one gives up and falls back to its own offload."""
         before = self._get_free_vram_gb()
+
+        def _freed_enough() -> bool:
+            return needed_gb > 0 and (self._get_free_vram_gb() - before) >= needed_gb
+
+        # Pass 1: evict everything idle right now (cheap, non-blocking).
+        _busy = []
         for key in list(self._lru_order()):
             if self._is_key_busy(key):
-                print(f"  [cosite-evict] '{key}' is busy — not releasing")
+                _busy.append(key)
                 continue
             try:
                 print(f"  [cosite-evict] releasing '{key}' for a co-located engine")
@@ -3770,6 +3788,37 @@ class MultiModelManager:
                     self.active_in_vram = None
             except Exception as e:
                 print(f"  [cosite-evict] failed to release '{key}': {e}")
+
+        # Pass 2: for a sibling that will otherwise OOM, wait for busy models to go
+        # idle at their next request boundary, then evict — a clean cross-engine swap.
+        if wait_for_busy and _busy and not _freed_enough():
+            import time as _time
+            _deadline = _time.time() + max(0.0, wait_timeout)
+            for key in _busy:
+                if _freed_enough():
+                    break
+                if key not in self.models and key not in self.model_pools:
+                    continue  # already gone
+                _remaining = _deadline - _time.time()
+                if _remaining <= 0:
+                    print(f"  [cosite-evict] wait budget exhausted — leaving '{key}' "
+                          f"busy (sibling will fall back to its own offload)")
+                    break
+                print(f"  [cosite-evict] '{key}' busy — waiting up to {_remaining:.0f}s "
+                      f"for it to finish its current unit, then releasing (clean swap)…")
+                if self._wait_until_idle(key, timeout=_remaining):
+                    try:
+                        print(f"  [cosite-evict] releasing now-idle '{key}' for a "
+                              f"co-located engine")
+                        self._evict_one(key)
+                        if key == self.active_in_vram:
+                            self.active_in_vram = None
+                    except Exception as e:
+                        print(f"  [cosite-evict] failed to release '{key}': {e}")
+                else:
+                    print(f"  [cosite-evict] '{key}' still busy after wait — leaving it "
+                          f"loaded")
+
         after = self._get_free_vram_gb()
         return max(0.0, after - before)