lora-train: evict co-located sibling engine's VRAM before training (fix OOM)

LoRA training freed VRAM with unload_all_models(), which only unloads THIS
engine's models. On the GGUF-isolation split the co-located gguf (text) engine
kept its model resident (~7.4 GB), so fp32 training (~16 GB) + the sibling
exceeded the 24 GB card → "CUDA out of memory. Tried to allocate 32 MiB … 26 MiB
free … Process 226 has 7.36 GiB" — every fighter LoRA (dlaba, zigo, zlo, …)
failed. Training also isn't covered by the front swap-gate, so nothing else
cleared the sibling.

Add multi_model_manager.evict_cosited_siblings(): invoke the registered
cross-engine VRAM releasers (the cosite releaser posts wait=True, so it waits for
a busy sibling to reach a safe point). Call it right after unload_all_models() in
both training paths (image + video/Wan), so training gets the whole card.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

codai/api/loras.py

@@ -999,6 +999,11 @@ def _train_lora_sync(req: LoraTrainRequest) -> dict:
         try:
             from codai.models.manager import multi_model_manager
             multi_model_manager.unload_all_models()
+            # Also release any co-located sibling engine's model (shared-GPU split):
+            # training needs the whole card and the swap-gate doesn't cover it.
+            _freed = multi_model_manager.evict_cosited_siblings()
+            if _freed:
+                print(f"  [lora] freed {_freed:.1f} GB from co-located engine(s) for training")
         except Exception as e:
             print(f"  [lora] could not unload models before training: {e}")
         device = "cuda" if __import__("torch").cuda.is_available() else "cpu"
@@ -1041,6 +1046,13 @@ def _train_lora_sync(req: LoraTrainRequest) -> dict:
     try:
         from codai.models.manager import multi_model_manager
         multi_model_manager.unload_all_models()
+        # Also release any co-located sibling engine's model (shared-GPU split):
+        # unload_all_models() only frees THIS engine; a sibling's resident model
+        # (e.g. the gguf text engine) would otherwise share the card and OOM
+        # training. The swap-gate doesn't cover training, so evict explicitly.
+        _freed = multi_model_manager.evict_cosited_siblings()
+        if _freed:
+            print(f"  [lora] freed {_freed:.1f} GB from co-located engine(s) for training")
     except Exception as e:
         print(f"  [lora] could not unload models before training: {e}")
 

codai/models/manager.py

@@ -867,6 +867,27 @@ class MultiModelManager:
         if callable(fn) and fn not in self._external_vram_releasers:
             self._external_vram_releasers.append(fn)
 
+    def evict_cosited_siblings(self, needed_gb: float = 999.0) -> float:
+        """Ask co-located sibling engines (a shared GPU) to release their VRAM.
+
+        `unload_all_models()` only frees THIS engine's models; a sibling engine on
+        the same card (the GGUF-isolation split) keeps its own model resident and
+        can't be evicted locally. Before an EXCLUSIVE GPU operation that needs the
+        whole card — LoRA training especially, which the front swap-gate doesn't
+        cover — call this so the sibling's model is released too (otherwise training
+        + a resident text model exceed VRAM and OOM). `needed_gb` defaults high so
+        the sibling frees everything it can; it waits for a busy sibling to reach a
+        safe point (the cosite releaser posts wait=True). Returns GB siblings freed."""
+        total = 0.0
+        for _rel in list(self._external_vram_releasers):
+            try:
+                freed = _rel(needed_gb)
+                if freed:
+                    total += float(freed)
+            except Exception as e:
+                print(f"  Warning in external VRAM releaser: {e}")
+        return total
+
     @staticmethod
     def _is_cuda_context_fatal(err: Exception) -> bool:
         """True for CUDA errors that corrupt the whole process context."""