engines: cross-engine VRAM eviction for co-located GGUF/torch engines

After the GGUF-isolation split, a torch engine and a gguf engine share one NVIDIA
card but each can only evict its OWN models. So loading an image model on the torch
engine while the gguf engine still holds a resident GGUF text model failed: local
eviction "freed 0.0 GB ... VRAM held elsewhere" → CUDA OOM.

Fix — co-located VRAM release:
- front passes each engine its same-GPU siblings' internal URLs via
  CODERAI_COSITED_URLS (matched by identical CODERAI_ENGINE_GPUS selectors).
- engine registers an external_vram_releaser that POSTs to each sibling's new
  /internal/evict-vram when local eviction can't free enough.
- /internal/evict-vram → manager.release_idle_vram(): evicts all idle (non-busy)
  models and returns GB freed; busy/actively-serving models are left alone.

Symmetric: the gguf engine can likewise reclaim VRAM from the torch engine's idle
diffusers models. Driver-level free VRAM (cross-process) is re-checked after each
releaser, so the loader proceeds once the sibling has freed enough.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.1.13"
+__version__ = "0.1.14"
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
 # expandable_segments lets the allocator return freed pages to the driver even

codai/api/app.py

@@ -343,6 +343,21 @@ async def internal_engine_state():
             "vram": vram, "tasks": tasks, "cooling": cooling, "paused": paused}
 
 
+@app.post("/internal/evict-vram", include_in_schema=False)
+async def internal_evict_vram(request: Request):
+    """A co-located sibling engine (same GPU) asks this engine to release VRAM it
+    can't evict itself — the GGUF-isolation split runs two engines on one NVIDIA
+    card. Evicts all idle (non-busy) models and reports GB freed. Runs in a thread
+    so eviction's blocking CUDA frees don't stall the event loop."""
+    import asyncio
+    try:
+        from codai.models.manager import multi_model_manager
+        freed = await asyncio.to_thread(multi_model_manager.release_idle_vram)
+        return {"ok": True, "freed_gb": float(freed)}
+    except Exception as e:
+        return {"ok": False, "error": str(e), "freed_gb": 0.0}
+
+
 @app.post("/internal/thermal-pause", include_in_schema=False)
 async def internal_thermal_pause(request: Request):
     """Front-driven thermal pause: the supervisor (which monitors temps centrally)

codai/frontproxy/engine_supervisor.py

@@ -179,6 +179,26 @@ class EngineSupervisor:
         return p
 
     # ----------------------------------------------------------------- planning
+    def _set_cosited_urls(self, engines) -> None:
+        """Tell each engine the internal URLs of co-located engines (same GPU), via
+        CODERAI_COSITED_URLS, so it can ask them to release VRAM when its own
+        eviction can't free enough on a shared card (the GGUF-isolation split puts
+        two engines — torch + gguf — on one NVIDIA card). Co-location is matched by
+        identical CODERAI_ENGINE_GPUS selectors."""
+        def _sel(e):
+            return (e.env.get("CODERAI_ENGINE_GPUS") or "").strip()
+        for e in engines:
+            if getattr(e, "role", "engine") == "system":
+                continue
+            mine = _sel(e)
+            if not mine:
+                continue
+            sibs = [f"http://127.0.0.1:{o.port}" for o in engines
+                    if o is not e and getattr(o, "role", "engine") != "system"
+                    and _sel(o) == mine]
+            if sibs:
+                e.env["CODERAI_COSITED_URLS"] = ",".join(sibs)
+
     def _build_engines(self) -> list:
         """Return the list of Engine objects to launch.
 
@@ -238,6 +258,7 @@ class EngineSupervisor:
                     name=spec.get("name") or f"engine#{idx}",
                     backend=backend, env=env, capabilities=caps,
                 ))
+            self._set_cosited_urls(engines)
             return engines
 
         # Auto: one engine per GPU vendor actually present on this machine. Vendors
@@ -290,6 +311,7 @@ class EngineSupervisor:
                     name=f"{name}-gguf", backend="nvidia", env=dict(env),
                     capabilities={"gguf"}))
                 next_id += 1
+        self._set_cosited_urls(engines)
         return engines
 
     # ------------------------------------------------------------------ spawning

codai/main.py

@@ -749,6 +749,35 @@ def main():
         except ValueError:
             pass
 
+    # Cross-engine VRAM release: on a shared GPU (the GGUF-isolation split puts a
+    # torch engine and a gguf engine on one NVIDIA card) neither can evict the
+    # other's models. When local eviction can't free enough, ask each co-located
+    # sibling (CODERAI_COSITED_URLS, set by the front) to release its idle models.
+    _cosited = [u for u in (os.environ.get("CODERAI_COSITED_URLS") or "").split(",") if u]
+    if _cosited:
+        _itok = os.environ.get("CODERAI_INTERNAL_TOKEN")
+
+        def _cosite_vram_releaser(needed_gb: float) -> float:
+            import httpx as _httpx
+            total = 0.0
+            for _u in _cosited:
+                try:
+                    _r = _httpx.post(f"{_u}/internal/evict-vram",
+                                     json={"needed_gb": needed_gb},
+                                     headers={"x-coderai-internal": _itok or ""},
+                                     timeout=180.0)
+                    if _r.status_code == 200:
+                        total += float((_r.json() or {}).get("freed_gb") or 0.0)
+                except Exception as _e:
+                    print(f"  [cosite-evict] sibling {_u} failed: {_e}", flush=True)
+            return total
+
+        try:
+            multi_model_manager.register_external_vram_releaser(_cosite_vram_releaser)
+            print(f"  Cross-engine VRAM release enabled (co-located: {', '.join(_cosited)})")
+        except Exception as _e:
+            print(f"  Cross-engine VRAM release setup failed: {_e}")
+
     print(f"\nLoad mode: {load_mode}")
     if load_mode == "ondemand":
         print("  (pre-load first model, unload/load on switch)")

codai/models/manager.py

@@ -3602,6 +3602,27 @@ class MultiModelManager:
                   f"Remaining models are busy or VRAM is held elsewhere — the new "
                   f"model will load with CPU/disk offload.")
 
+    def release_idle_vram(self) -> float:
+        """Evict every loaded model that isn't actively serving a request, to free
+        VRAM for a CO-LOCATED sibling engine sharing this GPU (the GGUF-isolation
+        split runs a torch engine and a gguf engine on one NVIDIA card; neither can
+        evict the other's models, so the one needing room asks the other to release
+        via /internal/evict-vram). Returns GB freed. Busy models are left alone."""
+        before = self._get_free_vram_gb()
+        for key in list(self._lru_order()):
+            if self._is_key_busy(key):
+                print(f"  [cosite-evict] '{key}' is busy — not releasing")
+                continue
+            try:
+                print(f"  [cosite-evict] releasing '{key}' for a co-located engine")
+                self._evict_one(key)
+                if key == self.active_in_vram:
+                    self.active_in_vram = None
+            except Exception as e:
+                print(f"  [cosite-evict] failed to release '{key}': {e}")
+        after = self._get_free_vram_gb()
+        return max(0.0, after - before)
+
     def ensure_vram_for(self, model_key: str, resolved_name: str = None,
                         extra_vram_gb: float = 0.0) -> None:
         """Make room in VRAM for a model about to load — ANY type, ANY load mode.

docs/gguf-process-isolation.md

@@ -90,11 +90,14 @@ No new mechanism: the gguf engine is a first-class engine, so the front already:
 - **Two processes per NVIDIA card.** Extra host RAM (the gguf engine imports the full
   stack) and a second, idle torch CUDA context (~0.3–0.6 GB VRAM). Acceptable for
   correctness; could be trimmed later by lazy-importing torch in gguf-only engines.
-- **No cross-engine eviction.** Each engine evicts only its own models, so on a single
-  shared card the gguf engine can't evict a resident diffusers model to reclaim VRAM
-  (and vice versa). llama.cpp's auto CPU-offload (`n_gpu_layers`) and the global RAM
-  cap keep this from hard-OOMing; tighter cross-engine VRAM coordination is a
-  follow-up.
+- **Cross-engine eviction (implemented).** Each engine evicts only its own models, so
+  on a shared card the torch engine can't directly evict a resident GGUF model (and
+  vice versa). Resolved via co-located VRAM release: the front passes each engine its
+  same-GPU siblings' internal URLs (`CODERAI_COSITED_URLS`); when an engine's local
+  eviction can't free enough, an `external_vram_releaser` POSTs to each sibling's
+  `/internal/evict-vram`, which evicts that engine's idle models and reports GB freed
+  (`manager.release_idle_vram`). Co-location is matched by identical
+  `CODERAI_ENGINE_GPUS` selectors. Busy (actively-serving) models are never released.
 - **`engine_specs` users opt out.** When `engine_specs` is set the auto-split is
   skipped — declare the torch/gguf split yourself (two specs on the same card: one
   `backend: nvidia` with `capabilities` minus `gguf`, one `backend: nvidia` /