vram: measure multi-GPU load delta across all devices (fix split models stuck at 0)

record_vram_delta() measured free VRAM via _free_vram_snapshot(), which only
read the default CUDA device (device 0). A gpu_split model (e.g. gemma on two
cards) lands most weights on the other card, so the device-0 delta came out
~0, tripped the 'delta <= 0' guard, and never persisted measured_vram_gb. The
estimate stayed 0 and every load OOM-retried from scratch even with
force_vram_update set.

Sum free VRAM across every visible CUDA device (scoped per engine via
CUDA_VISIBLE_DEVICES), plus amdgpu sysfs when cross-pooling is on or no CUDA
device is visible -- mirroring _get_free_vram_gb(). Before/after snapshots are
now symmetric, so the delta captures the full multi-card footprint and the
real value is persisted; subsequent loads size correctly and stop OOM-looping.

Bump version to 0.1.3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.1.2"
+__version__ = "0.1.3"
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
 # expandable_segments lets the allocator return freed pages to the driver even

codai/models/manager.py

@@ -2292,15 +2292,48 @@ class MultiModelManager:
 
     @staticmethod
     def _free_vram_snapshot() -> float:
-        """Return current free VRAM in bytes (CUDA) or -1 if unavailable."""
+        """Return current free VRAM in BYTES summed across every device this engine
+        can use, or -1 if nothing is measurable.
+
+        MUST be multi-device: a model loaded with gpu_split spans several cards (and,
+        under cross-backend pooling, a Radeon too). Measuring only the default CUDA
+        device (device 0) means a split that lands most weights on the OTHER card
+        yields a ~0 delta in record_vram_delta(), which trips its `delta <= 0` guard
+        and never persists measured_vram_gb — so the estimate stays 0 and every load
+        OOM-retries from scratch. Sum all CUDA devices (torch honours
+        CUDA_VISIBLE_DEVICES, so this is scoped to this engine's cards) and add
+        amdgpu sysfs when cross-pooling is on or no CUDA device is visible. CUDA and
+        amdgpu sysfs never name the same card, so there's no double count."""
+        total_free = 0.0
+        cuda_count = 0
         try:
             import torch
             if torch.cuda.is_available():
-                free, _ = torch.cuda.mem_get_info()
-                return float(free)
+                for i in range(torch.cuda.device_count()):
+                    try:
+                        free, _ = torch.cuda.mem_get_info(i)
+                        total_free += float(free)
+                        cuda_count += 1
+                    except Exception:
+                        pass
         except Exception:
             pass
-        return -1.0
+        got = cuda_count > 0
+        cross = MultiModelManager._cross_gpu_pooling_enabled()
+        if cross or cuda_count == 0:
+            try:
+                import glob
+                for total_path in sorted(glob.glob('/sys/class/drm/card*/device/mem_info_vram_total')):
+                    used_path = total_path.replace('vram_total', 'vram_used')
+                    with open(total_path) as f:
+                        total = int(f.read().strip())
+                    with open(used_path) as f:
+                        used = int(f.read().strip())
+                    total_free += float(total - used)
+                    got = True
+            except Exception:
+                pass
+        return total_free if got else -1.0
 
     def vram_before_load(self) -> float:
         """Call immediately before loading a model; returns a snapshot for delta measurement."""