free disk: find the weights engines keep outside the caches (v0.2.43)

"Free disk" on the colibri GLM-5.2 model returned 200 OK three times and freed
4 KB. The 400 GB was in /AI/offloads/colibri/models--mastouri--…, and the HF cache
held that model as a SYMLINK into it — so _do_delete_model deleted the link's
entry, found nothing else it knew about, and truthfully reported success.

Two separate failures, fixed separately.

IT DID NOT SAY HOW MUCH. Freeing 4 KB and freeing 400 GB were the same
{"success": true}. Every branch now measures what it removed and returns
freed_bytes/freed_human; the UI prints it, and says so plainly when the answer is
zero ("Nothing was on disk for this model — freed 0 B. Its files may live
somewhere coderai has not been told about."). This is the part that would have
made the problem visible on the first click instead of the third.

IT ONLY LOOKED IN TWO PLACES. codai/models/weight_stores.py adds the rest:

- The configured offload directory AND its siblings, one level. Engines are
  pointed at sibling roots by convention — coderai offloads to …/offloads/nvidia
  and colibri's weights landed in …/offloads/colibri. The scan widens where we
  LOOK; it never widens what we delete, which stays an exact models--<repo> name
  match. Nothing is ever removed by prefix or pattern.
- register_weight_root() for an engine that does not follow that convention.
  Deliberately not called from colibri_worker/ds4_worker: at import time
  config_manager is not loaded yet, so the offload directory is '' and the
  registration would be dead code. The sibling scan runs at DELETE time, off
  config, and covers both.

Measuring the size honestly took two passes:

- symlinks are not followed or counted (an HF snapshot is a tree of links into
  blobs/, so following them doubles the number);
- hardlinks are counted once by (device, inode), since a cache that links rather
  than copies would otherwise report the weights once per link.

And purging a store leaves the cache entry that pointed INTO it dangling, which
still looks like a present model to anything that only stats the name — so
dangling links to a purged store are pruned too.

Verified against the real model before touching it: a dry run found exactly one
path, 399.8 GB, and nothing else among the nine roots it searched. The delete
then freed it — /dev/nvme0n1p4 went from ~96% to 1% used, 444 GB free.

(Worth recording: I first read df on /AI and reported the space had NOT come back,
before noticing /AI/offloads is its own mount. Same wrong-filesystem mistake as
earlier in this session. The deletion had worked the whole time.)

100 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.2.42"
+__version__ = "0.2.43"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -1663,10 +1663,37 @@ def _do_clear_cache(cache_type: str) -> dict:
 
 
 def _do_delete_model(model_id: str, cache_type: str) -> dict:
+    """Delete a model's files and report how many BYTES that freed.
+
+    The size matters as much as the success flag. colibri showed why: its weights
+    lived in an engine-owned directory while the HF cache held a 4 KB shell of the
+    same repo, so this deleted the shell, returned {"success": True}, and left
+    400 GB on disk — a true answer to a question nobody was asking. Every branch
+    now measures what it removed, and engine-owned stores are swept too.
+    """
     import os, shutil
     from codai.models.cache import get_all_cache_dirs, get_model_cache_dir
+    from codai.models import weight_stores
     caches = get_all_cache_dirs()
 
+    _cfg = getattr(config_manager, "config", None) if config_manager is not None else None
+
+    def _finish(result: dict, counted: int = 0) -> dict:
+        """Add the engine-owned stores to whatever the cache branch freed."""
+        extra, paths = weight_stores.purge_external_weights(model_id, _cfg)
+        total = int(counted) + int(extra)
+        result = dict(result)
+        result["freed_bytes"] = total
+        result["freed_human"] = weight_stores.human_bytes(total)
+        if paths:
+            result["external_paths"] = paths
+        # Anything removed at all is a success, even if the cache branch found
+        # nothing: the files are gone, which is what Free disk is for.
+        if paths and not result.get("success"):
+            result["success"] = True
+            result.pop("detail", None)
+        return result
+
     if cache_type == "hf":
         hf_dir = caches.get("huggingface")
         if hf_dir:
@@ -1684,8 +1711,9 @@ def _do_delete_model(model_id: str, cache_type: str) -> dict:
                 repo = next((r for r in info.repos if r.repo_id == model_id), None)
                 if repo:
                     hashes = [r.commit_hash for r in repo.revisions]
+                    freed = int(getattr(repo, "size_on_disk", 0) or 0)
                     info.delete_revisions(*hashes).execute()
-                    return {"success": True}
+                    return _finish({"success": True}, freed)
             except Exception:
                 pass
             finally:
@@ -1693,12 +1721,14 @@ def _do_delete_model(model_id: str, cache_type: str) -> dict:
             # Fallback: remove the repo dir directly if it's still there.
             safe = model_id.replace("/", "--")
             d = os.path.join(hf_dir, f"models--{safe}")
+            freed = 0
             if os.path.exists(d):
+                freed = weight_stores.dir_size(d)
                 shutil.rmtree(d, ignore_errors=True)
             # Whether or not anything was on disk, the files are gone now — Free
             # disk is idempotent, so report success instead of a scary error.
-            return {"success": True}
-        return {"success": False, "detail": "HF cache directory not configured"}
+            return _finish({"success": True}, freed)
+        return _finish({"success": False, "detail": "HF cache directory not configured"})
 
     if cache_type == "gguf":
         gguf_dir = get_model_cache_dir()
@@ -1708,11 +1738,14 @@ def _do_delete_model(model_id: str, cache_type: str) -> dict:
         else:
             fp = os.path.join(gguf_dir, model_id)
         if os.path.isfile(fp):
+            freed = os.path.getsize(fp)
             os.remove(fp)
-            return {"success": True}
-        return {"success": False, "detail": "File not found"}
+            return _finish({"success": True}, freed)
+        # A GGUF entry can still have engine-owned weights elsewhere (a repo that
+        # was fetched as a directory), so the sweep runs before we call it missing.
+        return _finish({"success": False, "detail": "File not found"})
 
-    return {"success": False, "detail": "Unknown cache_type"}
+    return _finish({"success": False, "detail": "Unknown cache_type"})
 
 
 # Cache scans walk the whole HF/GGUF cache on disk (huggingface_hub.scan_cache_dir

codai/admin/templates/models.html

@@ -3505,7 +3505,20 @@ async function freeDiskConfirm(idx){
       body: JSON.stringify({path:m.path, cache_type:m.cacheType, source_repo:repo})
     });
     const d = await r.json();
-    if(d.success) refreshLocal();
+    if(d.success){
+      // Say how much came back. Freeing 4 KB and freeing 400 GB used to look
+      // identical here, which is how an engine-owned 400 GB store survived three
+      // "successful" deletes.
+      const n = d.external_paths ? d.external_paths.length : 0;
+      const where = n ? ' (incl. ' + n + ' engine store' + (n > 1 ? 's' : '') + ')' : '';
+      if(!d.freed_bytes){
+        showAlert('Nothing was on disk for this model \u2014 freed 0 B. Its files may '
+                + 'live somewhere coderai has not been told about.');
+      } else {
+        showAlert('Freed ' + (d.freed_human || '0 B') + where + '.');
+      }
+      refreshLocal();
+    }
     else showAlert('Error: '+(d.detail||d.error||'Unknown'));
   }catch(e){showAlert('Error: '+e.message)}
 }

codai/models/weight_stores.py

+"""Where model weights live OUTSIDE the HF and GGUF caches.
+
+An engine that fetches its own weights materialises them wherever it was pointed,
+which is not necessarily a cache coderai scans. colibri is the case that exposed
+this: its GLM-5.2 container sat in ``/AI/offloads/colibri/models--mastouri--…`` at
+400 GB, while the HF cache held a 4 KB shell of the same repo. "Free disk" deleted
+the shell, reported success, and freed nothing — truthfully, and uselessly.
+
+Two rules follow from that, and both are implemented here rather than in the
+endpoint, so every caller gets them:
+
+  * a delete reports the BYTES it freed, so "success" can never again mean 4 KB;
+  * the roots searched are extensible, so an engine that keeps weights of its own
+    is covered by declaring them rather than by someone remembering to patch the
+    delete path.
+
+Nothing here deletes by pattern or prefix. A directory is only ever removed when
+its name is exactly the HF cache spelling of the model being deleted, which is
+what keeps a one-level sibling scan safe.
+"""
+import os
+import shutil
+from typing import Iterable, List, Optional, Tuple
+
+# Roots declared by engines at import time: (path, label).
+_REGISTERED: List[Tuple[str, str]] = []
+
+
+def register_weight_root(path: str, label: str = "") -> None:
+    """Declare a directory where an engine keeps model weights.
+
+    Engines call this at import. Safe to call repeatedly and with a path that does
+    not exist yet — a root is only used if it is a directory when a delete runs.
+    """
+    path = os.path.abspath(os.path.expanduser((path or "").strip()))
+    if not path:
+        return
+    if not any(p == path for p, _ in _REGISTERED):
+        _REGISTERED.append((path, label or os.path.basename(path)))
+
+
+def registered_roots() -> List[Tuple[str, str]]:
+    return list(_REGISTERED)
+
+
+def hf_dir_name(model_id: str) -> str:
+    """The directory name huggingface_hub gives a repo: ``models--org--name``."""
+    return "models--" + (model_id or "").strip().replace("/", "--")
+
+
+def candidate_roots(config=None) -> List[Tuple[str, str]]:
+    """Directories that may hold weights for any model, as (path, label).
+
+    Beyond what engines register:
+
+      * the configured disk-offload directory, and
+      * its SIBLINGS, one level only.
+
+    The sibling scan exists because engines are pointed at sibling roots by
+    convention — coderai's own offload is ``…/offloads/nvidia`` and colibri's
+    weights landed in ``…/offloads/colibri``. It widens where we LOOK; it does not
+    widen what we delete, which stays an exact ``models--…`` name match.
+    """
+    roots: List[Tuple[str, str]] = []
+    seen = set()
+
+    def _add(path, label):
+        if not path:
+            return
+        path = os.path.abspath(os.path.expanduser(str(path).strip()))
+        if path and path not in seen and os.path.isdir(path):
+            seen.add(path)
+            roots.append((path, label))
+
+    for path, label in _REGISTERED:
+        _add(path, label)
+
+    offload = ""
+    try:
+        if config is not None:
+            offload = (getattr(getattr(config, "offload", None), "directory", "") or "").strip()
+    except Exception:
+        offload = ""
+    if offload:
+        offload = os.path.abspath(os.path.expanduser(offload))
+        _add(offload, "offload")
+        parent = os.path.dirname(offload)
+        if os.path.isdir(parent):
+            try:
+                for entry in sorted(os.listdir(parent)):
+                    _add(os.path.join(parent, entry), f"offload sibling: {entry}")
+            except OSError:
+                pass
+    return roots
+
+
+def _iter_repo_dirs(root: str, wanted: str) -> Iterable[str]:
+    """Directories under `root` whose name is exactly `wanted`, at depth 0 or 1.
+
+    Depth 1 covers an HF cache root given as ``<x>`` when the repos actually sit in
+    ``<x>/hub``, which is how huggingface_hub lays them out.
+    """
+    direct = os.path.join(root, wanted)
+    if os.path.isdir(direct):
+        yield direct
+    try:
+        children = sorted(os.listdir(root))
+    except OSError:
+        return
+    for child in children:
+        nested = os.path.join(root, child, wanted)
+        if os.path.isdir(nested):
+            yield nested
+
+
+def dir_size(path: str) -> int:
+    """Bytes a directory occupies, counting each piece of data once.
+
+    Two kinds of double-counting to avoid, both of which would overstate how much
+    a delete is about to free:
+
+      * symlinks are neither followed nor counted — an HF snapshot is a tree of
+        links into ``blobs/``;
+      * hardlinks are counted once, by (device, inode), since a cache that links
+        rather than copies would otherwise report the weights once per link.
+    """
+    total = 0
+    seen_inodes = set()
+    for dirpath, dirnames, filenames in os.walk(path, onerror=lambda e: None):
+        dirnames[:] = [d for d in dirnames
+                       if not os.path.islink(os.path.join(dirpath, d))]
+        for name in filenames:
+            full = os.path.join(dirpath, name)
+            try:
+                st = os.lstat(full)
+            except OSError:
+                continue
+            if os.path.islink(full):
+                continue
+            if st.st_nlink > 1:
+                key = (st.st_dev, st.st_ino)
+                if key in seen_inodes:
+                    continue
+                seen_inodes.add(key)
+            total += st.st_size
+    return total
+
+
+def prune_dangling_links(model_id: str, config=None) -> List[str]:
+    """Remove cache entries that are symlinks to a store that no longer exists.
+
+    The HF cache can hold the model as a SYMLINK into an engine's own directory
+    rather than as a copy — which is how coderai's cache showed 4 KB for a 400 GB
+    model. Once the store is purged that link dangles, and a dangling entry makes
+    the model look present to anything that only stats the name.
+    """
+    wanted = hf_dir_name(model_id)
+    if not wanted or wanted == "models--":
+        return []
+    removed = []
+    roots = [p for p, _ in candidate_roots(config)]
+    try:
+        from codai.models.cache import get_all_cache_dirs
+        roots.extend(d for d in get_all_cache_dirs().values() if d)
+    except Exception:
+        pass
+    checked = set()
+    for root in roots:
+        for candidate in (os.path.join(root, wanted),
+                          os.path.join(root, "hub", wanted)):
+            if candidate in checked:
+                continue
+            checked.add(candidate)
+            if os.path.islink(candidate) and not os.path.exists(candidate):
+                try:
+                    os.unlink(candidate)
+                    removed.append(candidate)
+                except OSError:
+                    pass
+    return removed
+
+
+def find_external_weights(model_id: str, config=None) -> List[Tuple[str, int]]:
+    """Every outside-the-cache directory holding `model_id`, as (path, bytes)."""
+    wanted = hf_dir_name(model_id)
+    if not wanted or wanted == "models--":
+        return []
+    found: List[Tuple[str, int]] = []
+    seen = set()
+    for root, _label in candidate_roots(config):
+        for path in _iter_repo_dirs(root, wanted):
+            real = os.path.realpath(path)
+            if real in seen:
+                continue
+            seen.add(real)
+            found.append((path, dir_size(path)))
+    return found
+
+
+def purge_external_weights(model_id: str, config=None,
+                           dry_run: bool = False) -> Tuple[int, List[str]]:
+    """Delete those directories. Returns (bytes freed, paths removed)."""
+    freed = 0
+    removed: List[str] = []
+    for path, size in find_external_weights(model_id, config):
+        if dry_run:
+            freed += size
+            removed.append(path)
+            continue
+        try:
+            shutil.rmtree(path)
+        except OSError:
+            continue
+        freed += size
+        removed.append(path)
+    if not dry_run:
+        # A cache entry pointing INTO the store we just removed now dangles.
+        removed.extend(prune_dangling_links(model_id, config))
+    return freed, removed
+
+
+def human_bytes(n: Optional[int]) -> str:
+    """Sizes as the admin UI shows them."""
+    try:
+        n = float(n or 0)
+    except (TypeError, ValueError):
+        return "0 B"
+    for unit in ("B", "KB", "MB", "GB", "TB"):
+        if abs(n) < 1024.0 or unit == "TB":
+            return f"{n:.0f} {unit}" if unit == "B" else f"{n:.1f} {unit}"
+        n /= 1024.0
+    return f"{n:.1f} TB"

tests/test_weight_stores.py

+"""Finding model weights that live outside the HF and GGUF caches.
+
+The case this exists for: colibri's GLM-5.2 container sat in
+/AI/offloads/colibri/models--mastouri--… at 400 GB while the HF cache held a 4 KB
+shell of the same repo. "Free disk" deleted the shell and reported success.
+"""
+import os
+import types
+
+import pytest
+
+from codai.models import weight_stores as WS
+
+
+def _cfg(directory):
+    return types.SimpleNamespace(offload=types.SimpleNamespace(directory=str(directory)))
+
+
+def _repo(root, model_id, blob_bytes=4096):
+    """An HF-cache-layout repo dir: a blob plus a snapshot symlink to it."""
+    d = root / WS.hf_dir_name(model_id)
+    (d / "blobs").mkdir(parents=True)
+    (d / "snapshots" / "abc").mkdir(parents=True)
+    blob = d / "blobs" / "deadbeef"
+    blob.write_bytes(b"\x00" * blob_bytes)
+    os.symlink(blob, d / "snapshots" / "abc" / "model.safetensors")
+    return d
+
+
+def test_the_hf_directory_name_matches_huggingface_hub():
+    assert WS.hf_dir_name("mastouri/GLM-5.2-colibri") == "models--mastouri--GLM-5.2-colibri"
+
+
+def test_a_sibling_of_the_offload_root_is_searched(tmp_path):
+    """The exact shape that was missed: coderai offloads to .../offloads/nvidia and
+    the engine put its weights in .../offloads/colibri."""
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    colibri = tmp_path / "offloads" / "colibri"
+    colibri.mkdir()
+    _repo(colibri, "mastouri/GLM-5.2", blob_bytes=5000)
+    found = WS.find_external_weights("mastouri/GLM-5.2",
+                                     _cfg(tmp_path / "offloads" / "nvidia"))
+    assert len(found) == 1
+    assert found[0][1] == 5000
+
+
+def test_a_symlinked_snapshot_is_not_counted_twice(tmp_path):
+    """An HF snapshot is a tree of links into blobs/; following them would report
+    the weights twice and make 'freed' a lie in the generous direction."""
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    store = tmp_path / "offloads" / "engine"
+    store.mkdir()
+    d = _repo(store, "org/model", blob_bytes=10000)
+    assert WS.dir_size(str(d)) == 10000
+
+
+def test_only_an_exact_name_match_is_ever_returned(tmp_path):
+    """The sibling scan widens where we look, never what we delete."""
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    store = tmp_path / "offloads" / "engine"
+    store.mkdir()
+    _repo(store, "org/model")
+    _repo(store, "org/model-v2")
+    (store / "some-unrelated-dir").mkdir()
+    found = WS.find_external_weights("org/model", _cfg(tmp_path / "offloads" / "nvidia"))
+    assert [os.path.basename(p) for p, _ in found] == ["models--org--model"]
+
+
+def test_a_repo_one_level_down_is_found(tmp_path):
+    """huggingface_hub puts repos under <root>/hub, so a root given without it
+    still has to resolve."""
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    hub = tmp_path / "offloads" / "engine" / "hub"
+    hub.mkdir(parents=True)
+    _repo(hub, "org/model")
+    found = WS.find_external_weights("org/model", _cfg(tmp_path / "offloads" / "nvidia"))
+    assert len(found) == 1
+
+
+def test_the_same_directory_is_not_reported_twice(tmp_path):
+    """Registered roots and the sibling scan can name the same place."""
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    store = tmp_path / "offloads" / "engine"
+    store.mkdir()
+    _repo(store, "org/model")
+    WS.register_weight_root(str(store), "engine")
+    try:
+        found = WS.find_external_weights("org/model", _cfg(tmp_path / "offloads" / "nvidia"))
+        assert len(found) == 1
+    finally:
+        WS._REGISTERED.clear()
+
+
+def test_purge_removes_the_files_and_reports_the_bytes(tmp_path):
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    store = tmp_path / "offloads" / "engine"
+    store.mkdir()
+    d = _repo(store, "org/model", blob_bytes=7777)
+    freed, paths = WS.purge_external_weights("org/model",
+                                             _cfg(tmp_path / "offloads" / "nvidia"))
+    assert freed == 7777
+    assert paths == [str(d)] or paths[0].endswith("models--org--model")
+    assert not d.exists()
+
+
+def test_a_dry_run_measures_without_deleting(tmp_path):
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    store = tmp_path / "offloads" / "engine"
+    store.mkdir()
+    d = _repo(store, "org/model", blob_bytes=1234)
+    freed, paths = WS.purge_external_weights("org/model",
+                                             _cfg(tmp_path / "offloads" / "nvidia"),
+                                             dry_run=True)
+    assert freed == 1234 and len(paths) == 1
+    assert d.exists()
+
+
+def test_nothing_anywhere_is_not_an_error(tmp_path):
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    freed, paths = WS.purge_external_weights("org/absent",
+                                             _cfg(tmp_path / "offloads" / "nvidia"))
+    assert freed == 0 and paths == []
+
+
+def test_an_empty_model_id_matches_nothing(tmp_path):
+    """Guards the degenerate 'models--' name against matching a directory."""
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    (tmp_path / "offloads" / "engine").mkdir()
+    (tmp_path / "offloads" / "engine" / "models--").mkdir()
+    assert WS.find_external_weights("", _cfg(tmp_path / "offloads" / "nvidia")) == []
+
+
+@pytest.mark.parametrize("n,expected", [
+    (0, "0 B"), (512, "512 B"), (4096, "4.0 KB"),
+    (400 * 1024 ** 3, "400.0 GB"), (None, "0 B"),
+])
+def test_sizes_read_the_way_the_admin_ui_shows_them(n, expected):
+    assert WS.human_bytes(n) == expected
+
+
+def test_hardlinked_data_is_counted_once(tmp_path):
+    """A cache that hardlinks rather than copies would otherwise have the weights
+    counted once per link, overstating what a delete frees."""
+    d = tmp_path / "repo"
+    d.mkdir()
+    real = d / "shard.safetensors"
+    real.write_bytes(b"\x00" * 9000)
+    os.link(real, d / "also-shard.safetensors")
+    assert WS.dir_size(str(d)) == 9000
+
+
+def test_a_cache_symlink_into_a_purged_store_is_removed(tmp_path):
+    """The exact leftover: the HF cache held the model as a SYMLINK into the
+    engine's directory, so purging the store left a dangling entry that still
+    looks like a present model to anything that only stats the name."""
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    store = tmp_path / "offloads" / "colibri"
+    store.mkdir()
+    real = _repo(store, "org/model", blob_bytes=2048)
+    hub = tmp_path / "offloads" / "hfcache" / "hub"
+    hub.mkdir(parents=True)
+    link = hub / WS.hf_dir_name("org/model")
+    os.symlink(real, link)
+
+    cfg = _cfg(tmp_path / "offloads" / "nvidia")
+    # The link and the store are the same data — counted once, not twice.
+    freed, _ = WS.purge_external_weights("org/model", cfg, dry_run=True)
+    assert freed == 2048
+
+    freed, paths = WS.purge_external_weights("org/model", cfg)
+    assert freed == 2048
+    assert not real.exists()
+    assert not os.path.islink(link), "the dangling cache entry should be gone"
+    assert str(link) in paths
+
+
+def test_a_live_cache_symlink_is_left_alone(tmp_path):
+    """Only a link whose target is GONE is pruned."""
+    (tmp_path / "offloads" / "nvidia").mkdir(parents=True)
+    store = tmp_path / "offloads" / "engine"
+    store.mkdir()
+    real = _repo(store, "org/keeper")
+    hub = tmp_path / "offloads" / "hfcache" / "hub"
+    hub.mkdir(parents=True)
+    link = hub / WS.hf_dir_name("org/keeper")
+    os.symlink(real, link)
+    assert WS.prune_dangling_links("org/other", _cfg(tmp_path / "offloads" / "nvidia")) == []
+    assert os.path.islink(link)