models page: stop paying for the scan on every open (v0.2.44)

The page was slow to open on a large configuration. Measuring first, because the
obvious suspect was wrong: the disk scans are not the problem — scan_cache_dir
over 59 repos is 0.42s and the GGUF walk 0.53s, and _scan_caches() is 0.08s once
warm. The server-side stale-while-revalidate cache was already doing its job.

What it was, in order of cost:

THE CACHE DIED WITH THE PROCESS. It starts empty in every new one, and _cached()
recomputes synchronously when it has nothing — so the first read after a restart
pays the whole scan while someone sits looking at a spinner. Measured on the real
configuration: 3.69s cold, 0.001s once the previous result is read back from
disk. It is now persisted (atomically, under <config>/cache/) and seeded on first
use. The stored SIGNATURE is deliberately not restored: it is a list of mtimes,
and trusting it across a restart would serve a possibly hours-old list without
revalidating. Leaving it unset means the stored result is returned instantly AND
a refresh starts behind it. A missing, truncated or future-version file just
falls back to scanning.

THE PAGE WAITED ON ITSELF. First paint ran refreshLoadedStatus, then
refreshQuantJobs, then the list — three round trips in a chain — while
_loadEngineInfo did two more in another chain and then rendered the list a SECOND
time. Inside the list load, /admin/api/cached-models and /admin/api/models also
went out one after the other. Now: the two status reads go out together, the two
list reads go out together, and concurrent loadCachedModels() calls collapse into
one in-flight load, so first paint does one fetch and one render instead of three.
A call made after it settles still refetches, which every post-action refresh
relies on.

IT PAINTED NOTHING UNTIL ALL OF THAT FINISHED. The last response is kept in
sessionStorage and painted before any network call, then revalidated in place. It
is a listing of what is on disk, so seconds of staleness cost nothing — and a
quota error or private mode just means the page behaves as it does today.

One thing the tests had to be corrected on: the first version asserted the
background rescan had not run, which failed because _cached() starts the refresh
thread and then reads state["data"] — with a trivial scan function the thread
wins the race. That race is harmless (fresher data), and it was the assertion
that was wrong: the property worth holding is that the CALLER is not blocked, so
the test now uses a slow scan and asserts the read returns in under 0.2s with the
stored value, and that the revalidation really did start.

112 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.2.43"
+__version__ = "0.2.44"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -1757,8 +1757,72 @@ def _do_delete_model(model_id: str, cache_type: str) -> dict:
 # A long backstop TTL re-scans occasionally even if mtimes somehow miss a change.
 import threading as _cache_thr
 _SCAN_TTL = 600.0
-_scan_state = {"data": None, "sig": None, "at": 0.0, "busy": False}
-_stats_state = {"data": None, "sig": None, "at": 0.0, "busy": False}
+_scan_state = {"data": None, "sig": None, "at": 0.0, "busy": False, "disk": "model-scan.json"}
+_stats_state = {"data": None, "sig": None, "at": 0.0, "busy": False, "disk": "model-stats.json"}
+
+# The in-process cache above is lost on every restart, so the FIRST read of the
+# models page after one pays the whole scan synchronously — which is exactly when
+# someone is sitting there waiting for the page. Persisting the last result means
+# that read is served from disk and revalidated in the background instead.
+# The file is a cache: a corrupt or absent one simply falls back to scanning.
+_DISK_CACHE_VERSION = 1
+
+
+def _disk_cache_path(state) -> str:
+    import os
+    name = state.get("disk")
+    if not name or config_manager is None:
+        return ""
+    base = getattr(config_manager, "config_dir", "") or ""
+    if not base:
+        return ""
+    return os.path.join(str(base), "cache", name)
+
+
+def _load_disk_cache(state) -> None:
+    """Seed the in-memory cache from disk, once per process."""
+    import json as _json
+    if state.get("disk_loaded"):
+        return
+    state["disk_loaded"] = True
+    path = _disk_cache_path(state)
+    if not path:
+        return
+    try:
+        with open(path, "r", encoding="utf-8") as fh:
+            blob = _json.load(fh)
+    except (OSError, ValueError):
+        return
+    if not isinstance(blob, dict) or blob.get("version") != _DISK_CACHE_VERSION:
+        return
+    if blob.get("data") is None:
+        return
+    state["data"] = blob["data"]
+    # The signature is deliberately NOT restored: it is a list of mtimes, and
+    # trusting it across a restart would serve a stale list without revalidating.
+    # Leaving it unset means the first read returns this instantly AND refreshes.
+    state["sig"] = None
+    state["at"] = 0.0
+
+
+def _save_disk_cache(state, data) -> None:
+    import json as _json, os, tempfile
+    path = _disk_cache_path(state)
+    if not path or data is None:
+        return
+    try:
+        os.makedirs(os.path.dirname(path), exist_ok=True)
+        # Written atomically: a half-written cache read by another process would
+        # just be discarded, but a rename keeps it from happening at all.
+        fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path), suffix=".tmp")
+        with os.fdopen(fd, "w", encoding="utf-8") as fh:
+            _json.dump({"version": _DISK_CACHE_VERSION, "data": data}, fh)
+        os.replace(tmp, path)
+    except (OSError, TypeError, ValueError):
+        try:
+            os.unlink(tmp)
+        except (OSError, NameError, UnboundLocalError):
+            pass
 
 
 def _scan_signature():
@@ -1802,6 +1866,7 @@ def _scan_signature():
 
 def _cached(state, fn):
     import time as _t
+    _load_disk_cache(state)
     now = _t.time()
     sig = _scan_signature()
     fresh = (state["sig"] == sig and (now - state["at"]) < _SCAN_TTL)
@@ -1816,6 +1881,7 @@ def _cached(state, fn):
         state["sig"] = sig
         state["at"] = _t.time()
         state["force_sync"] = False
+        _save_disk_cache(state, state["data"])
         return state["data"]
     if not state["busy"]:                          # changed: refresh in background
         state["busy"] = True
@@ -1826,6 +1892,7 @@ def _cached(state, fn):
                 state["data"] = d
                 state["sig"] = _scan_signature()
                 state["at"] = _t.time()
+                _save_disk_cache(state, d)
             finally:
                 state["busy"] = False
         _cache_thr.Thread(target=_bg, daemon=True).start()

codai/admin/templates/models.html

@@ -3034,7 +3034,62 @@ function resetWhisperServerBuilderDefaults(){
   if(serverPathInput && !serverPathInput.value.trim()) serverPathInput.value = defaultWhisperServerPath();
 }
 
-async function loadCachedModels(){
+// ── Model-list snapshot cache ────────────────────────────────────────────────
+// Opening this page costs two round trips and a full re-render before anything
+// appears, which on a large configuration is the whole reason it feels slow. The
+// last successful response is kept in sessionStorage, so a revisit paints from it
+// immediately and then revalidates in the background. The data is a listing of
+// what is on disk — a few seconds of staleness costs nothing, and the fresh
+// render replaces it in place.
+const _SNAP_KEY = 'coderai.models.snapshot.v1';
+const _SNAP_MAX_AGE = 5 * 60 * 1000;
+let _snapshotUsed = false;
+
+function _readSnapshot(){
+  try{
+    const raw = sessionStorage.getItem(_SNAP_KEY);
+    if(!raw) return null;
+    const snap = JSON.parse(raw);
+    if(!snap || !snap.at || (Date.now() - snap.at) > _SNAP_MAX_AGE) return null;
+    return snap;
+  }catch(e){ return null; }   // private mode / disabled storage: just skip it
+}
+
+function _writeSnapshot(d, whisperModels){
+  try{
+    sessionStorage.setItem(_SNAP_KEY, JSON.stringify({at: Date.now(), d, whisperModels}));
+  }catch(e){ /* quota exceeded — the page works without the snapshot */ }
+}
+
+async function _fetchModelData(){
+  // Both reads are served by the system worker and neither depends on the other,
+  // so they go out together instead of one after the other.
+  const [cr, models] = await Promise.all([
+    pageFetch('/admin/api/cached-models'),
+    pageFetch('/admin/api/models').then(r => r.ok ? r.json() : []).catch(() => []),
+  ]);
+  if(!cr.ok) throw new Error((await cr.json().catch(()=>({}))).detail || cr.statusText);
+  const d = await cr.json();
+  // Only whisper-server RUNNERS (they carry a model_path) belong in this card.
+  // A whisper MODEL config (a .gguf entry with backend=whisper-server but no
+  // model_path) is shown on the GGUF row instead, as that file's model config.
+  const whisperModels = (models || []).filter(m => m.backend === 'whisper-server' && m.model_path);
+  _writeSnapshot(d, whisperModels);
+  return {d, whisperModels};
+}
+
+// First paint calls this from three places (the snapshot paint, the engine-info
+// callback and refreshLocal). Collapsing the concurrent ones into a single
+// in-flight load means one fetch and one render instead of three; a call made
+// after it settles still refetches, which is what every post-action refresh needs.
+let _listPending = null;
+function loadCachedModels(){
+  if(_listPending) return _listPending;
+  _listPending = _loadCachedModelsImpl().finally(() => { _listPending = null; });
+  return _listPending;
+}
+
+async function _loadCachedModelsImpl(){
   _localModels = [];
   const hfEl   = document.getElementById('hf-models-list');
   const ggufEl = document.getElementById('gguf-models-list');
@@ -3045,14 +3100,20 @@ async function loadCachedModels(){
   const _firstLoad = !hfEl.dataset.loaded;
   if(_firstLoad) hfEl.innerHTML = ggufEl.innerHTML = '<span class="muted small">Loading…</span>';
   try{
-    const r = await pageFetch('/admin/api/cached-models');
-    if(!r.ok) throw new Error((await r.json()).detail||r.statusText);
-    const d = await r.json();
-    // Only whisper-server RUNNERS (they carry a model_path) belong in this card.
-    // A whisper MODEL config (a .gguf entry with backend=whisper-server but no
-    // model_path) is shown on the GGUF row instead, as that file's model config.
-    const whisperModels = (await pageFetch('/admin/api/models').then(r=>r.ok?r.json():[]))
-      .filter(m => m.backend === 'whisper-server' && m.model_path);
+    let _data = null;
+    if(!_snapshotUsed){
+      _snapshotUsed = true;
+      const snap = _readSnapshot();
+      if(snap){
+        // Paint what we had, then revalidate on the next tick (by then this call
+        // has settled, so the wrapper lets the refetch through).
+        _data = {d: snap.d, whisperModels: snap.whisperModels};
+        setTimeout(() => loadCachedModels(), 0);
+      }
+    }
+    if(!_data) _data = await _fetchModelData();
+    const d = _data.d;
+    const whisperModels = _data.whisperModels || [];
 
     // HF models
     const hf = d.hf||[];
@@ -3361,8 +3422,10 @@ function _quantBadge(modelId){
 }
 
 async function refreshLocal(){
-  await refreshLoadedStatus();
-  _quantSig = await refreshQuantJobs();
+  // The list renders loaded/quant badges, so it still waits for both — but they
+  // are independent of each other and no longer go out in a chain.
+  const [, sig] = await Promise.all([refreshLoadedStatus(), refreshQuantJobs()]);
+  _quantSig = sig;
   loadCacheStats();
   loadCachedModels();
 }
@@ -3382,8 +3445,11 @@ setInterval(async () => {
 // batch them through the engine (which was actually dragging the page back onto
 // the busy engine), so each call goes straight to its correct target.
 loadGlobalSettings();
-// Engine/card info first so per-model card tags render on first paint, then
-// re-render once it resolves (covers the fetch resolving after the list).
+// Paint the list from the last snapshot before anything goes over the network;
+// every call below revalidates it, and concurrent ones collapse into one fetch.
+loadCachedModels();
+// Engine/card info so per-model card tags render, re-rendering once it resolves
+// (covers the fetch resolving after the list).
 _loadEngineInfo().then(() => loadCachedModels());
 refreshLocal();
 pollDownloads();

tests/test_model_scan_cache.py

+"""The models page's scan cache, and the part of it that survives a restart.
+
+The in-process cache was already stale-while-revalidate, but it started empty in
+every new process — so the first read after a restart recomputed the whole scan
+synchronously, which is exactly when someone is waiting for the page. Measured on
+a real configuration: 3.69s cold, 0.001s once the result is read back from disk.
+"""
+import json
+import types
+
+import pytest
+
+from codai.admin import routes
+
+
+@pytest.fixture
+def state(tmp_path, monkeypatch):
+    monkeypatch.setattr(routes, "config_manager",
+                        types.SimpleNamespace(config_dir=tmp_path, models_path=None))
+    return {"data": None, "sig": None, "at": 0.0, "busy": False, "disk": "model-scan.json"}
+
+
+def test_a_result_is_written_where_the_next_process_will_find_it(state, tmp_path):
+    routes._save_disk_cache(state, {"hf": [{"id": "a/b"}], "gguf": []})
+    path = tmp_path / "cache" / "model-scan.json"
+    assert path.is_file()
+    blob = json.loads(path.read_text())
+    assert blob["version"] == routes._DISK_CACHE_VERSION
+    assert blob["data"]["hf"][0]["id"] == "a/b"
+
+
+def test_a_fresh_process_is_seeded_from_disk(state):
+    routes._save_disk_cache(state, {"hf": [{"id": "a/b"}], "gguf": []})
+    fresh = {"data": None, "sig": None, "at": 0.0, "busy": False, "disk": "model-scan.json"}
+    routes._load_disk_cache(fresh)
+    assert fresh["data"]["hf"][0]["id"] == "a/b"
+
+
+def test_the_restored_entry_is_never_treated_as_verified(state):
+    """The signature is a list of mtimes. Restoring it would serve a list that
+    might be hours out of date WITHOUT revalidating — so it is deliberately left
+    unset, which makes the first read return instantly and refresh."""
+    state["sig"] = ("something",)
+    routes._save_disk_cache(state, {"hf": [], "gguf": []})
+    fresh = {"data": None, "sig": ("stale",), "at": 999.0, "busy": False,
+             "disk": "model-scan.json"}
+    routes._load_disk_cache(fresh)
+    assert fresh["sig"] is None
+    assert fresh["at"] == 0.0
+
+
+def test_disk_is_read_once_per_process(state, tmp_path):
+    routes._save_disk_cache(state, {"hf": [{"id": "first"}], "gguf": []})
+    fresh = {"data": None, "sig": None, "at": 0.0, "busy": False, "disk": "model-scan.json"}
+    routes._load_disk_cache(fresh)
+    # A later write must not be picked up by a second seed call: the in-memory
+    # cache is authoritative once the process is running.
+    routes._save_disk_cache(state, {"hf": [{"id": "second"}], "gguf": []})
+    routes._load_disk_cache(fresh)
+    assert fresh["data"]["hf"][0]["id"] == "first"
+
+
+@pytest.mark.parametrize("content", [
+    "not json at all",
+    '{"version": 999, "data": {"hf": []}}',   # a version this build does not know
+    '{"version": 1}',                          # no data
+    '[]',                                      # not an object
+])
+def test_an_unusable_cache_file_falls_back_to_scanning(state, tmp_path, content):
+    path = tmp_path / "cache" / "model-scan.json"
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(content)
+    fresh = {"data": None, "sig": None, "at": 0.0, "busy": False, "disk": "model-scan.json"}
+    routes._load_disk_cache(fresh)
+    assert fresh["data"] is None, "a bad cache must not become the answer"
+
+
+def test_nothing_is_written_without_a_config_dir(monkeypatch):
+    monkeypatch.setattr(routes, "config_manager", None)
+    s = {"data": None, "disk": "model-scan.json"}
+    assert routes._disk_cache_path(s) == ""
+    routes._save_disk_cache(s, {"hf": []})      # must not raise
+
+
+def test_a_state_with_no_disk_name_is_memory_only(state):
+    s = dict(state)
+    s["disk"] = None
+    assert routes._disk_cache_path(s) == ""
+
+
+def test_the_cached_reader_does_not_block_on_a_rescan(state, monkeypatch):
+    """The point of persisting: a read after a restart returns the stored result
+    straight away and revalidates behind it, instead of making the caller wait for
+    the scan. Measured on a real configuration: 3.69s before, 0.001s after."""
+    import time
+    routes._save_disk_cache(state, {"hf": [{"id": "from-disk"}], "gguf": []})
+    fresh = {"data": None, "sig": None, "at": 0.0, "busy": False, "disk": "model-scan.json"}
+    started = []
+
+    def _slow_scan():
+        started.append(time.time())
+        time.sleep(0.6)
+        return {"hf": [{"id": "rescanned"}], "gguf": []}
+
+    monkeypatch.setattr(routes, "_scan_signature", lambda: ("sig",))
+    t0 = time.time()
+    got = routes._cached(fresh, _slow_scan)
+    elapsed = time.time() - t0
+    assert elapsed < 0.2, f"the caller waited {elapsed:.2f}s for the scan"
+    assert got["hf"][0]["id"] == "from-disk"
+    # …and the rescan really was kicked off, so the list does not stay stale.
+    assert started, "the background revalidation should have started"
+    for _ in range(40):
+        if fresh["data"]["hf"][0]["id"] == "rescanned":
+            break
+        time.sleep(0.05)
+    assert fresh["data"]["hf"][0]["id"] == "rescanned"
+
+
+def test_without_a_stored_result_the_first_read_still_scans(state, monkeypatch):
+    """No cache file (a genuinely first run) must not serve an empty list."""
+    monkeypatch.setattr(routes, "_scan_signature", lambda: ("sig",))
+    fresh = {"data": None, "sig": None, "at": 0.0, "busy": False, "disk": "model-scan.json"}
+    got = routes._cached(fresh, lambda: {"hf": [{"id": "scanned"}], "gguf": []})
+    assert got["hf"][0]["id"] == "scanned"