front: route GGUF pinned to torch engine -> gguf sibling; sync cached-models refresh

Two fixes after the GGUF process-isolation split:

1. Routing: a model with engine=nvidia in its config (a hard pin) but a GGUF body
   503'd, because the torch nvidia engine dropped the `gguf` capability. pick_engine
   now falls back to the co-located "<name>-gguf" sibling (same card) when the pinned
   engine can't serve the required capability (and symmetrically for a gguf-pinned
   model needing transformers). The pin's intent — that physical card — is honoured.

2. Config-save visibility: the cached-models scan is stale-while-revalidate, so the
   first read after an invalidation returned stale and refreshed in the background —
   a saved config didn't appear until a second refresh. _invalidate_cache_scan now
   sets force_sync so the next read recomputes synchronously (fresh immediately). The
   system worker invalidates on reload-config, so a save shows on the next page load.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
parent e715cc49
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API # Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here. # metadata and the admin web UI read from here.
__version__ = "0.1.12" __version__ = "0.1.13"
# Configure the CUDA caching allocator BEFORE torch is imported anywhere. # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
# expandable_segments lets the allocator return freed pages to the driver even # expandable_segments lets the allocator return freed pages to the driver even
......
...@@ -1562,10 +1562,15 @@ def _cached(state, fn): ...@@ -1562,10 +1562,15 @@ def _cached(state, fn):
fresh = (state["sig"] == sig and (now - state["at"]) < _SCAN_TTL) fresh = (state["sig"] == sig and (now - state["at"]) < _SCAN_TTL)
if state["data"] is not None and fresh: if state["data"] is not None and fresh:
return state["data"] # nothing changed → instant return state["data"] # nothing changed → instant
if state["data"] is None: if state["data"] is None or state.get("force_sync"):
state["data"] = fn() # first call: compute synchronously # First call, or an EXPLICIT invalidation (e.g. a model-config save pushed a
# reload here): recompute synchronously so the very next read is fresh, not
# stale-while-revalidate. Otherwise a save wouldn't show until a second
# refresh — the reported "saved config doesn't appear in the frontend" bug.
state["data"] = fn()
state["sig"] = sig state["sig"] = sig
state["at"] = _t.time() state["at"] = _t.time()
state["force_sync"] = False
return state["data"] return state["data"]
if not state["busy"]: # changed: refresh in background if not state["busy"]: # changed: refresh in background
state["busy"] = True state["busy"] = True
...@@ -1584,9 +1589,13 @@ def _cached(state, fn): ...@@ -1584,9 +1589,13 @@ def _cached(state, fn):
def _invalidate_cache_scan(): def _invalidate_cache_scan():
"""Force the next cached-models/cache-stats read to refresh (after a model is """Force the next cached-models/cache-stats read to refresh (after a model is
deleted/downloaded/freed).""" deleted/downloaded/freed, or a config save). force_sync makes that next read
recompute synchronously so the change is visible immediately, not on the read
after."""
_scan_state["sig"] = None _scan_state["sig"] = None
_scan_state["force_sync"] = True
_stats_state["sig"] = None _stats_state["sig"] = None
_stats_state["force_sync"] = True
@router.get("/admin/api/cached-models", summary="List cached models") @router.get("/admin/api/cached-models", summary="List cached models")
......
...@@ -128,6 +128,15 @@ def pick_engine(registry: EngineRegistry, path: str, method: str, ...@@ -128,6 +128,15 @@ def pick_engine(registry: EngineRegistry, path: str, method: str,
e = registry.by_name(pinned) e = registry.by_name(pinned)
if e is not None and e.can_serve(cap) and e.is_alive(): if e is not None and e.can_serve(cap) and e.is_alive():
return e return e
# Co-located GGUF-isolation sibling: the split runs a "<name>-gguf"
# engine on the SAME card. A model pinned to the torch engine but needing
# `gguf` (or pinned to the gguf engine but needing transformers) should
# transparently use whichever sibling can serve it — they share the GPU,
# so the pin's intent (that physical card) is still honoured. Not a 503.
sib_name = (pinned[:-5] if pinned.endswith("-gguf") else f"{pinned}-gguf")
sib = registry.by_name(sib_name)
if sib is not None and sib.can_serve(cap) and sib.is_alive():
return sib
_warn_bad_pin(model, pinned, cap, e, fallback=pin_fallback) _warn_bad_pin(model, pinned, cap, e, fallback=pin_fallback)
# Default: a hard pin fails rather than running elsewhere. When the # Default: a hard pin fails rather than running elsewhere. When the
# model opts in (engine_fallback), fall through to pick another engine. # model opts in (engine_fallback), fall through to pick another engine.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment