refuse a load that cannot fit in host RAM, and size the one that was skipping itself

A load is admitted to VRAM by estimate: ensure_vram_for sizes the model, evicts
what is in the way, and failing that a CUDA OOM is caught and reported. Host RAM
had no equivalent, and the kernel supplies its own: four engines SIGKILLed in
five hours on a 54 GB host, each one while a model staged its weights.

    Oct 29 23:03  Killed coderai-nvidia-gguf   anon-rss 46.3 GB
    Oct 30 03:10  Killed coderai-nvidia        anon-rss 47.7 GB
    Oct 30 03:36  Killed coderai-nvidia        anon-rss 31.6 GB
    Oct 30 03:49  Killed coderai-nvidia        anon-rss 32.7 GB
    front: engine#0 exited (code -9); respawning (1/5 in 120s)

The RAM watcher was awake for all of it and logged "RSS 50.6/43.0 GB (118%) --
mitigation [gc+trim+evict_idle] -> 53.7 GB": every rung of its ladder releases
memory already held, and none of them can decline the load doing the allocating.
SIGKILL then takes the engine rather than the request, so the client that asked
for the oversized model saw a dropped connection -- and so did every unrelated
request in flight, plus everything arriving during the 120-second respawn. One
bad load reads as the whole server being unreliable.

So offload.ram_admission (default on) adds the missing half. codai/models/
host_ram.py sizes a pending load -- measured host residency if the model has
loaded before, the whole footprint for an offload strategy that asks for it,
the spill past free VRAM otherwise, plus a staging allowance -- against the
tighter of MemAvailable and the headroom under max_ram_gb, less a floor kept
for the OS and the sibling engines. ensure_host_ram_for() runs the existing
pressure ladder FIRST and re-measures, so refusal is the last step; when it
comes, it is a 503 naming the model, the estimate, the budget, the shortfall
and what to change on the card. A model whose footprint cannot be determined
is never refused on a number nobody has.

And the proximate cause of the biggest of those kills was a sizing step that
reported itself as skipped:

    (auto-offload sizing skipped: cannot access local variable '_ck'
     where it is not associated with a value)

The GGUF auto-offload branch reads the configured KV-cache types to size its
headroom, from a binding made ~110 lines further down where llama_kwargs is
assembled. Every load that reached the branch raised UnboundLocalError, the
handler called it "skipped", and nothing reduced n_gpu_layers -- so llama.cpp
was asked to keep every layer of a 35B on a card that could not hold them, and
answered with a 43,562 MiB CPU KV buffer. The read is hoisted above its use
(tested on source order, not a line number), and the handler now says the load
is UNSIZED, because "skipped" invites the reader to assume nothing needed doing.

Separately, and found while checking which cards were really misconfigured:
a repo whose model_index.json declares a pipeline class diffusers does not have
is now detected and loaded with its own code when allow-listed, or refused by
name when not. The allow-list and the mechanism both already existed, but only
fired when a card named a custom_pipeline -- nothing ever did, so MOSS-Image
and Kromoss failed with "module diffusers has no attribute MOSSImagePipeline",
which names neither the cause nor the remedy and reads as a broken model.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.27"
+__version__ = "0.5.28"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -4032,6 +4032,8 @@ def build_settings_dict(c, gpu_cards):
             "ram_watch_poll_seconds": c.offload.ram_watch_poll_seconds,
             "ram_watch_soft_fraction": c.offload.ram_watch_soft_fraction,
             "ram_watch_cuda": c.offload.ram_watch_cuda,
+            "ram_admission": c.offload.ram_admission,
+            "ram_admission_floor_gb": c.offload.ram_admission_floor_gb,
             "gpu_split": c.offload.gpu_split,
             "tensor_split": c.offload.tensor_split,
             "split_strategy": c.offload.split_strategy,
@@ -4517,6 +4519,10 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
         if "ram_watch_soft_fraction" in off:
             c.offload.ram_watch_soft_fraction = float(off["ram_watch_soft_fraction"] or c.offload.ram_watch_soft_fraction)
         c.offload.ram_watch_cuda = bool(off.get("ram_watch_cuda", c.offload.ram_watch_cuda))
+        c.offload.ram_admission = bool(off.get("ram_admission", c.offload.ram_admission))
+        if "ram_admission_floor_gb" in off:
+            c.offload.ram_admission_floor_gb = float(
+                off["ram_admission_floor_gb"] or c.offload.ram_admission_floor_gb)
         c.offload.gpu_split = bool(off.get("gpu_split", c.offload.gpu_split))
         if "tensor_split" in off:
             c.offload.tensor_split = off["tensor_split"] or None
@@ -4552,6 +4558,8 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
                 ga.ram_watch_poll_seconds = c.offload.ram_watch_poll_seconds
                 ga.ram_watch_soft_fraction = c.offload.ram_watch_soft_fraction
                 ga.ram_watch_cuda = c.offload.ram_watch_cuda
+                ga.ram_admission = c.offload.ram_admission
+                ga.ram_admission_floor_gb = c.offload.ram_admission_floor_gb
                 ga.gpu_split = c.offload.gpu_split
                 ga.tensor_split = c.offload.tensor_split
                 ga.split_strategy = c.offload.split_strategy

codai/api/app.py

@@ -930,6 +930,36 @@ async def task_cancelled_handler(request: Request, exc: TaskCancelled):
     return JSONResponse(status_code=499, content={"detail": "Task cancelled", "task_id": tid})
 
 
+from codai.models.host_ram import HostRamExhausted
+
+
+@app.exception_handler(HostRamExhausted)
+async def host_ram_exhausted_handler(request: Request, exc: HostRamExhausted):
+    """A load was refused for want of host RAM.
+
+    503, because this is the server declining to start work it cannot finish,
+    and the condition is transient: the same request succeeds once a resident
+    model is evicted or a smaller build is asked for. Registered once here
+    rather than caught per route — the refusal can come from any path that
+    loads a model (chat, images, video, audio), and every one of them should
+    report the numbers instead of a bare 500.
+
+    Before this existed the request did not fail at all: the host ran out of
+    memory, the kernel killed the engine, and the client saw a dropped
+    connection while every other model's requests died with it.
+    """
+    return JSONResponse(
+        status_code=503,
+        content={"error": {"message": exc.message(),
+                           "type": "insufficient_host_memory",
+                           "param": exc.model,
+                           "code": "host_ram_exhausted",
+                           "needed_gb": exc.need.gb,
+                           "available_gb": exc.budget.gb,
+                           "shortfall_gb": exc.shortfall_gb}},
+    )
+
+
 @app.get("/v1/models", response_model=ModelList, summary="List available models", tags=["Core"])
 async def list_models():
     """List available models."""

codai/api/images.py

@@ -784,9 +784,21 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
         from codai.admin.routes import config_manager as _cm
         _img_remote = remote_code.kwargs_for(
             getattr(_cm, "config", None), model_name, _cfg('custom_pipeline'))
+        if not _img_remote:
+            # No card named a custom_pipeline — ask the repository itself. A
+            # checkpoint whose model_index.json declares a class diffusers does
+            # not have is telling us it ships its own, and letting that load
+            # proceed produced "module diffusers has no attribute
+            # MOSSImagePipeline": a true statement that names neither the cause
+            # nor the fix, and reads as a broken model.
+            _img_remote = remote_code.kwargs_for_declared_class(
+                getattr(_cm, "config", None), model_name)
         if _img_remote:
+            _declared = _img_remote.pop("_declared_class", None)
             print(f"  Custom pipeline from the repository: "
-                  f"{_img_remote['custom_pipeline']} (allow-listed)")
+                  f"{_img_remote['custom_pipeline']} (allow-listed)"
+                  + (f" — it declares {_declared}, which is not in diffusers"
+                     if _declared else ""))
     except PermissionError:
         raise
     except Exception as e:
@@ -907,6 +919,28 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
         except Exception as _ee:
             print(f"  Proactive eviction skipped: {_ee}")
 
+    # =====================================================================
+    # Host-RAM admission
+    # =====================================================================
+    # VRAM is handled above by eviction; host RAM has to be handled by refusal,
+    # because the kernel's answer to running out of it is to kill the engine. The
+    # image path is where it bit hardest: the cards that quantize only their text
+    # encoder stage a bf16 transformer into host RAM, and on a full card nearly
+    # all of it spills. Raised, not caught — the caller turns it into a 503 that
+    # names the model, the size and the fix. See codai/models/host_ram.py.
+    try:
+        from codai.models.manager import multi_model_manager as _mmm
+    except Exception:
+        _mmm = None
+    if _mmm is not None:
+        _hkey = None
+        for _k in (f"image:{model_name}", model_name):
+            if _k in _mmm.config:
+                _hkey = _k
+                break
+        _mmm.ensure_host_ram_for(_hkey or f"image:{model_name}", model_name,
+                                 _requested_id)
+
     # =====================================================================
     # Standard loading path (with OOM fallback)
     # =====================================================================

codai/backends/vulkan.py

@@ -1240,6 +1240,19 @@ class VulkanBackend(ModelBackend):
             try:
                 _exp = kwargs.get('expected_vram_gb')
                 _nlayers = _gguf_block_count(model_path)
+                # Read the KV-cache types HERE, not from the binding made further
+                # down when llama_kwargs is assembled. That binding happens after
+                # this block, so reading it here raised UnboundLocalError on every
+                # load that reached the offload branch -- caught below and printed
+                # as "(auto-offload sizing skipped: cannot access local variable
+                # '_ck' …)". Nothing then reduced n_gpu_layers, so a 35B that did
+                # not fit kept every layer on the GPU and llama.cpp answered with a
+                # 43,562 MiB CPU KV buffer: the host ran out of memory and the
+                # kernel killed the engine (code -9), three times in five hours.
+                # A skipped sizing step must not read as a sized one.
+                _kv_raw_cfg = kwargs.get('_raw_cfg') or {}
+                _ck = kwargs.get('cache_type_k', _kv_raw_cfg.get('cache_type_k'))
+                _cv = kwargs.get('cache_type_v', _kv_raw_cfg.get('cache_type_v'))
                 # Usable VRAM is the POOL across the cards this engine can use, not
                 # just main_gpu: all same-backend cards always (e.g. both 3090s), plus
                 # the other backend's cards when cross-split is on (3090 + RX 580).
@@ -1316,7 +1329,17 @@ class VulkanBackend(ModelBackend):
                           f"GPU, {_nlayers - _fit} on CPU (slower). Lower n_ctx or use a "
                           f"smaller model to keep it fully on GPU.", flush=True)
             except Exception as _off_e:
-                print(f"  (auto-offload sizing skipped: {_off_e})", flush=True)
+                # Loud on purpose, and it says what the consequence is. This
+                # printed "(auto-offload sizing skipped: …)" for an
+                # UnboundLocalError in its own code, which reads like a decision
+                # not to offload rather than a failure to decide -- and the model
+                # that was not sized then took the host down with a 43 GB CPU KV
+                # buffer. A sizing step that did not run leaves the load UNSIZED,
+                # so say that, with the error.
+                print(f"  WARNING: auto-offload sizing FAILED ({type(_off_e).__name__}: "
+                      f"{_off_e}) — this load is UNSIZED: every layer stays on the "
+                      f"GPU and llama.cpp may put the KV cache in host RAM. Check "
+                      f"n_gpu_layers / n_ctx for this model.", flush=True)
 
         # Configure context size
         if no_ram:

codai/config.py

@@ -292,6 +292,14 @@ class OffloadConfig:
     ram_watch_poll_seconds: float = 15.0    # how often the watcher samples RSS
     ram_watch_soft_fraction: float = 0.90   # mitigate at/above this fraction of the cap
     ram_watch_cuda: bool = True             # allow mitigation to call CUDA empty_cache()
+    # Host-RAM ADMISSION: refuse a load whose estimated host-RAM cost does not
+    # fit, instead of letting it run the host out of memory. The watcher above
+    # can only release memory already held, so it cannot stop an unquantized
+    # card mid-load -- on a 54 GB host that cost four OOM-killed engines in five
+    # hours, each reported to clients as "crashes the engine". See
+    # codai/models/host_ram.py. Off means loads proceed as they did before.
+    ram_admission: bool = True
+    ram_admission_floor_gb: float = 4.0     # host RAM kept free for OS + engines
     # Unload a model that has gone untouched for this long, in seconds (0 = never).
     #
     # Eviction already frees a model when something else needs the memory, so this
@@ -2213,6 +2221,8 @@ class ConfigManager:
                 "ram_watch_poll_seconds": self.config.offload.ram_watch_poll_seconds,
                 "ram_watch_soft_fraction": self.config.offload.ram_watch_soft_fraction,
                 "ram_watch_cuda": self.config.offload.ram_watch_cuda,
+                "ram_admission": self.config.offload.ram_admission,
+                "ram_admission_floor_gb": self.config.offload.ram_admission_floor_gb,
                 "gpu_split": self.config.offload.gpu_split,
                 "tensor_split": self.config.offload.tensor_split,
                 "split_strategy": self.config.offload.split_strategy,

codai/main.py

@@ -1221,6 +1221,10 @@ def main():
     global_args.ram_watch_poll_seconds = config.offload.ram_watch_poll_seconds
     global_args.ram_watch_soft_fraction = config.offload.ram_watch_soft_fraction
     global_args.ram_watch_cuda = config.offload.ram_watch_cuda
+    # Host-RAM admission: refuse a load that cannot fit rather than let the
+    # kernel's OOM killer take the engine (codai/models/host_ram.py).
+    global_args.ram_admission = config.offload.ram_admission
+    global_args.ram_admission_floor_gb = config.offload.ram_admission_floor_gb
     # Cross-backend GPU pooling defaults (per-model config can override). Read by
     # the GGUF loader to set llama.cpp split_mode/tensor_split.
     global_args.gpu_split = getattr(config.offload, "gpu_split", False)

codai/models/manager.py

@@ -2219,6 +2219,10 @@ class MultiModelManager:
                         self.ensure_vram_for(model_name, requested_id=requested_id)
                     except Exception as _ev_e:
                         print(f"  (ensure_vram_for warning: {_ev_e})")
+                    # Outside that except on purpose: a VRAM reservation failing
+                    # is a warning (the load can still work), while a host-RAM
+                    # refusal is the decision itself and must reach the caller.
+                    self.ensure_host_ram_for(model_name, requested_id=requested_id)
                 _snap = self.vram_before_load()
                 _ram_snap = self._get_own_ram_gb()
                 # Tell the backend how much VRAM this model is expected to need so
@@ -6019,6 +6023,130 @@ class MultiModelManager:
                   f"have {free_gb:.1f} GB free — evicting models to make room")
             self._evict_models_for_vram(needed_gb)
 
+    @staticmethod
+    def _ram_admission_enabled() -> bool:
+        """Whether a load may be refused for want of host RAM (default True)."""
+        try:
+            from codai.api.state import get_global_args
+            ga = get_global_args()
+            return bool(getattr(ga, 'ram_admission', True)) if ga else True
+        except Exception:
+            return True
+
+    @staticmethod
+    def _ram_admission_floor_gb() -> float:
+        """Host RAM never promised to a load, in GB."""
+        from codai.models import host_ram
+        try:
+            from codai.api.state import get_global_args
+            ga = get_global_args()
+            val = getattr(ga, 'ram_admission_floor_gb', None) if ga else None
+            return float(val) if val is not None else host_ram.DEFAULT_FLOOR_GB
+        except Exception:
+            return host_ram.DEFAULT_FLOOR_GB
+
+    def _host_ram_budget(self):
+        """The host-RAM budget a load may take right now."""
+        from codai.models import host_ram, proc_mem
+        return host_ram.budget(
+            kernel_available_gb=proc_mem.host_available_gb(),
+            cap_gb=self._ram_cap_gb(),
+            fleet_held_gb=self._get_process_ram_gb(),
+            floor_gb=self._ram_admission_floor_gb(),
+        )
+
+    def ensure_host_ram_for(self, model_key: str, resolved_name: str = None,
+                            requested_id: str = None, cfg: dict = None) -> None:
+        """Refuse a load that cannot fit in host RAM, before it starts.
+
+        The counterpart of :meth:`ensure_vram_for`, and the reason it exists is
+        that it had none. VRAM pressure is answered by eviction and, failing
+        that, by a CUDA OOM the server catches and reports. Host-RAM pressure is
+        answered by the kernel's OOM killer, which takes the whole engine with
+        it: four times in five hours on a 54 GB host the night this was written,
+        each time while an unquantized card staged its weights, each time
+        reported to the client as "crashes the engine" (see
+        :mod:`codai.models.host_ram`).
+
+        So the ladder here ends differently. Make room first -- trim, then evict
+        idle models, exactly as RAM pressure is handled elsewhere -- and if the
+        load still does not fit, raise :class:`~codai.models.host_ram.HostRamExhausted`.
+        One model then fails with its size, the budget and the fix, while every
+        other model keeps serving.
+
+        Never refuses a model that is already resident (it is not about to
+        allocate anything), and never refuses on a footprint it could not
+        determine.
+        """
+        from codai.models import host_ram
+        if not self._ram_admission_enabled():
+            return
+        for _k in (model_key, resolved_name, requested_id):
+            if _k and _k in self.models:
+                return
+
+        if cfg is None:
+            cfg = {}
+            for _k in (model_key, resolved_name, requested_id):
+                if _k and _k in self.config:
+                    cfg = self.config.get(_k) or {}
+                    break
+
+        try:
+            _measured = cfg.get("measured_ram_gb")
+            _measured = float(_measured) if _measured else None
+        except (TypeError, ValueError):
+            _measured = None
+
+        need = host_ram.estimate_need(
+            total_need_gb=self._get_model_used_vram_gb(
+                model_key, resolved_name, requested_id),
+            free_vram_gb=self._get_free_vram_gb(),
+            measured_ram_gb=_measured,
+            offload_strategy=cfg.get("offload_strategy"),
+            no_ram=bool(cfg.get("no_ram")),
+        )
+        bud = self._host_ram_budget()
+        if host_ram.verdict(need, bud).ok:
+            return
+        if need.basis == "unknown":
+            # Nothing was ever measured or configured for this model, so the
+            # only figure on the table is the load buffer — a guess about every
+            # model rather than a fact about this one. Refusing on it would
+            # block a 300 MB embedder on a tight host for no reason. Say that
+            # the protection does not cover this load, and let it proceed.
+            print(f"Loading '{model_key}': host-RAM admission has no footprint "
+                  f"for this model, so it cannot be checked — proceeding "
+                  f"unguarded (budget {bud.gb:.1f} GB; {bud.detail})")
+            return
+
+        print(f"Loading '{model_key}': needs ~{need.gb:.1f} GB host RAM "
+              f"({need.detail}), budget is {bud.gb:.1f} GB ({bud.detail}) — "
+              f"freeing host RAM before deciding")
+        try:
+            _trim_cpu_ram()
+        except Exception:
+            pass
+        if self._evict_idle_on_ram_enabled():
+            _cap = self._ram_cap_gb()
+            if _cap:
+                self._evict_models_for_ram(_cap * 0.85)
+            else:
+                # No cap configured, so there is no target to evict towards --
+                # free what has gone quiet and re-measure against the kernel's
+                # figure. 30 s rather than "any idle model": a model being used
+                # in a request loop is idle between its own requests, and pulling
+                # it to admit another load just trades one reload for another.
+                self.unload_idle_models(30.0)
+
+        bud = self._host_ram_budget()
+        if host_ram.verdict(need, bud).ok:
+            print(f"  host RAM freed — {bud.gb:.1f} GB now available, load admitted")
+            return
+        raise host_ram.HostRamExhausted(
+            str(requested_id or resolved_name or model_key), need, bud,
+            host_ram.remedy_for(cfg, need.basis))
+
     def request_model(self, requested_model: str, model_type: str = None,
                       extra_vram_gb: float = 0.0) -> Dict[str, Any]:
         """
@@ -6343,6 +6471,7 @@ class MultiModelManager:
             # Not loaded — make room in VRAM (evict other models as needed).
             self.ensure_vram_for(model_key, resolved_name, extra_vram_gb,
                                  requested_id=_requested_id)
+            self.ensure_host_ram_for(model_key, resolved_name, _requested_id)
             return {
                 'model_key': model_key,
                 'model_name': resolved_name,
@@ -6368,6 +6497,7 @@ class MultiModelManager:
             # model).  Make room again so it returns to VRAM instead of CPU.
             self.ensure_vram_for(model_key, resolved_name, extra_vram_gb,
                                  requested_id=_requested_id)
+            self.ensure_host_ram_for(model_key, resolved_name, _requested_id)
             return {
                 'model_key': model_key,
                 'model_name': resolved_name,
@@ -6395,6 +6525,7 @@ class MultiModelManager:
             print(f"Loadall mode: Model '{model_key}' not pre-loaded, will load now")
             self.ensure_vram_for(model_key, resolved_name, extra_vram_gb,
                                  requested_id=_requested_id)
+            self.ensure_host_ram_for(model_key, resolved_name, _requested_id)
             return {
                 'model_key': model_key,
                 'model_name': resolved_name,

codai/models/remote_code.py

@@ -90,3 +90,65 @@ def kwargs_for(cfg, repo: str, custom_pipeline: Optional[str]) -> dict:
         return {}
     require(cfg, repo, f"its own pipeline module {name!r}")
     return {"custom_pipeline": name, "trust_remote_code": True}
+
+
+def declared_pipeline_class(repo: str) -> Optional[str]:
+    """The ``_class_name`` the repo's ``model_index.json`` declares, or None.
+
+    Read rather than guessed, because the name is the only thing that says a
+    repository means to supply its own pipeline: MOSS-Image declares
+    ``MOSSImagePipeline``, which exists in the repo and nowhere in diffusers.
+    """
+    import json
+    import os
+    try:
+        if os.path.isdir(repo):
+            path = os.path.join(repo, "model_index.json")
+            if not os.path.isfile(path):
+                return None
+        else:
+            from huggingface_hub import hf_hub_download
+            # Fetched, not read cache-only: a repo being loaded for the first
+            # time has no cache yet, and a cache-only answer to "what class does
+            # this declare" is permanently wrong on a freshly-rented pod. The
+            # file is a few hundred bytes and the weights are about to follow.
+            path = hf_hub_download(repo, "model_index.json")
+        with open(path) as fh:
+            name = (json.load(fh) or {}).get("_class_name")
+        return str(name) if name else None
+    except Exception:
+        # Not a diffusers repo, not reachable, or no index. Either way there is
+        # nothing to declare, and the ordinary load must be left alone.
+        return None
+
+
+def kwargs_for_declared_class(cfg, repo: str) -> dict:
+    """``from_pretrained`` kwargs for a repo whose pipeline class is its own.
+
+    The companion to :func:`kwargs_for`, which only fires when a card already
+    names a ``custom_pipeline``. Nothing filled that field in for MOSS-Image or
+    Kromoss, so the load went ahead as an ordinary repo load and diffusers
+    reported ``module diffusers has no attribute MOSSImagePipeline`` -- true,
+    and no help at all: it names neither the cause (the class lives in the
+    repository) nor the remedy (allow-list it). The operator reads it as a
+    broken model.
+
+    So detect the case from the repo's own declaration. Allow-listed, it loads
+    with its code; not allow-listed, it is refused by name with the setting to
+    change, which is this module's whole policy.
+
+    Empty dict when the declared class is available locally -- the overwhelming
+    majority -- so an ordinary load never pays for this.
+    """
+    cls = declared_pipeline_class(repo)
+    if not cls:
+        return {}
+    try:
+        import diffusers
+        if getattr(diffusers, cls, None) is not None:
+            return {}
+    except Exception:
+        return {}
+    require(cfg, repo, f"its own pipeline class {cls!r}")
+    return {"custom_pipeline": repo, "trust_remote_code": True,
+            "_declared_class": cls}

docs/memory-accounting.md

@@ -112,6 +112,84 @@ cache that the kernel reclaims by itself and that evicting a model cannot free.
 The cap can only be met by releasing memory something actually holds, so the
 metric has to be one that releasing memory moves.
 
+## A load that cannot fit in host RAM is refused
+
+The cap above is a *pressure* mechanism: a watcher samples the fleet's memory
+and, past `offload.ram_watch_soft_fraction` of the cap, runs a ladder — collect,
+trim, drop the upscaler cache, drop prefix caches, evict idle models. Every rung
+releases memory that is already held. None of them can decline a load that is at
+that moment streaming weights into the host.
+
+That gap has a cost, measured on a 54 GB host running four engines:
+
+```
+Oct 29 23:03  Killed coderai-nvidia-gguf   anon-rss 46.3 GB
+Oct 30 03:10  Killed coderai-nvidia        anon-rss 47.7 GB
+Oct 30 03:36  Killed coderai-nvidia        anon-rss 31.6 GB
+Oct 30 03:49  Killed coderai-nvidia        anon-rss 32.7 GB
+front: engine#0 exited (code -9); respawning (1/5 in 120s)
+```
+
+The watcher was awake throughout and logged
+`RSS 50.6/43.0 GB (118%) — mitigation [gc+trim+evict_idle] → 53.7 GB`: it ran
+its ladder and the number went up anyway. The kernel settled it, and SIGKILL
+takes the engine, not the request — so the client that asked for the oversized
+model saw a dropped connection, and so did every unrelated request in flight,
+plus everything that arrived during the 120-second respawn. One bad load is
+reported as the whole server being unreliable.
+
+So `offload.ram_admission` (default on) adds the missing half: before a load
+starts, estimate what it will cost the host and refuse it if it does not fit.
+`codai/models/host_ram.py` holds the arithmetic, `ensure_host_ram_for()` the
+decision, and it sits beside `ensure_vram_for()` on every load path.
+
+The estimate follows what actually determines the cost:
+
+| situation | host-RAM cost |
+| --- | --- |
+| the model loaded before | `measured_ram_gb` — what it really parked on the host |
+| `offload_strategy` `model` / `sequential` / `cpu` | the whole footprint; that is what was asked for |
+| `offload_strategy: none`, or `no_ram` | the load buffer only — it fails on VRAM instead |
+| anything else | the **spill**: footprint minus free VRAM |
+| footprint unknown | the load buffer only, and the load proceeds unguarded |
+
+plus a 2 GB staging allowance for the shard buffer, the safetensors read window
+and dequantisation scratch.
+
+The budget is the tighter of what the kernel says is available (`MemAvailable`,
+which already counts reclaimable cache) and what is left under `max_ram_gb`,
+less `offload.ram_admission_floor_gb` (default 4 GB) kept free for the OS and
+the sibling engines.
+
+Refusal is the last step, never the first: the ladder runs, the budget is
+re-measured, and a load that fits afterwards is admitted. Only then does the
+request fail — as a 503 naming the model, the estimate, the budget, the
+shortfall, and what to change on the card. A model whose footprint could not be
+determined is never refused on a number nobody has; it loses the protection and
+says so in the log.
+
+### Sizing that skips itself is worse than no sizing
+
+The same night, the GGUF path printed this before loading a 35B:
+
+```
+(auto-offload sizing skipped: cannot access local variable '_ck'
+ where it is not associated with a value)
+```
+
+The auto-offload step reads the configured KV-cache types to decide how much
+headroom to reserve, and it read them from a binding made a hundred lines
+further down, where `llama_kwargs` is assembled. Every load that reached the
+branch raised `UnboundLocalError`, the handler reported it as *skipped*, and
+nothing reduced `n_gpu_layers`. llama.cpp was then asked to keep every layer on
+a card that could not hold them and answered with a **43,562 MiB CPU KV
+buffer** — which is what took the host down.
+
+Two things follow. The read is hoisted above its use, with a test asserting the
+source order rather than a line number. And the handler no longer says
+*skipped*: a step that did not run leaves the load UNSIZED, and the message says
+that, because "skipped" invites the reader to assume nothing needed doing.
+
 ## A resident model still needs room to run
 
 Every reservation above is taken when a model LOADS. A model that is already

tests/test_remote_code_allowlist.py

@@ -129,7 +129,11 @@ def test_a_permission_error_is_not_swallowed_by_the_loader():
     a refusal must not be one of them."""
     src = open("codai/api/images.py").read()
     head = src.index("_img_remote = {}")
-    body = src[head:head + 900]
+    # Bounded by the next statement after the block, not by a character count:
+    # a fixed window silently stops covering the code it was written for the
+    # moment anything is added inside it, which is how this test broke when the
+    # declared-class check went in.
+    body = src[head:src.index("_img_comp_models", head)]
     assert "except PermissionError:" in body and "raise" in body
 
 
@@ -147,3 +151,64 @@ def test_such_a_model_is_not_tried_as_sdxl_first():
 
 def test_the_field_is_savable_from_the_admin_page():
     assert '"custom_pipeline",' in open("codai/admin/routes.py").read()
+
+
+# ---------------------------------------------------------------------------
+# A repo that declares its own class without any card saying so
+# ---------------------------------------------------------------------------
+
+def test_a_repo_declaring_an_unavailable_class_is_detected(monkeypatch):
+    """MOSS-Image and Kromoss, exactly as they failed: no card named a
+    custom_pipeline, so nothing fired, and the load died on
+    'module diffusers has no attribute MOSSImagePipeline'."""
+    from codai.models import remote_code as R
+    monkeypatch.setattr(R, "declared_pipeline_class",
+                        lambda repo: "MOSSImagePipeline")
+    kw = R.kwargs_for_declared_class(_cfg(["openmoss-team/*"]),
+                                     "OpenMOSS-Team/MOSS-Image-SFT")
+    assert kw["trust_remote_code"] is True
+    assert kw["custom_pipeline"] == "OpenMOSS-Team/MOSS-Image-SFT"
+    assert kw["_declared_class"] == "MOSSImagePipeline"
+
+
+def test_such_a_repo_is_refused_by_name_when_not_allow_listed(monkeypatch):
+    """The refusal has to name the setting — that is the whole reason the
+    allow-list is spelled out rather than implied."""
+    from codai.models import remote_code as R
+    monkeypatch.setattr(R, "declared_pipeline_class",
+                        lambda repo: "MOSSImagePipeline")
+    with pytest.raises(PermissionError) as ei:
+        R.kwargs_for_declared_class(_cfg([]), "lodestones/Kromoss")
+    msg = str(ei.value)
+    assert "models.trust_remote_code" in msg
+    assert "lodestones/*" in msg
+    assert "MOSSImagePipeline" in msg
+
+
+def test_an_ordinary_repo_pays_nothing(monkeypatch):
+    """The class diffusers already has must not trigger any of this: every
+    normal image model goes through this code path."""
+    from codai.models import remote_code as R
+    monkeypatch.setattr(R, "declared_pipeline_class",
+                        lambda repo: "StableDiffusionXLPipeline")
+    assert R.kwargs_for_declared_class(_cfg([]), "any/sdxl") == {}
+
+
+def test_a_repo_that_declares_nothing_is_left_alone(monkeypatch):
+    from codai.models import remote_code as R
+    monkeypatch.setattr(R, "declared_pipeline_class", lambda repo: None)
+    assert R.kwargs_for_declared_class(_cfg([]), "any/model") == {}
+
+
+def test_the_marker_never_reaches_from_pretrained():
+    """_declared_class is ours, for the log line. Passing it on would be a
+    TypeError from diffusers at the worst possible moment."""
+    src = open("codai/api/images.py").read()
+    head = src.index("_img_remote = {}")
+    body = src[head:src.index("_img_comp_models", head)]
+    assert '_img_remote.pop("_declared_class"' in body
+
+
+def test_the_loader_asks_the_repository_when_no_card_did():
+    src = open("codai/api/images.py").read()
+    assert "kwargs_for_declared_class(" in src