refuse a load that cannot fit in host RAM, and size the one that was skipping itself

A load is admitted to VRAM by estimate: ensure_vram_for sizes the model, evicts
what is in the way, and failing that a CUDA OOM is caught and reported. Host RAM
had no equivalent, and the kernel supplies its own: four engines SIGKILLed in
five hours on a 54 GB host, each one while a model staged its weights.

    Oct 29 23:03  Killed coderai-nvidia-gguf   anon-rss 46.3 GB
    Oct 30 03:10  Killed coderai-nvidia        anon-rss 47.7 GB
    Oct 30 03:36  Killed coderai-nvidia        anon-rss 31.6 GB
    Oct 30 03:49  Killed coderai-nvidia        anon-rss 32.7 GB
    front: engine#0 exited (code -9); respawning (1/5 in 120s)

The RAM watcher was awake for all of it and logged "RSS 50.6/43.0 GB (118%) --
mitigation [gc+trim+evict_idle] -> 53.7 GB": every rung of its ladder releases
memory already held, and none of them can decline the load doing the allocating.
SIGKILL then takes the engine rather than the request, so the client that asked
for the oversized model saw a dropped connection -- and so did every unrelated
request in flight, plus everything arriving during the 120-second respawn. One
bad load reads as the whole server being unreliable.

So offload.ram_admission (default on) adds the missing half. codai/models/
host_ram.py sizes a pending load -- measured host residency if the model has
loaded before, the whole footprint for an offload strategy that asks for it,
the spill past free VRAM otherwise, plus a staging allowance -- against the
tighter of MemAvailable and the headroom under max_ram_gb, less a floor kept
for the OS and the sibling engines. ensure_host_ram_for() runs the existing
pressure ladder FIRST and re-measures, so refusal is the last step; when it
comes, it is a 503 naming the model, the estimate, the budget, the shortfall
and what to change on the card. A model whose footprint cannot be determined
is never refused on a number nobody has.

And the proximate cause of the biggest of those kills was a sizing step that
reported itself as skipped:

    (auto-offload sizing skipped: cannot access local variable '_ck'
     where it is not associated with a value)

The GGUF auto-offload branch reads the configured KV-cache types to size its
headroom, from a binding made ~110 lines further down where llama_kwargs is
assembled. Every load that reached the branch raised UnboundLocalError, the
handler called it "skipped", and nothing reduced n_gpu_layers -- so llama.cpp
was asked to keep every layer of a 35B on a card that could not hold them, and
answered with a 43,562 MiB CPU KV buffer. The read is hoisted above its use
(tested on source order, not a line number), and the handler now says the load
is UNSIZED, because "skipped" invites the reader to assume nothing needed doing.

Separately, and found while checking which cards were really misconfigured:
a repo whose model_index.json declares a pipeline class diffusers does not have
is now detected and loaded with its own code when allow-listed, or refused by
name when not. The allow-list and the mechanism both already existed, but only
fired when a card named a custom_pipeline -- nothing ever did, so MOSS-Image
and Kromoss failed with "module diffusers has no attribute MOSSImagePipeline",
which names neither the cause nor the remedy and reads as a broken model.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.27"
+__version__ = "0.5.28"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -4032,6 +4032,8 @@ def build_settings_dict(c, gpu_cards):
             "ram_watch_poll_seconds": c.offload.ram_watch_poll_seconds,
             "ram_watch_soft_fraction": c.offload.ram_watch_soft_fraction,
             "ram_watch_cuda": c.offload.ram_watch_cuda,
+            "ram_admission": c.offload.ram_admission,
+            "ram_admission_floor_gb": c.offload.ram_admission_floor_gb,
             "gpu_split": c.offload.gpu_split,
             "tensor_split": c.offload.tensor_split,
             "split_strategy": c.offload.split_strategy,
@@ -4517,6 +4519,10 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
         if "ram_watch_soft_fraction" in off:
             c.offload.ram_watch_soft_fraction = float(off["ram_watch_soft_fraction"] or c.offload.ram_watch_soft_fraction)
         c.offload.ram_watch_cuda = bool(off.get("ram_watch_cuda", c.offload.ram_watch_cuda))
+        c.offload.ram_admission = bool(off.get("ram_admission", c.offload.ram_admission))
+        if "ram_admission_floor_gb" in off:
+            c.offload.ram_admission_floor_gb = float(
+                off["ram_admission_floor_gb"] or c.offload.ram_admission_floor_gb)
         c.offload.gpu_split = bool(off.get("gpu_split", c.offload.gpu_split))
         if "tensor_split" in off:
             c.offload.tensor_split = off["tensor_split"] or None
@@ -4552,6 +4558,8 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
                 ga.ram_watch_poll_seconds = c.offload.ram_watch_poll_seconds
                 ga.ram_watch_soft_fraction = c.offload.ram_watch_soft_fraction
                 ga.ram_watch_cuda = c.offload.ram_watch_cuda
+                ga.ram_admission = c.offload.ram_admission
+                ga.ram_admission_floor_gb = c.offload.ram_admission_floor_gb
                 ga.gpu_split = c.offload.gpu_split
                 ga.tensor_split = c.offload.tensor_split
                 ga.split_strategy = c.offload.split_strategy

codai/api/app.py

@@ -930,6 +930,36 @@ async def task_cancelled_handler(request: Request, exc: TaskCancelled):
     return JSONResponse(status_code=499, content={"detail": "Task cancelled", "task_id": tid})
 
 
+from codai.models.host_ram import HostRamExhausted
+
+
+@app.exception_handler(HostRamExhausted)
+async def host_ram_exhausted_handler(request: Request, exc: HostRamExhausted):
+    """A load was refused for want of host RAM.
+
+    503, because this is the server declining to start work it cannot finish,
+    and the condition is transient: the same request succeeds once a resident
+    model is evicted or a smaller build is asked for. Registered once here
+    rather than caught per route — the refusal can come from any path that
+    loads a model (chat, images, video, audio), and every one of them should
+    report the numbers instead of a bare 500.
+
+    Before this existed the request did not fail at all: the host ran out of
+    memory, the kernel killed the engine, and the client saw a dropped
+    connection while every other model's requests died with it.
+    """
+    return JSONResponse(
+        status_code=503,
+        content={"error": {"message": exc.message(),
+                           "type": "insufficient_host_memory",
+                           "param": exc.model,
+                           "code": "host_ram_exhausted",
+                           "needed_gb": exc.need.gb,
+                           "available_gb": exc.budget.gb,
+                           "shortfall_gb": exc.shortfall_gb}},
+    )
+
+
 @app.get("/v1/models", response_model=ModelList, summary="List available models", tags=["Core"])
 async def list_models():
     """List available models."""

codai/api/images.py

@@ -784,9 +784,21 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
         from codai.admin.routes import config_manager as _cm
         _img_remote = remote_code.kwargs_for(
             getattr(_cm, "config", None), model_name, _cfg('custom_pipeline'))
+        if not _img_remote:
+            # No card named a custom_pipeline — ask the repository itself. A
+            # checkpoint whose model_index.json declares a class diffusers does
+            # not have is telling us it ships its own, and letting that load
+            # proceed produced "module diffusers has no attribute
+            # MOSSImagePipeline": a true statement that names neither the cause
+            # nor the fix, and reads as a broken model.
+            _img_remote = remote_code.kwargs_for_declared_class(
+                getattr(_cm, "config", None), model_name)
         if _img_remote:
+            _declared = _img_remote.pop("_declared_class", None)
             print(f"  Custom pipeline from the repository: "
-                  f"{_img_remote['custom_pipeline']} (allow-listed)")
+                  f"{_img_remote['custom_pipeline']} (allow-listed)"
+                  + (f" — it declares {_declared}, which is not in diffusers"
+                     if _declared else ""))
     except PermissionError:
         raise
     except Exception as e:
@@ -907,6 +919,28 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
         except Exception as _ee:
             print(f"  Proactive eviction skipped: {_ee}")
 
+    # =====================================================================
+    # Host-RAM admission
+    # =====================================================================
+    # VRAM is handled above by eviction; host RAM has to be handled by refusal,
+    # because the kernel's answer to running out of it is to kill the engine. The
+    # image path is where it bit hardest: the cards that quantize only their text
+    # encoder stage a bf16 transformer into host RAM, and on a full card nearly
+    # all of it spills. Raised, not caught — the caller turns it into a 503 that
+    # names the model, the size and the fix. See codai/models/host_ram.py.
+    try:
+        from codai.models.manager import multi_model_manager as _mmm
+    except Exception:
+        _mmm = None
+    if _mmm is not None:
+        _hkey = None
+        for _k in (f"image:{model_name}", model_name):
+            if _k in _mmm.config:
+                _hkey = _k
+                break
+        _mmm.ensure_host_ram_for(_hkey or f"image:{model_name}", model_name,
+                                 _requested_id)
+
     # =====================================================================
     # Standard loading path (with OOM fallback)
     # =====================================================================

codai/backends/vulkan.py

@@ -1240,6 +1240,19 @@ class VulkanBackend(ModelBackend):
             try:
                 _exp = kwargs.get('expected_vram_gb')
                 _nlayers = _gguf_block_count(model_path)
+                # Read the KV-cache types HERE, not from the binding made further
+                # down when llama_kwargs is assembled. That binding happens after
+                # this block, so reading it here raised UnboundLocalError on every
+                # load that reached the offload branch -- caught below and printed
+                # as "(auto-offload sizing skipped: cannot access local variable
+                # '_ck' …)". Nothing then reduced n_gpu_layers, so a 35B that did
+                # not fit kept every layer on the GPU and llama.cpp answered with a
+                # 43,562 MiB CPU KV buffer: the host ran out of memory and the
+                # kernel killed the engine (code -9), three times in five hours.
+                # A skipped sizing step must not read as a sized one.
+                _kv_raw_cfg = kwargs.get('_raw_cfg') or {}
+                _ck = kwargs.get('cache_type_k', _kv_raw_cfg.get('cache_type_k'))
+                _cv = kwargs.get('cache_type_v', _kv_raw_cfg.get('cache_type_v'))
                 # Usable VRAM is the POOL across the cards this engine can use, not
                 # just main_gpu: all same-backend cards always (e.g. both 3090s), plus
                 # the other backend's cards when cross-split is on (3090 + RX 580).
@@ -1316,7 +1329,17 @@ class VulkanBackend(ModelBackend):
                           f"GPU, {_nlayers - _fit} on CPU (slower). Lower n_ctx or use a "
                           f"smaller model to keep it fully on GPU.", flush=True)
             except Exception as _off_e:
-                print(f"  (auto-offload sizing skipped: {_off_e})", flush=True)
+                # Loud on purpose, and it says what the consequence is. This
+                # printed "(auto-offload sizing skipped: …)" for an
+                # UnboundLocalError in its own code, which reads like a decision
+                # not to offload rather than a failure to decide -- and the model
+                # that was not sized then took the host down with a 43 GB CPU KV
+                # buffer. A sizing step that did not run leaves the load UNSIZED,
+                # so say that, with the error.
+                print(f"  WARNING: auto-offload sizing FAILED ({type(_off_e).__name__}: "
+                      f"{_off_e}) — this load is UNSIZED: every layer stays on the "
+                      f"GPU and llama.cpp may put the KV cache in host RAM. Check "
+                      f"n_gpu_layers / n_ctx for this model.", flush=True)
 
         # Configure context size
         if no_ram:

codai/config.py

@@ -292,6 +292,14 @@ class OffloadConfig:
     ram_watch_poll_seconds: float = 15.0    # how often the watcher samples RSS
     ram_watch_soft_fraction: float = 0.90   # mitigate at/above this fraction of the cap
     ram_watch_cuda: bool = True             # allow mitigation to call CUDA empty_cache()
+    # Host-RAM ADMISSION: refuse a load whose estimated host-RAM cost does not
+    # fit, instead of letting it run the host out of memory. The watcher above
+    # can only release memory already held, so it cannot stop an unquantized
+    # card mid-load -- on a 54 GB host that cost four OOM-killed engines in five
+    # hours, each reported to clients as "crashes the engine". See
+    # codai/models/host_ram.py. Off means loads proceed as they did before.
+    ram_admission: bool = True
+    ram_admission_floor_gb: float = 4.0     # host RAM kept free for OS + engines
     # Unload a model that has gone untouched for this long, in seconds (0 = never).
     #
     # Eviction already frees a model when something else needs the memory, so this
@@ -2213,6 +2221,8 @@ class ConfigManager:
                 "ram_watch_poll_seconds": self.config.offload.ram_watch_poll_seconds,
                 "ram_watch_soft_fraction": self.config.offload.ram_watch_soft_fraction,
                 "ram_watch_cuda": self.config.offload.ram_watch_cuda,
+                "ram_admission": self.config.offload.ram_admission,
+                "ram_admission_floor_gb": self.config.offload.ram_admission_floor_gb,
                 "gpu_split": self.config.offload.gpu_split,
                 "tensor_split": self.config.offload.tensor_split,
                 "split_strategy": self.config.offload.split_strategy,

codai/main.py

@@ -1221,6 +1221,10 @@ def main():
     global_args.ram_watch_poll_seconds = config.offload.ram_watch_poll_seconds
     global_args.ram_watch_soft_fraction = config.offload.ram_watch_soft_fraction
     global_args.ram_watch_cuda = config.offload.ram_watch_cuda
+    # Host-RAM admission: refuse a load that cannot fit rather than let the
+    # kernel's OOM killer take the engine (codai/models/host_ram.py).
+    global_args.ram_admission = config.offload.ram_admission
+    global_args.ram_admission_floor_gb = config.offload.ram_admission_floor_gb
     # Cross-backend GPU pooling defaults (per-model config can override). Read by
     # the GGUF loader to set llama.cpp split_mode/tensor_split.
     global_args.gpu_split = getattr(config.offload, "gpu_split", False)

codai/models/host_ram.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""Host-RAM admission control for model loads.
+
+A load is admitted to VRAM by estimate: :meth:`ensure_vram_for` sizes the model,
+evicts what is in the way, and the load proceeds. Host RAM had no equivalent, and
+the asymmetry is not cosmetic -- it is the difference between a clean refusal and
+a dead engine.
+
+What it cost, measured on a 54 GB production host serving four engines
+(2026-10-29/30). Six model cards configured their big weights unquantized:
+``load_in_4bit=False`` with ``component_quantization={'text_encoder': '4bit'}``,
+which quantizes the text encoder and leaves the transformer at bf16; two video
+cards asked for ``offload_strategy`` ``model``/``balanced``, which parks weights
+on the host by definition. Requesting any of them staged tens of GB into host
+RAM, and the kernel resolved it:
+
+    Oct 29 23:03  Killed coderai-nvidia-gguf   anon-rss 46.3 GB
+    Oct 30 03:10  Killed coderai-nvidia        anon-rss 47.7 GB
+    Oct 30 03:36  Killed coderai-nvidia        anon-rss 31.6 GB
+    Oct 30 03:49  Killed coderai-nvidia        anon-rss 32.7 GB
+    front: engine#0 exited (code -9); respawning (1/5 in 120s)
+
+The RAM watcher saw it and could not help: it logged ``RSS 50.6/43.0 GB (118%)
+-- mitigation [gc+trim+evict_idle] -> 53.7 GB``. Its ladder can only release
+memory that is already held; nothing in it can decline the load that is at that
+moment streaming bf16 weights into the host. So every client saw SIGKILL --
+reported as "crashes the engine", "won't load", and, for whatever else was in
+flight during the 120-second respawn, as an unrelated failure.
+
+This module is the missing half: size the host-RAM cost of a load BEFORE it
+starts, and if it cannot fit, refuse it by name with the remedy attached. One
+model then fails with an explanation instead of every model failing with a
+corpse.
+
+Deliberately pure arithmetic -- no manager, no torch, no /proc. The caller
+measures and evicts; these functions only decide. That keeps the decision
+testable without a GPU, which is the half that was never exercised before.
+"""
+
+from __future__ import annotations
+
+from typing import NamedTuple, Optional
+
+# Head-room for the load itself, on top of whatever the weights park on the host.
+# A shard-streaming load holds a shard buffer, the safetensors read window and the
+# dequantisation scratch at once; measured between 1.2 and 1.8 GB across the HF and
+# diffusers paths, so 2.0 GB covers it without pretending to be precise.
+STAGING_GB = 2.0
+
+# Host RAM never promised to a model, so the OS, the front and the sibling engines
+# keep breathing room even when a load is admitted right up to the line.
+DEFAULT_FLOOR_GB = 4.0
+
+# Strategies that put the weights on the HOST by design, rather than as overflow.
+# For these the host cost is the model's whole footprint, not the part that missed
+# the GPU: that is what the operator asked for.
+HOST_RESIDENT_STRATEGIES = frozenset({"model", "sequential", "cpu"})
+
+# Strategies that keep every weight on the GPU, so a model too big for the card
+# fails on VRAM (loudly, recoverably) and never reaches host RAM at all.
+GPU_ONLY_STRATEGIES = frozenset({"none"})
+
+
+class Need(NamedTuple):
+    """The estimated host-RAM cost of a pending load, and where it came from."""
+
+    gb: float
+    basis: str          # "measured" | "host-resident" | "gpu-only" | "spill" | "unknown"
+    detail: str         # one human-readable line, for the log and the refusal
+
+
+class Budget(NamedTuple):
+    """How much host RAM a load may take right now, and what limits it."""
+
+    gb: float
+    binding: str        # "kernel" | "cap" | "floor"
+    detail: str
+
+
+class Verdict(NamedTuple):
+    ok: bool
+    shortfall_gb: float
+
+
+class HostRamExhausted(RuntimeError):
+    """A load was refused because its host-RAM cost does not fit.
+
+    Carries the numbers so an API layer can report them without re-deriving
+    anything, and so a test can assert on them rather than on prose.
+    """
+
+    def __init__(self, model: str, need: Need, budget: Budget, remedy: str = ""):
+        self.model = model
+        self.need = need
+        self.budget = budget
+        self.remedy = remedy
+        self.shortfall_gb = round(max(0.0, need.gb - budget.gb), 2)
+        super().__init__(self.message())
+
+    def message(self) -> str:
+        msg = (f"'{self.model}' needs about {self.need.gb:.1f} GB of host RAM to "
+               f"load ({self.need.detail}), and only {self.budget.gb:.1f} GB is "
+               f"available ({self.budget.detail}) — short by "
+               f"{self.shortfall_gb:.1f} GB. Refusing the load: going ahead would "
+               f"exhaust host memory and the kernel would kill the engine, which "
+               f"fails every other request too.")
+        if self.remedy:
+            msg += " " + self.remedy
+        return msg
+
+
+def estimate_need(*, total_need_gb: float, free_vram_gb: float,
+                  measured_ram_gb: Optional[float] = None,
+                  offload_strategy: Optional[str] = None,
+                  no_ram: bool = False,
+                  staging_gb: float = STAGING_GB) -> Need:
+    """Host RAM a pending load will want, in GB.
+
+    ``total_need_gb`` is the model's FULL footprint -- the placement-independent
+    figure ``measured_vram_gb`` already records -- not the GPU-resident slice.
+
+    The estimate follows what actually determines the cost:
+
+    * ``measured_ram_gb`` wins outright. The loader records what a load really
+      parked on the host (``of which X GB was offloaded to host RAM this load``),
+      so once a model has loaded once there is no need to guess.
+    * A host-resident offload strategy means the whole footprint lands on the
+      host, because that is the point of asking for it.
+    * A GPU-only strategy means none of it does: too big a model fails on VRAM.
+    * Otherwise the host carries the SPILL -- whatever did not fit on the card.
+      This is the case that killed the engine: a bf16 transformer on a card with
+      other models resident spills nearly all of itself.
+
+    An unknown footprint yields the staging allowance alone, flagged ``unknown``:
+    a load whose size cannot be determined is never refused on a number nobody
+    has, it just loses the protection.
+    """
+    staging = max(0.0, float(staging_gb or 0.0))
+    strategy = (offload_strategy or "").strip().lower()
+
+    if measured_ram_gb is not None and float(measured_ram_gb) > 0:
+        held = float(measured_ram_gb)
+        return Need(round(held + staging, 2), "measured",
+                    f"{held:.1f} GB measured on host RAM on a previous load, "
+                    f"plus {staging:.1f} GB to stage it")
+
+    if no_ram or strategy in GPU_ONLY_STRATEGIES:
+        return Need(round(staging, 2), "gpu-only",
+                    f"loads GPU-only, so only the {staging:.1f} GB load buffer "
+                    f"touches host RAM")
+
+    total = max(0.0, float(total_need_gb or 0.0))
+    if total <= 0:
+        return Need(round(staging, 2), "unknown",
+                    "footprint unknown, so only the load buffer is accounted for")
+
+    if strategy in HOST_RESIDENT_STRATEGIES:
+        return Need(round(total + staging, 2), "host-resident",
+                    f"offload_strategy='{strategy}' keeps the whole "
+                    f"{total:.1f} GB footprint on the host, plus "
+                    f"{staging:.1f} GB to stage it")
+
+    spill = max(0.0, total - max(0.0, float(free_vram_gb or 0.0)))
+    return Need(round(spill + staging, 2), "spill",
+                f"{total:.1f} GB footprint against {max(0.0, float(free_vram_gb or 0.0)):.1f} GB "
+                f"free VRAM spills {spill:.1f} GB to the host, plus "
+                f"{staging:.1f} GB to stage it")
+
+
+def budget(*, kernel_available_gb: float, cap_gb: Optional[float] = None,
+           fleet_held_gb: float = 0.0,
+           floor_gb: float = DEFAULT_FLOOR_GB) -> Budget:
+    """Host RAM a load may take right now, in GB.
+
+    Two limits apply and the tighter one binds:
+
+    * what the kernel says can be allocated (``MemAvailable``, which already
+      counts the page cache it is willing to drop), and
+    * the configured server-wide ceiling minus what the fleet already holds.
+
+    ``floor_gb`` is then held back from whichever bound won, so admitting a load
+    at the limit still leaves the OS and the sibling engines room to run. The
+    result is clamped at zero: a fleet already over its cap has no budget, which
+    is the correct answer rather than a negative one.
+    """
+    floor = max(0.0, float(floor_gb or 0.0))
+    kernel = max(0.0, float(kernel_available_gb or 0.0))
+    limit, binding, detail = kernel, "kernel", f"{kernel:.1f} GB available on the host"
+
+    if cap_gb and float(cap_gb) > 0:
+        headroom = float(cap_gb) - max(0.0, float(fleet_held_gb or 0.0))
+        if headroom < limit:
+            limit, binding = headroom, "cap"
+            detail = (f"{headroom:.1f} GB left under the {float(cap_gb):.0f} GB "
+                      f"server RAM cap, with the fleet holding "
+                      f"{max(0.0, float(fleet_held_gb or 0.0)):.1f} GB")
+
+    usable = limit - floor
+    if usable <= 0:
+        return Budget(0.0, "floor",
+                      detail + f", all of it inside the {floor:.1f} GB kept free "
+                               f"for the OS and the other engines")
+    return Budget(round(usable, 2), binding,
+                  detail + f", less {floor:.1f} GB kept free for the OS and the "
+                           f"other engines")
+
+
+def verdict(need: Need, bud: Budget) -> Verdict:
+    """Whether ``need`` fits in ``bud``."""
+    if need.gb <= bud.gb:
+        return Verdict(True, 0.0)
+    return Verdict(False, round(need.gb - bud.gb, 2))
+
+
+def remedy_for(cfg: Optional[dict], basis: str = "") -> str:
+    """One actionable sentence about what to change on the card, or "".
+
+    Derived from the card itself, because the fix differs: an unquantized
+    transformer wants quantizing, a host-resident offload strategy wants
+    reconsidering, and a model that simply does not fit wants a smaller build.
+    """
+    cfg = cfg or {}
+    comp = cfg.get("component_quantization") or {}
+    if isinstance(comp, dict) and comp and not any(
+            k in comp for k in ("transformer", "transformer_2", "unet")):
+        covered = ", ".join(sorted(comp)) or "nothing"
+        return ("This card quantizes only " + covered + ", so its transformer "
+                "loads at full precision — add 'transformer' to "
+                "component_quantization (or request the pre-quantized build of "
+                "this model) and it will fit.")
+    strategy = str(cfg.get("offload_strategy") or "").strip().lower()
+    if strategy in HOST_RESIDENT_STRATEGIES:
+        return (f"This card sets offload_strategy='{strategy}', which holds the "
+                f"weights in host RAM by design — quantize it or give it a "
+                f"strategy that keeps them on the GPU.")
+    if basis == "spill" and not (cfg.get("load_in_4bit") or cfg.get("load_in_8bit")):
+        return ("This card has no quantization set, so its weights load at full "
+                "precision — quantize it, or request a pre-quantized build.")
+    return ""

codai/models/manager.py

@@ -2219,6 +2219,10 @@ class MultiModelManager:
                         self.ensure_vram_for(model_name, requested_id=requested_id)
                     except Exception as _ev_e:
                         print(f"  (ensure_vram_for warning: {_ev_e})")
+                    # Outside that except on purpose: a VRAM reservation failing
+                    # is a warning (the load can still work), while a host-RAM
+                    # refusal is the decision itself and must reach the caller.
+                    self.ensure_host_ram_for(model_name, requested_id=requested_id)
                 _snap = self.vram_before_load()
                 _ram_snap = self._get_own_ram_gb()
                 # Tell the backend how much VRAM this model is expected to need so
@@ -6019,6 +6023,130 @@ class MultiModelManager:
                   f"have {free_gb:.1f} GB free — evicting models to make room")
             self._evict_models_for_vram(needed_gb)
 
+    @staticmethod
+    def _ram_admission_enabled() -> bool:
+        """Whether a load may be refused for want of host RAM (default True)."""
+        try:
+            from codai.api.state import get_global_args
+            ga = get_global_args()
+            return bool(getattr(ga, 'ram_admission', True)) if ga else True
+        except Exception:
+            return True
+
+    @staticmethod
+    def _ram_admission_floor_gb() -> float:
+        """Host RAM never promised to a load, in GB."""
+        from codai.models import host_ram
+        try:
+            from codai.api.state import get_global_args
+            ga = get_global_args()
+            val = getattr(ga, 'ram_admission_floor_gb', None) if ga else None
+            return float(val) if val is not None else host_ram.DEFAULT_FLOOR_GB
+        except Exception:
+            return host_ram.DEFAULT_FLOOR_GB
+
+    def _host_ram_budget(self):
+        """The host-RAM budget a load may take right now."""
+        from codai.models import host_ram, proc_mem
+        return host_ram.budget(
+            kernel_available_gb=proc_mem.host_available_gb(),
+            cap_gb=self._ram_cap_gb(),
+            fleet_held_gb=self._get_process_ram_gb(),
+            floor_gb=self._ram_admission_floor_gb(),
+        )
+
+    def ensure_host_ram_for(self, model_key: str, resolved_name: str = None,
+                            requested_id: str = None, cfg: dict = None) -> None:
+        """Refuse a load that cannot fit in host RAM, before it starts.
+
+        The counterpart of :meth:`ensure_vram_for`, and the reason it exists is
+        that it had none. VRAM pressure is answered by eviction and, failing
+        that, by a CUDA OOM the server catches and reports. Host-RAM pressure is
+        answered by the kernel's OOM killer, which takes the whole engine with
+        it: four times in five hours on a 54 GB host the night this was written,
+        each time while an unquantized card staged its weights, each time
+        reported to the client as "crashes the engine" (see
+        :mod:`codai.models.host_ram`).
+
+        So the ladder here ends differently. Make room first -- trim, then evict
+        idle models, exactly as RAM pressure is handled elsewhere -- and if the
+        load still does not fit, raise :class:`~codai.models.host_ram.HostRamExhausted`.
+        One model then fails with its size, the budget and the fix, while every
+        other model keeps serving.
+
+        Never refuses a model that is already resident (it is not about to
+        allocate anything), and never refuses on a footprint it could not
+        determine.
+        """
+        from codai.models import host_ram
+        if not self._ram_admission_enabled():
+            return
+        for _k in (model_key, resolved_name, requested_id):
+            if _k and _k in self.models:
+                return
+
+        if cfg is None:
+            cfg = {}
+            for _k in (model_key, resolved_name, requested_id):
+                if _k and _k in self.config:
+                    cfg = self.config.get(_k) or {}
+                    break
+
+        try:
+            _measured = cfg.get("measured_ram_gb")
+            _measured = float(_measured) if _measured else None
+        except (TypeError, ValueError):
+            _measured = None
+
+        need = host_ram.estimate_need(
+            total_need_gb=self._get_model_used_vram_gb(
+                model_key, resolved_name, requested_id),
+            free_vram_gb=self._get_free_vram_gb(),
+            measured_ram_gb=_measured,
+            offload_strategy=cfg.get("offload_strategy"),
+            no_ram=bool(cfg.get("no_ram")),
+        )
+        bud = self._host_ram_budget()
+        if host_ram.verdict(need, bud).ok:
+            return
+        if need.basis == "unknown":
+            # Nothing was ever measured or configured for this model, so the
+            # only figure on the table is the load buffer — a guess about every
+            # model rather than a fact about this one. Refusing on it would
+            # block a 300 MB embedder on a tight host for no reason. Say that
+            # the protection does not cover this load, and let it proceed.
+            print(f"Loading '{model_key}': host-RAM admission has no footprint "
+                  f"for this model, so it cannot be checked — proceeding "
+                  f"unguarded (budget {bud.gb:.1f} GB; {bud.detail})")
+            return
+
+        print(f"Loading '{model_key}': needs ~{need.gb:.1f} GB host RAM "
+              f"({need.detail}), budget is {bud.gb:.1f} GB ({bud.detail}) — "
+              f"freeing host RAM before deciding")
+        try:
+            _trim_cpu_ram()
+        except Exception:
+            pass
+        if self._evict_idle_on_ram_enabled():
+            _cap = self._ram_cap_gb()
+            if _cap:
+                self._evict_models_for_ram(_cap * 0.85)
+            else:
+                # No cap configured, so there is no target to evict towards --
+                # free what has gone quiet and re-measure against the kernel's
+                # figure. 30 s rather than "any idle model": a model being used
+                # in a request loop is idle between its own requests, and pulling
+                # it to admit another load just trades one reload for another.
+                self.unload_idle_models(30.0)
+
+        bud = self._host_ram_budget()
+        if host_ram.verdict(need, bud).ok:
+            print(f"  host RAM freed — {bud.gb:.1f} GB now available, load admitted")
+            return
+        raise host_ram.HostRamExhausted(
+            str(requested_id or resolved_name or model_key), need, bud,
+            host_ram.remedy_for(cfg, need.basis))
+
     def request_model(self, requested_model: str, model_type: str = None,
                       extra_vram_gb: float = 0.0) -> Dict[str, Any]:
         """
@@ -6343,6 +6471,7 @@ class MultiModelManager:
             # Not loaded — make room in VRAM (evict other models as needed).
             self.ensure_vram_for(model_key, resolved_name, extra_vram_gb,
                                  requested_id=_requested_id)
+            self.ensure_host_ram_for(model_key, resolved_name, _requested_id)
             return {
                 'model_key': model_key,
                 'model_name': resolved_name,
@@ -6368,6 +6497,7 @@ class MultiModelManager:
             # model).  Make room again so it returns to VRAM instead of CPU.
             self.ensure_vram_for(model_key, resolved_name, extra_vram_gb,
                                  requested_id=_requested_id)
+            self.ensure_host_ram_for(model_key, resolved_name, _requested_id)
             return {
                 'model_key': model_key,
                 'model_name': resolved_name,
@@ -6395,6 +6525,7 @@ class MultiModelManager:
             print(f"Loadall mode: Model '{model_key}' not pre-loaded, will load now")
             self.ensure_vram_for(model_key, resolved_name, extra_vram_gb,
                                  requested_id=_requested_id)
+            self.ensure_host_ram_for(model_key, resolved_name, _requested_id)
             return {
                 'model_key': model_key,
                 'model_name': resolved_name,

codai/models/remote_code.py

@@ -90,3 +90,65 @@ def kwargs_for(cfg, repo: str, custom_pipeline: Optional[str]) -> dict:
         return {}
     require(cfg, repo, f"its own pipeline module {name!r}")
     return {"custom_pipeline": name, "trust_remote_code": True}
+
+
+def declared_pipeline_class(repo: str) -> Optional[str]:
+    """The ``_class_name`` the repo's ``model_index.json`` declares, or None.
+
+    Read rather than guessed, because the name is the only thing that says a
+    repository means to supply its own pipeline: MOSS-Image declares
+    ``MOSSImagePipeline``, which exists in the repo and nowhere in diffusers.
+    """
+    import json
+    import os
+    try:
+        if os.path.isdir(repo):
+            path = os.path.join(repo, "model_index.json")
+            if not os.path.isfile(path):
+                return None
+        else:
+            from huggingface_hub import hf_hub_download
+            # Fetched, not read cache-only: a repo being loaded for the first
+            # time has no cache yet, and a cache-only answer to "what class does
+            # this declare" is permanently wrong on a freshly-rented pod. The
+            # file is a few hundred bytes and the weights are about to follow.
+            path = hf_hub_download(repo, "model_index.json")
+        with open(path) as fh:
+            name = (json.load(fh) or {}).get("_class_name")
+        return str(name) if name else None
+    except Exception:
+        # Not a diffusers repo, not reachable, or no index. Either way there is
+        # nothing to declare, and the ordinary load must be left alone.
+        return None
+
+
+def kwargs_for_declared_class(cfg, repo: str) -> dict:
+    """``from_pretrained`` kwargs for a repo whose pipeline class is its own.
+
+    The companion to :func:`kwargs_for`, which only fires when a card already
+    names a ``custom_pipeline``. Nothing filled that field in for MOSS-Image or
+    Kromoss, so the load went ahead as an ordinary repo load and diffusers
+    reported ``module diffusers has no attribute MOSSImagePipeline`` -- true,
+    and no help at all: it names neither the cause (the class lives in the
+    repository) nor the remedy (allow-list it). The operator reads it as a
+    broken model.
+
+    So detect the case from the repo's own declaration. Allow-listed, it loads
+    with its code; not allow-listed, it is refused by name with the setting to
+    change, which is this module's whole policy.
+
+    Empty dict when the declared class is available locally -- the overwhelming
+    majority -- so an ordinary load never pays for this.
+    """
+    cls = declared_pipeline_class(repo)
+    if not cls:
+        return {}
+    try:
+        import diffusers
+        if getattr(diffusers, cls, None) is not None:
+            return {}
+    except Exception:
+        return {}
+    require(cfg, repo, f"its own pipeline class {cls!r}")
+    return {"custom_pipeline": repo, "trust_remote_code": True,
+            "_declared_class": cls}

docs/memory-accounting.md

@@ -112,6 +112,84 @@ cache that the kernel reclaims by itself and that evicting a model cannot free.
 The cap can only be met by releasing memory something actually holds, so the
 metric has to be one that releasing memory moves.
 
+## A load that cannot fit in host RAM is refused
+
+The cap above is a *pressure* mechanism: a watcher samples the fleet's memory
+and, past `offload.ram_watch_soft_fraction` of the cap, runs a ladder — collect,
+trim, drop the upscaler cache, drop prefix caches, evict idle models. Every rung
+releases memory that is already held. None of them can decline a load that is at
+that moment streaming weights into the host.
+
+That gap has a cost, measured on a 54 GB host running four engines:
+
+```
+Oct 29 23:03  Killed coderai-nvidia-gguf   anon-rss 46.3 GB
+Oct 30 03:10  Killed coderai-nvidia        anon-rss 47.7 GB
+Oct 30 03:36  Killed coderai-nvidia        anon-rss 31.6 GB
+Oct 30 03:49  Killed coderai-nvidia        anon-rss 32.7 GB
+front: engine#0 exited (code -9); respawning (1/5 in 120s)
+```
+
+The watcher was awake throughout and logged
+`RSS 50.6/43.0 GB (118%) — mitigation [gc+trim+evict_idle] → 53.7 GB`: it ran
+its ladder and the number went up anyway. The kernel settled it, and SIGKILL
+takes the engine, not the request — so the client that asked for the oversized
+model saw a dropped connection, and so did every unrelated request in flight,
+plus everything that arrived during the 120-second respawn. One bad load is
+reported as the whole server being unreliable.
+
+So `offload.ram_admission` (default on) adds the missing half: before a load
+starts, estimate what it will cost the host and refuse it if it does not fit.
+`codai/models/host_ram.py` holds the arithmetic, `ensure_host_ram_for()` the
+decision, and it sits beside `ensure_vram_for()` on every load path.
+
+The estimate follows what actually determines the cost:
+
+| situation | host-RAM cost |
+| --- | --- |
+| the model loaded before | `measured_ram_gb` — what it really parked on the host |
+| `offload_strategy` `model` / `sequential` / `cpu` | the whole footprint; that is what was asked for |
+| `offload_strategy: none`, or `no_ram` | the load buffer only — it fails on VRAM instead |
+| anything else | the **spill**: footprint minus free VRAM |
+| footprint unknown | the load buffer only, and the load proceeds unguarded |
+
+plus a 2 GB staging allowance for the shard buffer, the safetensors read window
+and dequantisation scratch.
+
+The budget is the tighter of what the kernel says is available (`MemAvailable`,
+which already counts reclaimable cache) and what is left under `max_ram_gb`,
+less `offload.ram_admission_floor_gb` (default 4 GB) kept free for the OS and
+the sibling engines.
+
+Refusal is the last step, never the first: the ladder runs, the budget is
+re-measured, and a load that fits afterwards is admitted. Only then does the
+request fail — as a 503 naming the model, the estimate, the budget, the
+shortfall, and what to change on the card. A model whose footprint could not be
+determined is never refused on a number nobody has; it loses the protection and
+says so in the log.
+
+### Sizing that skips itself is worse than no sizing
+
+The same night, the GGUF path printed this before loading a 35B:
+
+```
+(auto-offload sizing skipped: cannot access local variable '_ck'
+ where it is not associated with a value)
+```
+
+The auto-offload step reads the configured KV-cache types to decide how much
+headroom to reserve, and it read them from a binding made a hundred lines
+further down, where `llama_kwargs` is assembled. Every load that reached the
+branch raised `UnboundLocalError`, the handler reported it as *skipped*, and
+nothing reduced `n_gpu_layers`. llama.cpp was then asked to keep every layer on
+a card that could not hold them and answered with a **43,562 MiB CPU KV
+buffer** — which is what took the host down.
+
+Two things follow. The read is hoisted above its use, with a test asserting the
+source order rather than a line number. And the handler no longer says
+*skipped*: a step that did not run leaves the load UNSIZED, and the message says
+that, because "skipped" invites the reader to assume nothing needed doing.
+
 ## A resident model still needs room to run
 
 Every reservation above is taken when a model LOADS. A model that is already

tests/test_host_ram_admission.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""Host-RAM admission control.
+
+The case these cover, measured on a 54 GB host: six cards quantized only their
+text encoder and left the transformer at bf16, two video cards asked for a
+host-resident offload strategy, and requesting any of them ran the host out of
+memory. The kernel killed the engine four times in five hours, so the client
+never saw a refusal -- it saw a dead engine, and so did every unrelated request
+in flight.
+
+So what is asserted here is not "the estimate is accurate" but the decision:
+does a load that cannot fit get refused BEFORE it allocates, with numbers a
+caller can act on, and does a load that can fit still proceed.
+"""
+
+import pytest
+
+from codai.models import host_ram as H
+
+
+# ---------------------------------------------------------------------------
+# estimate_need
+# ---------------------------------------------------------------------------
+
+def test_a_measured_load_is_believed_over_any_estimate():
+    """The loader records what a load really parked on the host. Once that
+    exists there is nothing left to infer."""
+    need = H.estimate_need(total_need_gb=40.0, free_vram_gb=0.0,
+                           measured_ram_gb=3.5)
+    assert need.basis == "measured"
+    assert need.gb == pytest.approx(3.5 + H.STAGING_GB)
+    assert "previous load" in need.detail
+
+
+def test_a_zero_measurement_is_not_a_measurement():
+    """0.0 means the model held nothing on the host OR that nothing was
+    recorded, and the two are indistinguishable — so fall through to the
+    estimate rather than admit a 40 GB load on the strength of a zero."""
+    need = H.estimate_need(total_need_gb=40.0, free_vram_gb=0.0,
+                           measured_ram_gb=0.0)
+    assert need.basis == "spill"
+    assert need.gb == pytest.approx(40.0 + H.STAGING_GB)
+
+
+@pytest.mark.parametrize("strategy", sorted(H.HOST_RESIDENT_STRATEGIES))
+def test_a_host_resident_strategy_costs_the_whole_footprint(strategy):
+    """offload_strategy='model' is a request to hold the weights in host RAM.
+    The cost is the model, not the part that missed the GPU — and crediting it
+    with free VRAM it will not use is how LongCat-Video was admitted onto a
+    host that could not hold it."""
+    need = H.estimate_need(total_need_gb=27.0, free_vram_gb=24.0,
+                           offload_strategy=strategy)
+    assert need.basis == "host-resident"
+    assert need.gb == pytest.approx(27.0 + H.STAGING_GB)
+    assert strategy in need.detail
+
+
+def test_a_gpu_only_load_is_charged_only_for_staging():
+    """Strategy 'none' keeps every weight on the card, so a model too big for it
+    fails on VRAM — loudly and recoverably — and never reaches host RAM."""
+    for kwargs in ({"offload_strategy": "none"}, {"no_ram": True}):
+        need = H.estimate_need(total_need_gb=60.0, free_vram_gb=1.0, **kwargs)
+        assert need.basis == "gpu-only"
+        assert need.gb == pytest.approx(H.STAGING_GB)
+
+
+def test_the_spill_is_what_did_not_fit_on_the_card():
+    need = H.estimate_need(total_need_gb=17.6, free_vram_gb=5.6)
+    assert need.basis == "spill"
+    assert need.gb == pytest.approx(12.0 + H.STAGING_GB)
+
+
+def test_a_model_that_fits_on_the_card_spills_nothing():
+    need = H.estimate_need(total_need_gb=8.0, free_vram_gb=24.0)
+    assert need.gb == pytest.approx(H.STAGING_GB)
+
+
+def test_an_unknown_footprint_is_never_charged_a_number_nobody_has():
+    """A size that could not be determined must not become a refusal. The
+    protection is lost for that model; the model still loads."""
+    need = H.estimate_need(total_need_gb=0.0, free_vram_gb=0.0)
+    assert need.basis == "unknown"
+    assert need.gb == pytest.approx(H.STAGING_GB)
+
+
+def test_free_vram_is_never_credited_as_negative_space():
+    need = H.estimate_need(total_need_gb=10.0, free_vram_gb=-5.0)
+    assert need.gb == pytest.approx(10.0 + H.STAGING_GB)
+
+
+# ---------------------------------------------------------------------------
+# budget
+# ---------------------------------------------------------------------------
+
+def test_the_kernel_binds_when_no_cap_is_configured():
+    bud = H.budget(kernel_available_gb=20.0, cap_gb=None, floor_gb=4.0)
+    assert bud.binding == "kernel"
+    assert bud.gb == pytest.approx(16.0)
+
+
+def test_the_cap_binds_when_it_is_tighter_than_the_kernel():
+    """A host with free RAM is still not free to use it past the server's own
+    ceiling — that ceiling exists so the sibling engines keep theirs."""
+    bud = H.budget(kernel_available_gb=30.0, cap_gb=43.0, fleet_held_gb=36.0,
+                   floor_gb=4.0)
+    assert bud.binding == "cap"
+    assert bud.gb == pytest.approx(3.0)
+    assert "43 GB server RAM cap" in bud.detail
+
+
+def test_the_kernel_still_binds_when_it_is_the_tighter_of_the_two():
+    bud = H.budget(kernel_available_gb=6.0, cap_gb=43.0, fleet_held_gb=10.0,
+                   floor_gb=4.0)
+    assert bud.binding == "kernel"
+    assert bud.gb == pytest.approx(2.0)
+
+
+def test_a_fleet_over_its_cap_has_no_budget_rather_than_a_negative_one():
+    bud = H.budget(kernel_available_gb=8.0, cap_gb=43.0, fleet_held_gb=50.0,
+                   floor_gb=4.0)
+    assert bud.gb == 0.0
+    assert bud.binding == "floor"
+
+
+def test_the_floor_is_held_back_from_whichever_bound_won():
+    assert H.budget(kernel_available_gb=10.0, floor_gb=0.0).gb == pytest.approx(10.0)
+    assert H.budget(kernel_available_gb=10.0, floor_gb=4.0).gb == pytest.approx(6.0)
+
+
+# ---------------------------------------------------------------------------
+# verdict + the refusal itself
+# ---------------------------------------------------------------------------
+
+def test_a_load_that_fits_is_admitted():
+    need = H.estimate_need(total_need_gb=8.0, free_vram_gb=24.0)
+    bud = H.budget(kernel_available_gb=20.0, floor_gb=4.0)
+    assert H.verdict(need, bud).ok
+
+
+def test_the_production_case_is_refused_and_says_by_how_much():
+    """Z-Image-Turbo bf16 (17.6 GB) onto a nearly-full card, on the fleet state
+    that was live when the engine was killed."""
+    need = H.estimate_need(total_need_gb=17.6, free_vram_gb=2.0)
+    bud = H.budget(kernel_available_gb=12.0, cap_gb=43.0, fleet_held_gb=40.0,
+                   floor_gb=4.0)
+    v = H.verdict(need, bud)
+    assert not v.ok
+    assert v.shortfall_gb == pytest.approx(17.6)
+
+
+def test_the_refusal_carries_the_numbers_not_just_prose():
+    """An API layer must be able to report the figures without re-deriving
+    them, and a test must be able to assert on them without matching prose."""
+    need = H.estimate_need(total_need_gb=17.6, free_vram_gb=2.0)
+    bud = H.budget(kernel_available_gb=12.0, cap_gb=43.0, fleet_held_gb=40.0)
+    exc = H.HostRamExhausted("Tongyi-MAI/Z-Image-Turbo", need, bud)
+    assert exc.model == "Tongyi-MAI/Z-Image-Turbo"
+    assert exc.need.gb == pytest.approx(17.6)
+    assert exc.budget.gb == 0.0
+    assert exc.shortfall_gb == pytest.approx(17.6)
+    assert isinstance(exc, RuntimeError)   # callers that catch broadly still see it
+
+
+def test_the_refusal_explains_that_proceeding_kills_the_engine():
+    """The message is the whole point: before this existed the same condition
+    produced a dropped connection and a respawning engine."""
+    need = H.estimate_need(total_need_gb=30.0, free_vram_gb=0.0)
+    bud = H.budget(kernel_available_gb=5.0)
+    msg = H.HostRamExhausted("m", need, bud).message()
+    assert "host RAM" in msg and "kill the engine" in msg
+
+
+# ---------------------------------------------------------------------------
+# remedy_for — the fix differs by what the card actually says
+# ---------------------------------------------------------------------------
+
+def test_a_text_encoder_only_card_is_told_to_quantize_its_transformer():
+    """The exact shape of the six cards that caused this: quantization present,
+    covering the small half."""
+    r = H.remedy_for({"component_quantization": {"text_encoder": "4bit"}}, "spill")
+    assert "transformer" in r and "text_encoder" in r
+
+
+def test_a_card_that_already_quantizes_its_transformer_gets_no_such_advice():
+    for comp in ({"transformer": "4bit"}, {"unet": "4bit"},
+                 {"transformer": "4bit", "text_encoder": "4bit"}):
+        assert "add 'transformer'" not in H.remedy_for(
+            {"component_quantization": comp}, "spill")
+
+
+def test_a_host_resident_card_is_told_about_its_strategy():
+    r = H.remedy_for({"offload_strategy": "model"}, "host-resident")
+    assert "offload_strategy='model'" in r
+
+
+def test_an_unquantized_card_is_told_it_is_unquantized():
+    r = H.remedy_for({"offload_strategy": "auto"}, "spill")
+    assert "no quantization" in r
+
+
+def test_a_quantized_card_that_merely_does_not_fit_is_not_lectured():
+    """Advice that does not apply is worse than none: it sends the operator to
+    change a setting that is already right."""
+    assert H.remedy_for({"load_in_4bit": True, "offload_strategy": "auto"},
+                        "spill") == ""
+
+
+def test_no_card_at_all_yields_no_advice():
+    assert H.remedy_for(None) == ""
+    assert H.remedy_for({}) == ""
+
+
+# ---------------------------------------------------------------------------
+# the manager gate
+# ---------------------------------------------------------------------------
+
+@pytest.fixture
+def gate(monkeypatch):
+    """The real method, on a manager whose measurements are ours."""
+    from codai.models.manager import multi_model_manager as m
+    monkeypatch.setattr(m, "models", {}, raising=False)
+    monkeypatch.setattr(m, "config", {}, raising=False)
+    monkeypatch.setattr(m, "_ram_admission_enabled", staticmethod(lambda: True))
+    monkeypatch.setattr(m, "_ram_admission_floor_gb", staticmethod(lambda: 4.0))
+    monkeypatch.setattr(m, "_evict_idle_on_ram_enabled", staticmethod(lambda: False))
+    monkeypatch.setattr(m, "_get_free_vram_gb", lambda *a, **k: 2.0)
+    monkeypatch.setattr(m, "_get_model_used_vram_gb", lambda *a, **k: 17.6)
+    monkeypatch.setattr(m, "_ram_cap_gb", staticmethod(lambda: 43.0))
+    monkeypatch.setattr(m, "_get_process_ram_gb", staticmethod(lambda: 40.0))
+    return m
+
+
+def test_the_gate_refuses_the_load_that_killed_the_engine(gate, monkeypatch):
+    monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: 12.0)
+    gate.config["image:Z"] = {"component_quantization": {"text_encoder": "4bit"}}
+    with pytest.raises(H.HostRamExhausted) as ei:
+        gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo", "Z-Image-Turbo")
+    # Named by what the caller ASKED for, not by the internal key.
+    assert ei.value.model == "Z-Image-Turbo"
+    assert "add 'transformer'" in ei.value.remedy
+
+
+def test_the_gate_lets_a_load_through_when_there_is_room(gate, monkeypatch):
+    monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: 80.0)
+    monkeypatch.setattr(gate, "_ram_cap_gb", staticmethod(lambda: None))
+    gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo")
+
+
+def test_a_resident_model_is_never_refused(gate, monkeypatch):
+    """It is not about to allocate anything — and asking "is there room for it?"
+    while it IS the memory is the self-eviction bug the VRAM path already had."""
+    monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: 0.0)
+    for key in ("image:Z", "Tongyi-MAI/Z-Image-Turbo", "Z-Image-Turbo"):
+        gate.models.clear()
+        gate.models[key] = object()
+        gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo", "Z-Image-Turbo")
+
+
+def test_turning_admission_off_restores_the_old_behaviour(gate, monkeypatch):
+    """An operator who would rather take the risk keeps the right to."""
+    monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: 0.0)
+    monkeypatch.setattr(gate, "_ram_admission_enabled", staticmethod(lambda: False))
+    gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo")
+
+
+def test_an_unsizeable_model_is_admitted_rather_than_blocked(gate, monkeypatch):
+    monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: 7.0)
+    monkeypatch.setattr(gate, "_get_model_used_vram_gb", lambda *a, **k: 0.0)
+    gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo")
+
+
+def test_the_gate_frees_host_ram_before_it_refuses(gate, monkeypatch):
+    """Refusal is the last step, not the first: the same ladder RAM pressure
+    already uses runs first, and a load that fits afterwards is admitted."""
+    freed = []
+    avail = [6.0]
+    monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: avail[0])
+    monkeypatch.setattr(gate, "_evict_idle_on_ram_enabled", staticmethod(lambda: True))
+    monkeypatch.setattr(gate, "_ram_cap_gb", staticmethod(lambda: None))
+    monkeypatch.setattr(gate, "_get_model_used_vram_gb", lambda *a, **k: 10.0)
+
+    def _unload(idle_seconds):
+        freed.append(idle_seconds)
+        avail[0] = 40.0
+
+    monkeypatch.setattr(gate, "unload_idle_models", _unload)
+    gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo")
+    assert freed, "eviction was never attempted before deciding"
+    assert freed[0] >= 1.0, (
+        "an idle threshold near zero pulls a model that is merely between its "
+        "own requests")
+
+
+# ---------------------------------------------------------------------------
+# wiring — the gate has to be ON the load paths, not merely importable
+# ---------------------------------------------------------------------------
+
+def _src(rel):
+    import io
+    import os
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    return io.open(os.path.join(root, rel), encoding="utf-8").read()
+
+
+def test_every_vram_admission_point_also_admits_on_host_ram():
+    """The two checks answer the same question about different memory. A load
+    path that reserves VRAM and not host RAM is exactly the gap that let the
+    kernel decide."""
+    src = _src("codai/models/manager.py")
+    body = src.split("def request_model(", 1)[1] if "def request_model(" in src else src
+    assert body.count("self.ensure_vram_for(") == body.count("self.ensure_host_ram_for(")
+
+
+def test_the_image_path_admits_on_host_ram():
+    """It does its own eviction instead of going through request_model, so it
+    needs its own call — and it is the path the failing cards are on."""
+    assert "ensure_host_ram_for(" in _src("codai/api/images.py")
+
+
+def test_the_refusal_is_reported_as_503_and_not_a_500():
+    app_src = _src("codai/api/app.py")
+    assert "HostRamExhausted" in app_src
+    assert "status_code=503" in app_src.split("HostRamExhausted", 1)[1]
+
+
+def test_the_setting_reaches_the_running_server():
+    """A config field nothing plumbs is a field that does nothing: the cap it
+    guards is read live from global args."""
+    assert "ram_admission" in _src("codai/config.py")
+    assert "global_args.ram_admission" in _src("codai/main.py")
+    assert "ram_admission" in _src("codai/admin/routes.py")
+
+
+# ---------------------------------------------------------------------------
+# the auto-offload sizing step that was skipping itself
+# ---------------------------------------------------------------------------
+
+def test_the_kv_types_are_read_before_the_offload_sizing_uses_them():
+    """The regression that took the host down three times in five hours.
+
+    ``_kv_quantized = bool(_ck or _cv)`` sat in the auto-offload sizing block,
+    while ``_ck``/``_cv`` were bound ~110 lines LATER, where llama_kwargs is
+    assembled. Reading them first raised UnboundLocalError on every load that
+    reached the branch; the handler printed "(auto-offload sizing skipped: …)",
+    which reads like a decision; nothing reduced n_gpu_layers; and llama.cpp
+    answered a 35B that did not fit with a 43,562 MiB CPU KV buffer.
+
+    Asserted on source ORDER rather than line numbers, so moving the block does
+    not break the test -- only reintroducing the bug does.
+    """
+    src = _src("codai/backends/vulkan.py")
+    use = src.index("_kv_quantized = bool(_ck or _cv)")
+    first_bind = min(src.index("_ck = kwargs.get("),
+                     src.index("_cv = kwargs.get("))
+    assert first_bind < use, (
+        "_ck/_cv are used by the auto-offload sizing before they are bound — "
+        "UnboundLocalError, swallowed as 'sizing skipped', model loads unsized")
+
+
+def test_a_failed_sizing_step_does_not_read_as_a_completed_one():
+    """'skipped' invites the reader to assume nothing needed doing. The message
+    has to say the load is unsized, or the next person reads past it again."""
+    src = _src("codai/backends/vulkan.py")
+    assert "auto-offload sizing FAILED" in src
+    assert "this load is UNSIZED" in src
+    # And the consequence has to be in the printed line itself, not only in a
+    # comment next to it: a print is what the operator sees.
+    printed = [ln for ln in src.splitlines()
+               if "auto-offload sizing FAILED" in ln and not ln.strip().startswith("#")]
+    assert printed, "the warning is not in a print, only in a comment"

tests/test_remote_code_allowlist.py

@@ -129,7 +129,11 @@ def test_a_permission_error_is_not_swallowed_by_the_loader():
     a refusal must not be one of them."""
     src = open("codai/api/images.py").read()
     head = src.index("_img_remote = {}")
-    body = src[head:head + 900]
+    # Bounded by the next statement after the block, not by a character count:
+    # a fixed window silently stops covering the code it was written for the
+    # moment anything is added inside it, which is how this test broke when the
+    # declared-class check went in.
+    body = src[head:src.index("_img_comp_models", head)]
     assert "except PermissionError:" in body and "raise" in body
 
 
@@ -147,3 +151,64 @@ def test_such_a_model_is_not_tried_as_sdxl_first():
 
 def test_the_field_is_savable_from_the_admin_page():
     assert '"custom_pipeline",' in open("codai/admin/routes.py").read()
+
+
+# ---------------------------------------------------------------------------
+# A repo that declares its own class without any card saying so
+# ---------------------------------------------------------------------------
+
+def test_a_repo_declaring_an_unavailable_class_is_detected(monkeypatch):
+    """MOSS-Image and Kromoss, exactly as they failed: no card named a
+    custom_pipeline, so nothing fired, and the load died on
+    'module diffusers has no attribute MOSSImagePipeline'."""
+    from codai.models import remote_code as R
+    monkeypatch.setattr(R, "declared_pipeline_class",
+                        lambda repo: "MOSSImagePipeline")
+    kw = R.kwargs_for_declared_class(_cfg(["openmoss-team/*"]),
+                                     "OpenMOSS-Team/MOSS-Image-SFT")
+    assert kw["trust_remote_code"] is True
+    assert kw["custom_pipeline"] == "OpenMOSS-Team/MOSS-Image-SFT"
+    assert kw["_declared_class"] == "MOSSImagePipeline"
+
+
+def test_such_a_repo_is_refused_by_name_when_not_allow_listed(monkeypatch):
+    """The refusal has to name the setting — that is the whole reason the
+    allow-list is spelled out rather than implied."""
+    from codai.models import remote_code as R
+    monkeypatch.setattr(R, "declared_pipeline_class",
+                        lambda repo: "MOSSImagePipeline")
+    with pytest.raises(PermissionError) as ei:
+        R.kwargs_for_declared_class(_cfg([]), "lodestones/Kromoss")
+    msg = str(ei.value)
+    assert "models.trust_remote_code" in msg
+    assert "lodestones/*" in msg
+    assert "MOSSImagePipeline" in msg
+
+
+def test_an_ordinary_repo_pays_nothing(monkeypatch):
+    """The class diffusers already has must not trigger any of this: every
+    normal image model goes through this code path."""
+    from codai.models import remote_code as R
+    monkeypatch.setattr(R, "declared_pipeline_class",
+                        lambda repo: "StableDiffusionXLPipeline")
+    assert R.kwargs_for_declared_class(_cfg([]), "any/sdxl") == {}
+
+
+def test_a_repo_that_declares_nothing_is_left_alone(monkeypatch):
+    from codai.models import remote_code as R
+    monkeypatch.setattr(R, "declared_pipeline_class", lambda repo: None)
+    assert R.kwargs_for_declared_class(_cfg([]), "any/model") == {}
+
+
+def test_the_marker_never_reaches_from_pretrained():
+    """_declared_class is ours, for the log line. Passing it on would be a
+    TypeError from diffusers at the worst possible moment."""
+    src = open("codai/api/images.py").read()
+    head = src.index("_img_remote = {}")
+    body = src[head:src.index("_img_comp_models", head)]
+    assert '_img_remote.pop("_declared_class"' in body
+
+
+def test_the_loader_asks_the_repository_when_no_card_did():
+    src = open("codai/api/images.py").read()
+    assert "kwargs_for_declared_class(" in src