Skip to content

C
coderai
Commit fdbc7701 authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files
Options

refuse a load that cannot fit in host RAM, and size the one that was skipping itself 

A load is admitted to VRAM by estimate: ensure_vram_for sizes the model, evicts
what is in the way, and failing that a CUDA OOM is caught and reported. Host RAM
had no equivalent, and the kernel supplies its own: four engines SIGKILLed in
five hours on a 54 GB host, each one while a model staged its weights.

    Oct 29 23:03  Killed coderai-nvidia-gguf   anon-rss 46.3 GB
    Oct 30 03:10  Killed coderai-nvidia        anon-rss 47.7 GB
    Oct 30 03:36  Killed coderai-nvidia        anon-rss 31.6 GB
    Oct 30 03:49  Killed coderai-nvidia        anon-rss 32.7 GB
    front: engine#0 exited (code -9); respawning (1/5 in 120s)

The RAM watcher was awake for all of it and logged "RSS 50.6/43.0 GB (118%) --
mitigation [gc+trim+evict_idle] -> 53.7 GB": every rung of its ladder releases
memory already held, and none of them can decline the load doing the allocating.
SIGKILL then takes the engine rather than the request, so the client that asked
for the oversized model saw a dropped connection -- and so did every unrelated
request in flight, plus everything arriving during the 120-second respawn. One
bad load reads as the whole server being unreliable.

So offload.ram_admission (default on) adds the missing half. codai/models/
host_ram.py sizes a pending load -- measured host residency if the model has
loaded before, the whole footprint for an offload strategy that asks for it,
the spill past free VRAM otherwise, plus a staging allowance -- against the
tighter of MemAvailable and the headroom under max_ram_gb, less a floor kept
for the OS and the sibling engines. ensure_host_ram_for() runs the existing
pressure ladder FIRST and re-measures, so refusal is the last step; when it
comes, it is a 503 naming the model, the estimate, the budget, the shortfall
and what to change on the card. A model whose footprint cannot be determined
is never refused on a number nobody has.

And the proximate cause of the biggest of those kills was a sizing step that
reported itself as skipped:

    (auto-offload sizing skipped: cannot access local variable '_ck'
     where it is not associated with a value)

The GGUF auto-offload branch reads the configured KV-cache types to size its
headroom, from a binding made ~110 lines further down where llama_kwargs is
assembled. Every load that reached the branch raised UnboundLocalError, the
handler called it "skipped", and nothing reduced n_gpu_layers -- so llama.cpp
was asked to keep every layer of a 35B on a card that could not hold them, and
answered with a 43,562 MiB CPU KV buffer. The read is hoisted above its use
(tested on source order, not a line number), and the handler now says the load
is UNSIZED, because "skipped" invites the reader to assume nothing needed doing.

Separately, and found while checking which cards were really misconfigured:
a repo whose model_index.json declares a pipeline class diffusers does not have
is now detected and loaded with its own code when allow-listed, or refused by
name when not. The allow-list and the mechanism both already existed, but only
fired when a card named a custom_pipeline -- nothing ever did, so MOSS-Image
and Kromoss failed with "module diffusers has no attribute MOSSImagePipeline",
which names neither the cause nor the remedy and reads as a broken model.
Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
parent c6fca1d8
Show whitespace changes
Inline Side-by-side
Showing with 1088 additions and 4 deletions
codai/__init__.py
View file @ fdbc7701
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API # Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here. # metadata and the admin web UI read from here.
__version__ = "0.5.27" __version__ = "0.5.28"
__tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference." __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
# Configure the CUDA caching allocator BEFORE torch is imported anywhere. # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
......
codai/admin/routes.py
View file @ fdbc7701
...@@ -4032,6 +4032,8 @@ def build_settings_dict(c, gpu_cards): ...@@ -4032,6 +4032,8 @@ def build_settings_dict(c, gpu_cards):
"ram_watch_poll_seconds": c.offload.ram_watch_poll_seconds, "ram_watch_poll_seconds": c.offload.ram_watch_poll_seconds,
"ram_watch_soft_fraction": c.offload.ram_watch_soft_fraction, "ram_watch_soft_fraction": c.offload.ram_watch_soft_fraction,
"ram_watch_cuda": c.offload.ram_watch_cuda, "ram_watch_cuda": c.offload.ram_watch_cuda,
"ram_admission": c.offload.ram_admission,
"ram_admission_floor_gb": c.offload.ram_admission_floor_gb,
"gpu_split": c.offload.gpu_split, "gpu_split": c.offload.gpu_split,
"tensor_split": c.offload.tensor_split, "tensor_split": c.offload.tensor_split,
"split_strategy": c.offload.split_strategy, "split_strategy": c.offload.split_strategy,
...@@ -4517,6 +4519,10 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad ...@@ -4517,6 +4519,10 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
if "ram_watch_soft_fraction" in off: if "ram_watch_soft_fraction" in off:
c.offload.ram_watch_soft_fraction = float(off["ram_watch_soft_fraction"] or c.offload.ram_watch_soft_fraction) c.offload.ram_watch_soft_fraction = float(off["ram_watch_soft_fraction"] or c.offload.ram_watch_soft_fraction)
c.offload.ram_watch_cuda = bool(off.get("ram_watch_cuda", c.offload.ram_watch_cuda)) c.offload.ram_watch_cuda = bool(off.get("ram_watch_cuda", c.offload.ram_watch_cuda))
c.offload.ram_admission = bool(off.get("ram_admission", c.offload.ram_admission))
if "ram_admission_floor_gb" in off:
c.offload.ram_admission_floor_gb = float(
off["ram_admission_floor_gb"] or c.offload.ram_admission_floor_gb)
c.offload.gpu_split = bool(off.get("gpu_split", c.offload.gpu_split)) c.offload.gpu_split = bool(off.get("gpu_split", c.offload.gpu_split))
if "tensor_split" in off: if "tensor_split" in off:
c.offload.tensor_split = off["tensor_split"] or None c.offload.tensor_split = off["tensor_split"] or None
...@@ -4552,6 +4558,8 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad ...@@ -4552,6 +4558,8 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
ga.ram_watch_poll_seconds = c.offload.ram_watch_poll_seconds ga.ram_watch_poll_seconds = c.offload.ram_watch_poll_seconds
ga.ram_watch_soft_fraction = c.offload.ram_watch_soft_fraction ga.ram_watch_soft_fraction = c.offload.ram_watch_soft_fraction
ga.ram_watch_cuda = c.offload.ram_watch_cuda ga.ram_watch_cuda = c.offload.ram_watch_cuda
ga.ram_admission = c.offload.ram_admission
ga.ram_admission_floor_gb = c.offload.ram_admission_floor_gb
ga.gpu_split = c.offload.gpu_split ga.gpu_split = c.offload.gpu_split
ga.tensor_split = c.offload.tensor_split ga.tensor_split = c.offload.tensor_split
ga.split_strategy = c.offload.split_strategy ga.split_strategy = c.offload.split_strategy
......
codai/api/app.py
View file @ fdbc7701
...@@ -930,6 +930,36 @@ async def task_cancelled_handler(request: Request, exc: TaskCancelled): ...@@ -930,6 +930,36 @@ async def task_cancelled_handler(request: Request, exc: TaskCancelled):
return JSONResponse(status_code=499, content={"detail": "Task cancelled", "task_id": tid}) return JSONResponse(status_code=499, content={"detail": "Task cancelled", "task_id": tid})
from codai.models.host_ram import HostRamExhausted
@app.exception_handler(HostRamExhausted)
async def host_ram_exhausted_handler(request: Request, exc: HostRamExhausted):
"""A load was refused for want of host RAM.
503, because this is the server declining to start work it cannot finish,
and the condition is transient: the same request succeeds once a resident
model is evicted or a smaller build is asked for. Registered once here
rather than caught per route — the refusal can come from any path that
loads a model (chat, images, video, audio), and every one of them should
report the numbers instead of a bare 500.
Before this existed the request did not fail at all: the host ran out of
memory, the kernel killed the engine, and the client saw a dropped
connection while every other model's requests died with it.
"""
return JSONResponse(
status_code=503,
content={"error": {"message": exc.message(),
"type": "insufficient_host_memory",
"param": exc.model,
"code": "host_ram_exhausted",
"needed_gb": exc.need.gb,
"available_gb": exc.budget.gb,
"shortfall_gb": exc.shortfall_gb}},
)
@app.get("/v1/models", response_model=ModelList, summary="List available models", tags=["Core"]) @app.get("/v1/models", response_model=ModelList, summary="List available models", tags=["Core"])
async def list_models(): async def list_models():
"""List available models.""" """List available models."""
......
codai/api/images.py
View file @ fdbc7701
...@@ -784,9 +784,21 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict = ...@@ -784,9 +784,21 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
from codai.admin.routes import config_manager as _cm from codai.admin.routes import config_manager as _cm
_img_remote = remote_code.kwargs_for( _img_remote = remote_code.kwargs_for(
getattr(_cm, "config", None), model_name, _cfg('custom_pipeline')) getattr(_cm, "config", None), model_name, _cfg('custom_pipeline'))
if not _img_remote:
# No card named a custom_pipeline — ask the repository itself. A
# checkpoint whose model_index.json declares a class diffusers does
# not have is telling us it ships its own, and letting that load
# proceed produced "module diffusers has no attribute
# MOSSImagePipeline": a true statement that names neither the cause
# nor the fix, and reads as a broken model.
_img_remote = remote_code.kwargs_for_declared_class(
getattr(_cm, "config", None), model_name)
if _img_remote: if _img_remote:
_declared = _img_remote.pop("_declared_class", None)
print(f" Custom pipeline from the repository: " print(f" Custom pipeline from the repository: "
f"{_img_remote['custom_pipeline']} (allow-listed)") f"{_img_remote['custom_pipeline']} (allow-listed)"
+ (f" — it declares {_declared}, which is not in diffusers"
if _declared else ""))
except PermissionError: except PermissionError:
raise raise
except Exception as e: except Exception as e:
...@@ -907,6 +919,28 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict = ...@@ -907,6 +919,28 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
except Exception as _ee: except Exception as _ee:
print(f" Proactive eviction skipped: {_ee}") print(f" Proactive eviction skipped: {_ee}")
# =====================================================================
# Host-RAM admission
# =====================================================================
# VRAM is handled above by eviction; host RAM has to be handled by refusal,
# because the kernel's answer to running out of it is to kill the engine. The
# image path is where it bit hardest: the cards that quantize only their text
# encoder stage a bf16 transformer into host RAM, and on a full card nearly
# all of it spills. Raised, not caught — the caller turns it into a 503 that
# names the model, the size and the fix. See codai/models/host_ram.py.
try:
from codai.models.manager import multi_model_manager as _mmm
except Exception:
_mmm = None
if _mmm is not None:
_hkey = None
for _k in (f"image:{model_name}", model_name):
if _k in _mmm.config:
_hkey = _k
break
_mmm.ensure_host_ram_for(_hkey or f"image:{model_name}", model_name,
_requested_id)
# ===================================================================== # =====================================================================
# Standard loading path (with OOM fallback) # Standard loading path (with OOM fallback)
# ===================================================================== # =====================================================================
......
codai/backends/vulkan.py
View file @ fdbc7701
...@@ -1240,6 +1240,19 @@ class VulkanBackend(ModelBackend): ...@@ -1240,6 +1240,19 @@ class VulkanBackend(ModelBackend):
try: try:
_exp = kwargs.get('expected_vram_gb') _exp = kwargs.get('expected_vram_gb')
_nlayers = _gguf_block_count(model_path) _nlayers = _gguf_block_count(model_path)
# Read the KV-cache types HERE, not from the binding made further
# down when llama_kwargs is assembled. That binding happens after
# this block, so reading it here raised UnboundLocalError on every
# load that reached the offload branch -- caught below and printed
# as "(auto-offload sizing skipped: cannot access local variable
# '_ck' …)". Nothing then reduced n_gpu_layers, so a 35B that did
# not fit kept every layer on the GPU and llama.cpp answered with a
# 43,562 MiB CPU KV buffer: the host ran out of memory and the
# kernel killed the engine (code -9), three times in five hours.
# A skipped sizing step must not read as a sized one.
_kv_raw_cfg = kwargs.get('_raw_cfg') or {}
_ck = kwargs.get('cache_type_k', _kv_raw_cfg.get('cache_type_k'))
_cv = kwargs.get('cache_type_v', _kv_raw_cfg.get('cache_type_v'))
# Usable VRAM is the POOL across the cards this engine can use, not # Usable VRAM is the POOL across the cards this engine can use, not
# just main_gpu: all same-backend cards always (e.g. both 3090s), plus # just main_gpu: all same-backend cards always (e.g. both 3090s), plus
# the other backend's cards when cross-split is on (3090 + RX 580). # the other backend's cards when cross-split is on (3090 + RX 580).
...@@ -1316,7 +1329,17 @@ class VulkanBackend(ModelBackend): ...@@ -1316,7 +1329,17 @@ class VulkanBackend(ModelBackend):
f"GPU, {_nlayers - _fit} on CPU (slower). Lower n_ctx or use a " f"GPU, {_nlayers - _fit} on CPU (slower). Lower n_ctx or use a "
f"smaller model to keep it fully on GPU.", flush=True) f"smaller model to keep it fully on GPU.", flush=True)
except Exception as _off_e: except Exception as _off_e:
print(f" (auto-offload sizing skipped: {_off_e})", flush=True) # Loud on purpose, and it says what the consequence is. This
# printed "(auto-offload sizing skipped: …)" for an
# UnboundLocalError in its own code, which reads like a decision
# not to offload rather than a failure to decide -- and the model
# that was not sized then took the host down with a 43 GB CPU KV
# buffer. A sizing step that did not run leaves the load UNSIZED,
# so say that, with the error.
print(f" WARNING: auto-offload sizing FAILED ({type(_off_e).__name__}: "
f"{_off_e}) — this load is UNSIZED: every layer stays on the "
f"GPU and llama.cpp may put the KV cache in host RAM. Check "
f"n_gpu_layers / n_ctx for this model.", flush=True)
# Configure context size # Configure context size
if no_ram: if no_ram:
......
codai/config.py
View file @ fdbc7701
...@@ -292,6 +292,14 @@ class OffloadConfig: ...@@ -292,6 +292,14 @@ class OffloadConfig:
ram_watch_poll_seconds: float = 15.0 # how often the watcher samples RSS ram_watch_poll_seconds: float = 15.0 # how often the watcher samples RSS
ram_watch_soft_fraction: float = 0.90 # mitigate at/above this fraction of the cap ram_watch_soft_fraction: float = 0.90 # mitigate at/above this fraction of the cap
ram_watch_cuda: bool = True # allow mitigation to call CUDA empty_cache() ram_watch_cuda: bool = True # allow mitigation to call CUDA empty_cache()
# Host-RAM ADMISSION: refuse a load whose estimated host-RAM cost does not
# fit, instead of letting it run the host out of memory. The watcher above
# can only release memory already held, so it cannot stop an unquantized
# card mid-load -- on a 54 GB host that cost four OOM-killed engines in five
# hours, each reported to clients as "crashes the engine". See
# codai/models/host_ram.py. Off means loads proceed as they did before.
ram_admission: bool = True
ram_admission_floor_gb: float = 4.0 # host RAM kept free for OS + engines
# Unload a model that has gone untouched for this long, in seconds (0 = never). # Unload a model that has gone untouched for this long, in seconds (0 = never).
# #
# Eviction already frees a model when something else needs the memory, so this # Eviction already frees a model when something else needs the memory, so this
...@@ -2213,6 +2221,8 @@ class ConfigManager: ...@@ -2213,6 +2221,8 @@ class ConfigManager:
"ram_watch_poll_seconds": self.config.offload.ram_watch_poll_seconds, "ram_watch_poll_seconds": self.config.offload.ram_watch_poll_seconds,
"ram_watch_soft_fraction": self.config.offload.ram_watch_soft_fraction, "ram_watch_soft_fraction": self.config.offload.ram_watch_soft_fraction,
"ram_watch_cuda": self.config.offload.ram_watch_cuda, "ram_watch_cuda": self.config.offload.ram_watch_cuda,
"ram_admission": self.config.offload.ram_admission,
"ram_admission_floor_gb": self.config.offload.ram_admission_floor_gb,
"gpu_split": self.config.offload.gpu_split, "gpu_split": self.config.offload.gpu_split,
"tensor_split": self.config.offload.tensor_split, "tensor_split": self.config.offload.tensor_split,
"split_strategy": self.config.offload.split_strategy, "split_strategy": self.config.offload.split_strategy,
......
codai/main.py
View file @ fdbc7701
...@@ -1221,6 +1221,10 @@ def main(): ...@@ -1221,6 +1221,10 @@ def main():
global_args.ram_watch_poll_seconds = config.offload.ram_watch_poll_seconds global_args.ram_watch_poll_seconds = config.offload.ram_watch_poll_seconds
global_args.ram_watch_soft_fraction = config.offload.ram_watch_soft_fraction global_args.ram_watch_soft_fraction = config.offload.ram_watch_soft_fraction
global_args.ram_watch_cuda = config.offload.ram_watch_cuda global_args.ram_watch_cuda = config.offload.ram_watch_cuda
# Host-RAM admission: refuse a load that cannot fit rather than let the
# kernel's OOM killer take the engine (codai/models/host_ram.py).
global_args.ram_admission = config.offload.ram_admission
global_args.ram_admission_floor_gb = config.offload.ram_admission_floor_gb
# Cross-backend GPU pooling defaults (per-model config can override). Read by # Cross-backend GPU pooling defaults (per-model config can override). Read by
# the GGUF loader to set llama.cpp split_mode/tensor_split. # the GGUF loader to set llama.cpp split_mode/tensor_split.
global_args.gpu_split = getattr(config.offload, "gpu_split", False) global_args.gpu_split = getattr(config.offload, "gpu_split", False)
......
codai/models/host_ram.py 0 → 100644
View file @ fdbc7701
# CoderAI - OpenAI-compatible API server
# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
"""Host-RAM admission control for model loads.
A load is admitted to VRAM by estimate: :meth:`ensure_vram_for` sizes the model,
evicts what is in the way, and the load proceeds. Host RAM had no equivalent, and
the asymmetry is not cosmetic -- it is the difference between a clean refusal and
a dead engine.
What it cost, measured on a 54 GB production host serving four engines
(2026-10-29/30). Six model cards configured their big weights unquantized:
``load_in_4bit=False`` with ``component_quantization={'text_encoder': '4bit'}``,
which quantizes the text encoder and leaves the transformer at bf16; two video
cards asked for ``offload_strategy`` ``model``/``balanced``, which parks weights
on the host by definition. Requesting any of them staged tens of GB into host
RAM, and the kernel resolved it:
Oct 29 23:03 Killed coderai-nvidia-gguf anon-rss 46.3 GB
Oct 30 03:10 Killed coderai-nvidia anon-rss 47.7 GB
Oct 30 03:36 Killed coderai-nvidia anon-rss 31.6 GB
Oct 30 03:49 Killed coderai-nvidia anon-rss 32.7 GB
front: engine#0 exited (code -9); respawning (1/5 in 120s)
The RAM watcher saw it and could not help: it logged ``RSS 50.6/43.0 GB (118%)
-- mitigation [gc+trim+evict_idle] -> 53.7 GB``. Its ladder can only release
memory that is already held; nothing in it can decline the load that is at that
moment streaming bf16 weights into the host. So every client saw SIGKILL --
reported as "crashes the engine", "won't load", and, for whatever else was in
flight during the 120-second respawn, as an unrelated failure.
This module is the missing half: size the host-RAM cost of a load BEFORE it
starts, and if it cannot fit, refuse it by name with the remedy attached. One
model then fails with an explanation instead of every model failing with a
corpse.
Deliberately pure arithmetic -- no manager, no torch, no /proc. The caller
measures and evicts; these functions only decide. That keeps the decision
testable without a GPU, which is the half that was never exercised before.
"""
from __future__ import annotations
from typing import NamedTuple, Optional
# Head-room for the load itself, on top of whatever the weights park on the host.
# A shard-streaming load holds a shard buffer, the safetensors read window and the
# dequantisation scratch at once; measured between 1.2 and 1.8 GB across the HF and
# diffusers paths, so 2.0 GB covers it without pretending to be precise.
STAGING_GB = 2.0
# Host RAM never promised to a model, so the OS, the front and the sibling engines
# keep breathing room even when a load is admitted right up to the line.
DEFAULT_FLOOR_GB = 4.0
# Strategies that put the weights on the HOST by design, rather than as overflow.
# For these the host cost is the model's whole footprint, not the part that missed
# the GPU: that is what the operator asked for.
HOST_RESIDENT_STRATEGIES = frozenset({"model", "sequential", "cpu"})
# Strategies that keep every weight on the GPU, so a model too big for the card
# fails on VRAM (loudly, recoverably) and never reaches host RAM at all.
GPU_ONLY_STRATEGIES = frozenset({"none"})
class Need(NamedTuple):
"""The estimated host-RAM cost of a pending load, and where it came from."""
gb: float
basis: str # "measured" | "host-resident" | "gpu-only" | "spill" | "unknown"
detail: str # one human-readable line, for the log and the refusal
class Budget(NamedTuple):
"""How much host RAM a load may take right now, and what limits it."""
gb: float
binding: str # "kernel" | "cap" | "floor"
detail: str
class Verdict(NamedTuple):
ok: bool
shortfall_gb: float
class HostRamExhausted(RuntimeError):
"""A load was refused because its host-RAM cost does not fit.
Carries the numbers so an API layer can report them without re-deriving
anything, and so a test can assert on them rather than on prose.
"""
def __init__(self, model: str, need: Need, budget: Budget, remedy: str = ""):
self.model = model
self.need = need
self.budget = budget
self.remedy = remedy
self.shortfall_gb = round(max(0.0, need.gb - budget.gb), 2)
super().__init__(self.message())
def message(self) -> str:
msg = (f"'{self.model}' needs about {self.need.gb:.1f} GB of host RAM to "
f"load ({self.need.detail}), and only {self.budget.gb:.1f} GB is "
f"available ({self.budget.detail}) — short by "
f"{self.shortfall_gb:.1f} GB. Refusing the load: going ahead would "
f"exhaust host memory and the kernel would kill the engine, which "
f"fails every other request too.")
if self.remedy:
msg += " " + self.remedy
return msg
def estimate_need(*, total_need_gb: float, free_vram_gb: float,
measured_ram_gb: Optional[float] = None,
offload_strategy: Optional[str] = None,
no_ram: bool = False,
staging_gb: float = STAGING_GB) -> Need:
"""Host RAM a pending load will want, in GB.
``total_need_gb`` is the model's FULL footprint -- the placement-independent
figure ``measured_vram_gb`` already records -- not the GPU-resident slice.
The estimate follows what actually determines the cost:
* ``measured_ram_gb`` wins outright. The loader records what a load really
parked on the host (``of which X GB was offloaded to host RAM this load``),
so once a model has loaded once there is no need to guess.
* A host-resident offload strategy means the whole footprint lands on the
host, because that is the point of asking for it.
* A GPU-only strategy means none of it does: too big a model fails on VRAM.
* Otherwise the host carries the SPILL -- whatever did not fit on the card.
This is the case that killed the engine: a bf16 transformer on a card with
other models resident spills nearly all of itself.
An unknown footprint yields the staging allowance alone, flagged ``unknown``:
a load whose size cannot be determined is never refused on a number nobody
has, it just loses the protection.
"""
staging = max(0.0, float(staging_gb or 0.0))
strategy = (offload_strategy or "").strip().lower()
if measured_ram_gb is not None and float(measured_ram_gb) > 0:
held = float(measured_ram_gb)
return Need(round(held + staging, 2), "measured",
f"{held:.1f} GB measured on host RAM on a previous load, "
f"plus {staging:.1f} GB to stage it")
if no_ram or strategy in GPU_ONLY_STRATEGIES:
return Need(round(staging, 2), "gpu-only",
f"loads GPU-only, so only the {staging:.1f} GB load buffer "
f"touches host RAM")
total = max(0.0, float(total_need_gb or 0.0))
if total <= 0:
return Need(round(staging, 2), "unknown",
"footprint unknown, so only the load buffer is accounted for")
if strategy in HOST_RESIDENT_STRATEGIES:
return Need(round(total + staging, 2), "host-resident",
f"offload_strategy='{strategy}' keeps the whole "
f"{total:.1f} GB footprint on the host, plus "
f"{staging:.1f} GB to stage it")
spill = max(0.0, total - max(0.0, float(free_vram_gb or 0.0)))
return Need(round(spill + staging, 2), "spill",
f"{total:.1f} GB footprint against {max(0.0, float(free_vram_gb or 0.0)):.1f} GB "
f"free VRAM spills {spill:.1f} GB to the host, plus "
f"{staging:.1f} GB to stage it")
def budget(*, kernel_available_gb: float, cap_gb: Optional[float] = None,
fleet_held_gb: float = 0.0,
floor_gb: float = DEFAULT_FLOOR_GB) -> Budget:
"""Host RAM a load may take right now, in GB.
Two limits apply and the tighter one binds:
* what the kernel says can be allocated (``MemAvailable``, which already
counts the page cache it is willing to drop), and
* the configured server-wide ceiling minus what the fleet already holds.
``floor_gb`` is then held back from whichever bound won, so admitting a load
at the limit still leaves the OS and the sibling engines room to run. The
result is clamped at zero: a fleet already over its cap has no budget, which
is the correct answer rather than a negative one.
"""
floor = max(0.0, float(floor_gb or 0.0))
kernel = max(0.0, float(kernel_available_gb or 0.0))
limit, binding, detail = kernel, "kernel", f"{kernel:.1f} GB available on the host"
if cap_gb and float(cap_gb) > 0:
headroom = float(cap_gb) - max(0.0, float(fleet_held_gb or 0.0))
if headroom < limit:
limit, binding = headroom, "cap"
detail = (f"{headroom:.1f} GB left under the {float(cap_gb):.0f} GB "
f"server RAM cap, with the fleet holding "
f"{max(0.0, float(fleet_held_gb or 0.0)):.1f} GB")
usable = limit - floor
if usable <= 0:
return Budget(0.0, "floor",
detail + f", all of it inside the {floor:.1f} GB kept free "
f"for the OS and the other engines")
return Budget(round(usable, 2), binding,
detail + f", less {floor:.1f} GB kept free for the OS and the "
f"other engines")
def verdict(need: Need, bud: Budget) -> Verdict:
"""Whether ``need`` fits in ``bud``."""
if need.gb <= bud.gb:
return Verdict(True, 0.0)
return Verdict(False, round(need.gb - bud.gb, 2))
def remedy_for(cfg: Optional[dict], basis: str = "") -> str:
"""One actionable sentence about what to change on the card, or "".
Derived from the card itself, because the fix differs: an unquantized
transformer wants quantizing, a host-resident offload strategy wants
reconsidering, and a model that simply does not fit wants a smaller build.
"""
cfg = cfg or {}
comp = cfg.get("component_quantization") or {}
if isinstance(comp, dict) and comp and not any(
k in comp for k in ("transformer", "transformer_2", "unet")):
covered = ", ".join(sorted(comp)) or "nothing"
return ("This card quantizes only " + covered + ", so its transformer "
"loads at full precision — add 'transformer' to "
"component_quantization (or request the pre-quantized build of "
"this model) and it will fit.")
strategy = str(cfg.get("offload_strategy") or "").strip().lower()
if strategy in HOST_RESIDENT_STRATEGIES:
return (f"This card sets offload_strategy='{strategy}', which holds the "
f"weights in host RAM by design — quantize it or give it a "
f"strategy that keeps them on the GPU.")
if basis == "spill" and not (cfg.get("load_in_4bit") or cfg.get("load_in_8bit")):
return ("This card has no quantization set, so its weights load at full "
"precision — quantize it, or request a pre-quantized build.")
return ""
codai/models/manager.py
View file @ fdbc7701
...@@ -2219,6 +2219,10 @@ class MultiModelManager: ...@@ -2219,6 +2219,10 @@ class MultiModelManager:
self.ensure_vram_for(model_name, requested_id=requested_id) self.ensure_vram_for(model_name, requested_id=requested_id)
except Exception as _ev_e: except Exception as _ev_e:
print(f" (ensure_vram_for warning: {_ev_e})") print(f" (ensure_vram_for warning: {_ev_e})")
# Outside that except on purpose: a VRAM reservation failing
# is a warning (the load can still work), while a host-RAM
# refusal is the decision itself and must reach the caller.
self.ensure_host_ram_for(model_name, requested_id=requested_id)
_snap = self.vram_before_load() _snap = self.vram_before_load()
_ram_snap = self._get_own_ram_gb() _ram_snap = self._get_own_ram_gb()
# Tell the backend how much VRAM this model is expected to need so # Tell the backend how much VRAM this model is expected to need so
...@@ -6019,6 +6023,130 @@ class MultiModelManager: ...@@ -6019,6 +6023,130 @@ class MultiModelManager:
f"have {free_gb:.1f} GB free — evicting models to make room") f"have {free_gb:.1f} GB free — evicting models to make room")
self._evict_models_for_vram(needed_gb) self._evict_models_for_vram(needed_gb)
@staticmethod
def _ram_admission_enabled() -> bool:
"""Whether a load may be refused for want of host RAM (default True)."""
try:
from codai.api.state import get_global_args
ga = get_global_args()
return bool(getattr(ga, 'ram_admission', True)) if ga else True
except Exception:
return True
@staticmethod
def _ram_admission_floor_gb() -> float:
"""Host RAM never promised to a load, in GB."""
from codai.models import host_ram
try:
from codai.api.state import get_global_args
ga = get_global_args()
val = getattr(ga, 'ram_admission_floor_gb', None) if ga else None
return float(val) if val is not None else host_ram.DEFAULT_FLOOR_GB
except Exception:
return host_ram.DEFAULT_FLOOR_GB
def _host_ram_budget(self):
"""The host-RAM budget a load may take right now."""
from codai.models import host_ram, proc_mem
return host_ram.budget(
kernel_available_gb=proc_mem.host_available_gb(),
cap_gb=self._ram_cap_gb(),
fleet_held_gb=self._get_process_ram_gb(),
floor_gb=self._ram_admission_floor_gb(),
)
def ensure_host_ram_for(self, model_key: str, resolved_name: str = None,
requested_id: str = None, cfg: dict = None) -> None:
"""Refuse a load that cannot fit in host RAM, before it starts.
The counterpart of :meth:`ensure_vram_for`, and the reason it exists is
that it had none. VRAM pressure is answered by eviction and, failing
that, by a CUDA OOM the server catches and reports. Host-RAM pressure is
answered by the kernel's OOM killer, which takes the whole engine with
it: four times in five hours on a 54 GB host the night this was written,
each time while an unquantized card staged its weights, each time
reported to the client as "crashes the engine" (see
:mod:`codai.models.host_ram`).
So the ladder here ends differently. Make room first -- trim, then evict
idle models, exactly as RAM pressure is handled elsewhere -- and if the
load still does not fit, raise :class:`~codai.models.host_ram.HostRamExhausted`.
One model then fails with its size, the budget and the fix, while every
other model keeps serving.
Never refuses a model that is already resident (it is not about to
allocate anything), and never refuses on a footprint it could not
determine.
"""
from codai.models import host_ram
if not self._ram_admission_enabled():
return
for _k in (model_key, resolved_name, requested_id):
if _k and _k in self.models:
return
if cfg is None:
cfg = {}
for _k in (model_key, resolved_name, requested_id):
if _k and _k in self.config:
cfg = self.config.get(_k) or {}
break
try:
_measured = cfg.get("measured_ram_gb")
_measured = float(_measured) if _measured else None
except (TypeError, ValueError):
_measured = None
need = host_ram.estimate_need(
total_need_gb=self._get_model_used_vram_gb(
model_key, resolved_name, requested_id),
free_vram_gb=self._get_free_vram_gb(),
measured_ram_gb=_measured,
offload_strategy=cfg.get("offload_strategy"),
no_ram=bool(cfg.get("no_ram")),
)
bud = self._host_ram_budget()
if host_ram.verdict(need, bud).ok:
return
if need.basis == "unknown":
# Nothing was ever measured or configured for this model, so the
# only figure on the table is the load buffer — a guess about every
# model rather than a fact about this one. Refusing on it would
# block a 300 MB embedder on a tight host for no reason. Say that
# the protection does not cover this load, and let it proceed.
print(f"Loading '{model_key}': host-RAM admission has no footprint "
f"for this model, so it cannot be checked — proceeding "
f"unguarded (budget {bud.gb:.1f} GB; {bud.detail})")
return
print(f"Loading '{model_key}': needs ~{need.gb:.1f} GB host RAM "
f"({need.detail}), budget is {bud.gb:.1f} GB ({bud.detail}) — "
f"freeing host RAM before deciding")
try:
_trim_cpu_ram()
except Exception:
pass
if self._evict_idle_on_ram_enabled():
_cap = self._ram_cap_gb()
if _cap:
self._evict_models_for_ram(_cap * 0.85)
else:
# No cap configured, so there is no target to evict towards --
# free what has gone quiet and re-measure against the kernel's
# figure. 30 s rather than "any idle model": a model being used
# in a request loop is idle between its own requests, and pulling
# it to admit another load just trades one reload for another.
self.unload_idle_models(30.0)
bud = self._host_ram_budget()
if host_ram.verdict(need, bud).ok:
print(f" host RAM freed — {bud.gb:.1f} GB now available, load admitted")
return
raise host_ram.HostRamExhausted(
str(requested_id or resolved_name or model_key), need, bud,
host_ram.remedy_for(cfg, need.basis))
def request_model(self, requested_model: str, model_type: str = None, def request_model(self, requested_model: str, model_type: str = None,
extra_vram_gb: float = 0.0) -> Dict[str, Any]: extra_vram_gb: float = 0.0) -> Dict[str, Any]:
""" """
...@@ -6343,6 +6471,7 @@ class MultiModelManager: ...@@ -6343,6 +6471,7 @@ class MultiModelManager:
# Not loaded — make room in VRAM (evict other models as needed). # Not loaded — make room in VRAM (evict other models as needed).
self.ensure_vram_for(model_key, resolved_name, extra_vram_gb, self.ensure_vram_for(model_key, resolved_name, extra_vram_gb,
requested_id=_requested_id) requested_id=_requested_id)
self.ensure_host_ram_for(model_key, resolved_name, _requested_id)
return { return {
'model_key': model_key, 'model_key': model_key,
'model_name': resolved_name, 'model_name': resolved_name,
...@@ -6368,6 +6497,7 @@ class MultiModelManager: ...@@ -6368,6 +6497,7 @@ class MultiModelManager:
# model). Make room again so it returns to VRAM instead of CPU. # model). Make room again so it returns to VRAM instead of CPU.
self.ensure_vram_for(model_key, resolved_name, extra_vram_gb, self.ensure_vram_for(model_key, resolved_name, extra_vram_gb,
requested_id=_requested_id) requested_id=_requested_id)
self.ensure_host_ram_for(model_key, resolved_name, _requested_id)
return { return {
'model_key': model_key, 'model_key': model_key,
'model_name': resolved_name, 'model_name': resolved_name,
...@@ -6395,6 +6525,7 @@ class MultiModelManager: ...@@ -6395,6 +6525,7 @@ class MultiModelManager:
print(f"Loadall mode: Model '{model_key}' not pre-loaded, will load now") print(f"Loadall mode: Model '{model_key}' not pre-loaded, will load now")
self.ensure_vram_for(model_key, resolved_name, extra_vram_gb, self.ensure_vram_for(model_key, resolved_name, extra_vram_gb,
requested_id=_requested_id) requested_id=_requested_id)
self.ensure_host_ram_for(model_key, resolved_name, _requested_id)
return { return {
'model_key': model_key, 'model_key': model_key,
'model_name': resolved_name, 'model_name': resolved_name,
......
codai/models/remote_code.py
View file @ fdbc7701
...@@ -90,3 +90,65 @@ def kwargs_for(cfg, repo: str, custom_pipeline: Optional[str]) -> dict: ...@@ -90,3 +90,65 @@ def kwargs_for(cfg, repo: str, custom_pipeline: Optional[str]) -> dict:
return {} return {}
require(cfg, repo, f"its own pipeline module {name!r}") require(cfg, repo, f"its own pipeline module {name!r}")
return {"custom_pipeline": name, "trust_remote_code": True} return {"custom_pipeline": name, "trust_remote_code": True}
def declared_pipeline_class(repo: str) -> Optional[str]:
"""The ``_class_name`` the repo's ``model_index.json`` declares, or None.
Read rather than guessed, because the name is the only thing that says a
repository means to supply its own pipeline: MOSS-Image declares
``MOSSImagePipeline``, which exists in the repo and nowhere in diffusers.
"""
import json
import os
try:
if os.path.isdir(repo):
path = os.path.join(repo, "model_index.json")
if not os.path.isfile(path):
return None
else:
from huggingface_hub import hf_hub_download
# Fetched, not read cache-only: a repo being loaded for the first
# time has no cache yet, and a cache-only answer to "what class does
# this declare" is permanently wrong on a freshly-rented pod. The
# file is a few hundred bytes and the weights are about to follow.
path = hf_hub_download(repo, "model_index.json")
with open(path) as fh:
name = (json.load(fh) or {}).get("_class_name")
return str(name) if name else None
except Exception:
# Not a diffusers repo, not reachable, or no index. Either way there is
# nothing to declare, and the ordinary load must be left alone.
return None
def kwargs_for_declared_class(cfg, repo: str) -> dict:
"""``from_pretrained`` kwargs for a repo whose pipeline class is its own.
The companion to :func:`kwargs_for`, which only fires when a card already
names a ``custom_pipeline``. Nothing filled that field in for MOSS-Image or
Kromoss, so the load went ahead as an ordinary repo load and diffusers
reported ``module diffusers has no attribute MOSSImagePipeline`` -- true,
and no help at all: it names neither the cause (the class lives in the
repository) nor the remedy (allow-list it). The operator reads it as a
broken model.
So detect the case from the repo's own declaration. Allow-listed, it loads
with its code; not allow-listed, it is refused by name with the setting to
change, which is this module's whole policy.
Empty dict when the declared class is available locally -- the overwhelming
majority -- so an ordinary load never pays for this.
"""
cls = declared_pipeline_class(repo)
if not cls:
return {}
try:
import diffusers
if getattr(diffusers, cls, None) is not None:
return {}
except Exception:
return {}
require(cfg, repo, f"its own pipeline class {cls!r}")
return {"custom_pipeline": repo, "trust_remote_code": True,
"_declared_class": cls}
docs/memory-accounting.md
View file @ fdbc7701
...@@ -112,6 +112,84 @@ cache that the kernel reclaims by itself and that evicting a model cannot free. ...@@ -112,6 +112,84 @@ cache that the kernel reclaims by itself and that evicting a model cannot free.
The cap can only be met by releasing memory something actually holds, so the The cap can only be met by releasing memory something actually holds, so the
metric has to be one that releasing memory moves. metric has to be one that releasing memory moves.
## A load that cannot fit in host RAM is refused
The cap above is a *pressure* mechanism: a watcher samples the fleet's memory
and, past `offload.ram_watch_soft_fraction` of the cap, runs a ladder — collect,
trim, drop the upscaler cache, drop prefix caches, evict idle models. Every rung
releases memory that is already held. None of them can decline a load that is at
that moment streaming weights into the host.
That gap has a cost, measured on a 54 GB host running four engines:
```
Oct 29 23:03 Killed coderai-nvidia-gguf anon-rss 46.3 GB
Oct 30 03:10 Killed coderai-nvidia anon-rss 47.7 GB
Oct 30 03:36 Killed coderai-nvidia anon-rss 31.6 GB
Oct 30 03:49 Killed coderai-nvidia anon-rss 32.7 GB
front: engine#0 exited (code -9); respawning (1/5 in 120s)
```
The watcher was awake throughout and logged
`RSS 50.6/43.0 GB (118%) — mitigation [gc+trim+evict_idle] → 53.7 GB`: it ran
its ladder and the number went up anyway. The kernel settled it, and SIGKILL
takes the engine, not the request — so the client that asked for the oversized
model saw a dropped connection, and so did every unrelated request in flight,
plus everything that arrived during the 120-second respawn. One bad load is
reported as the whole server being unreliable.
So `offload.ram_admission` (default on) adds the missing half: before a load
starts, estimate what it will cost the host and refuse it if it does not fit.
`codai/models/host_ram.py` holds the arithmetic, `ensure_host_ram_for()` the
decision, and it sits beside `ensure_vram_for()` on every load path.
The estimate follows what actually determines the cost:
| situation | host-RAM cost |
| --- | --- |
| the model loaded before | `measured_ram_gb` — what it really parked on the host |
| `offload_strategy` `model` / `sequential` / `cpu` | the whole footprint; that is what was asked for |
| `offload_strategy: none`, or `no_ram` | the load buffer only — it fails on VRAM instead |
| anything else | the **spill**: footprint minus free VRAM |
| footprint unknown | the load buffer only, and the load proceeds unguarded |
plus a 2 GB staging allowance for the shard buffer, the safetensors read window
and dequantisation scratch.
The budget is the tighter of what the kernel says is available (`MemAvailable`,
which already counts reclaimable cache) and what is left under `max_ram_gb`,
less `offload.ram_admission_floor_gb` (default 4 GB) kept free for the OS and
the sibling engines.
Refusal is the last step, never the first: the ladder runs, the budget is
re-measured, and a load that fits afterwards is admitted. Only then does the
request fail — as a 503 naming the model, the estimate, the budget, the
shortfall, and what to change on the card. A model whose footprint could not be
determined is never refused on a number nobody has; it loses the protection and
says so in the log.
### Sizing that skips itself is worse than no sizing
The same night, the GGUF path printed this before loading a 35B:
```
(auto-offload sizing skipped: cannot access local variable '_ck'
where it is not associated with a value)
```
The auto-offload step reads the configured KV-cache types to decide how much
headroom to reserve, and it read them from a binding made a hundred lines
further down, where `llama_kwargs` is assembled. Every load that reached the
branch raised `UnboundLocalError`, the handler reported it as *skipped*, and
nothing reduced `n_gpu_layers`. llama.cpp was then asked to keep every layer on
a card that could not hold them and answered with a **43,562 MiB CPU KV
buffer** — which is what took the host down.
Two things follow. The read is hoisted above its use, with a test asserting the
source order rather than a line number. And the handler no longer says
*skipped*: a step that did not run leaves the load UNSIZED, and the message says
that, because "skipped" invites the reader to assume nothing needed doing.
## A resident model still needs room to run ## A resident model still needs room to run
Every reservation above is taken when a model LOADS. A model that is already Every reservation above is taken when a model LOADS. A model that is already
......
tests/test_host_ram_admission.py 0 → 100644
View file @ fdbc7701
# CoderAI - OpenAI-compatible API server
# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
"""Host-RAM admission control.
The case these cover, measured on a 54 GB host: six cards quantized only their
text encoder and left the transformer at bf16, two video cards asked for a
host-resident offload strategy, and requesting any of them ran the host out of
memory. The kernel killed the engine four times in five hours, so the client
never saw a refusal -- it saw a dead engine, and so did every unrelated request
in flight.
So what is asserted here is not "the estimate is accurate" but the decision:
does a load that cannot fit get refused BEFORE it allocates, with numbers a
caller can act on, and does a load that can fit still proceed.
"""
import pytest
from codai.models import host_ram as H
# ---------------------------------------------------------------------------
# estimate_need
# ---------------------------------------------------------------------------
def test_a_measured_load_is_believed_over_any_estimate():
"""The loader records what a load really parked on the host. Once that
exists there is nothing left to infer."""
need = H.estimate_need(total_need_gb=40.0, free_vram_gb=0.0,
measured_ram_gb=3.5)
assert need.basis == "measured"
assert need.gb == pytest.approx(3.5 + H.STAGING_GB)
assert "previous load" in need.detail
def test_a_zero_measurement_is_not_a_measurement():
"""0.0 means the model held nothing on the host OR that nothing was
recorded, and the two are indistinguishable — so fall through to the
estimate rather than admit a 40 GB load on the strength of a zero."""
need = H.estimate_need(total_need_gb=40.0, free_vram_gb=0.0,
measured_ram_gb=0.0)
assert need.basis == "spill"
assert need.gb == pytest.approx(40.0 + H.STAGING_GB)
@pytest.mark.parametrize("strategy", sorted(H.HOST_RESIDENT_STRATEGIES))
def test_a_host_resident_strategy_costs_the_whole_footprint(strategy):
"""offload_strategy='model' is a request to hold the weights in host RAM.
The cost is the model, not the part that missed the GPU — and crediting it
with free VRAM it will not use is how LongCat-Video was admitted onto a
host that could not hold it."""
need = H.estimate_need(total_need_gb=27.0, free_vram_gb=24.0,
offload_strategy=strategy)
assert need.basis == "host-resident"
assert need.gb == pytest.approx(27.0 + H.STAGING_GB)
assert strategy in need.detail
def test_a_gpu_only_load_is_charged_only_for_staging():
"""Strategy 'none' keeps every weight on the card, so a model too big for it
fails on VRAM — loudly and recoverably — and never reaches host RAM."""
for kwargs in ({"offload_strategy": "none"}, {"no_ram": True}):
need = H.estimate_need(total_need_gb=60.0, free_vram_gb=1.0, **kwargs)
assert need.basis == "gpu-only"
assert need.gb == pytest.approx(H.STAGING_GB)
def test_the_spill_is_what_did_not_fit_on_the_card():
need = H.estimate_need(total_need_gb=17.6, free_vram_gb=5.6)
assert need.basis == "spill"
assert need.gb == pytest.approx(12.0 + H.STAGING_GB)
def test_a_model_that_fits_on_the_card_spills_nothing():
need = H.estimate_need(total_need_gb=8.0, free_vram_gb=24.0)
assert need.gb == pytest.approx(H.STAGING_GB)
def test_an_unknown_footprint_is_never_charged_a_number_nobody_has():
"""A size that could not be determined must not become a refusal. The
protection is lost for that model; the model still loads."""
need = H.estimate_need(total_need_gb=0.0, free_vram_gb=0.0)
assert need.basis == "unknown"
assert need.gb == pytest.approx(H.STAGING_GB)
def test_free_vram_is_never_credited_as_negative_space():
need = H.estimate_need(total_need_gb=10.0, free_vram_gb=-5.0)
assert need.gb == pytest.approx(10.0 + H.STAGING_GB)
# ---------------------------------------------------------------------------
# budget
# ---------------------------------------------------------------------------
def test_the_kernel_binds_when_no_cap_is_configured():
bud = H.budget(kernel_available_gb=20.0, cap_gb=None, floor_gb=4.0)
assert bud.binding == "kernel"
assert bud.gb == pytest.approx(16.0)
def test_the_cap_binds_when_it_is_tighter_than_the_kernel():
"""A host with free RAM is still not free to use it past the server's own
ceiling — that ceiling exists so the sibling engines keep theirs."""
bud = H.budget(kernel_available_gb=30.0, cap_gb=43.0, fleet_held_gb=36.0,
floor_gb=4.0)
assert bud.binding == "cap"
assert bud.gb == pytest.approx(3.0)
assert "43 GB server RAM cap" in bud.detail
def test_the_kernel_still_binds_when_it_is_the_tighter_of_the_two():
bud = H.budget(kernel_available_gb=6.0, cap_gb=43.0, fleet_held_gb=10.0,
floor_gb=4.0)
assert bud.binding == "kernel"
assert bud.gb == pytest.approx(2.0)
def test_a_fleet_over_its_cap_has_no_budget_rather_than_a_negative_one():
bud = H.budget(kernel_available_gb=8.0, cap_gb=43.0, fleet_held_gb=50.0,
floor_gb=4.0)
assert bud.gb == 0.0
assert bud.binding == "floor"
def test_the_floor_is_held_back_from_whichever_bound_won():
assert H.budget(kernel_available_gb=10.0, floor_gb=0.0).gb == pytest.approx(10.0)
assert H.budget(kernel_available_gb=10.0, floor_gb=4.0).gb == pytest.approx(6.0)
# ---------------------------------------------------------------------------
# verdict + the refusal itself
# ---------------------------------------------------------------------------
def test_a_load_that_fits_is_admitted():
need = H.estimate_need(total_need_gb=8.0, free_vram_gb=24.0)
bud = H.budget(kernel_available_gb=20.0, floor_gb=4.0)
assert H.verdict(need, bud).ok
def test_the_production_case_is_refused_and_says_by_how_much():
"""Z-Image-Turbo bf16 (17.6 GB) onto a nearly-full card, on the fleet state
that was live when the engine was killed."""
need = H.estimate_need(total_need_gb=17.6, free_vram_gb=2.0)
bud = H.budget(kernel_available_gb=12.0, cap_gb=43.0, fleet_held_gb=40.0,
floor_gb=4.0)
v = H.verdict(need, bud)
assert not v.ok
assert v.shortfall_gb == pytest.approx(17.6)
def test_the_refusal_carries_the_numbers_not_just_prose():
"""An API layer must be able to report the figures without re-deriving
them, and a test must be able to assert on them without matching prose."""
need = H.estimate_need(total_need_gb=17.6, free_vram_gb=2.0)
bud = H.budget(kernel_available_gb=12.0, cap_gb=43.0, fleet_held_gb=40.0)
exc = H.HostRamExhausted("Tongyi-MAI/Z-Image-Turbo", need, bud)
assert exc.model == "Tongyi-MAI/Z-Image-Turbo"
assert exc.need.gb == pytest.approx(17.6)
assert exc.budget.gb == 0.0
assert exc.shortfall_gb == pytest.approx(17.6)
assert isinstance(exc, RuntimeError) # callers that catch broadly still see it
def test_the_refusal_explains_that_proceeding_kills_the_engine():
"""The message is the whole point: before this existed the same condition
produced a dropped connection and a respawning engine."""
need = H.estimate_need(total_need_gb=30.0, free_vram_gb=0.0)
bud = H.budget(kernel_available_gb=5.0)
msg = H.HostRamExhausted("m", need, bud).message()
assert "host RAM" in msg and "kill the engine" in msg
# ---------------------------------------------------------------------------
# remedy_for — the fix differs by what the card actually says
# ---------------------------------------------------------------------------
def test_a_text_encoder_only_card_is_told_to_quantize_its_transformer():
"""The exact shape of the six cards that caused this: quantization present,
covering the small half."""
r = H.remedy_for({"component_quantization": {"text_encoder": "4bit"}}, "spill")
assert "transformer" in r and "text_encoder" in r
def test_a_card_that_already_quantizes_its_transformer_gets_no_such_advice():
for comp in ({"transformer": "4bit"}, {"unet": "4bit"},
{"transformer": "4bit", "text_encoder": "4bit"}):
assert "add 'transformer'" not in H.remedy_for(
{"component_quantization": comp}, "spill")
def test_a_host_resident_card_is_told_about_its_strategy():
r = H.remedy_for({"offload_strategy": "model"}, "host-resident")
assert "offload_strategy='model'" in r
def test_an_unquantized_card_is_told_it_is_unquantized():
r = H.remedy_for({"offload_strategy": "auto"}, "spill")
assert "no quantization" in r
def test_a_quantized_card_that_merely_does_not_fit_is_not_lectured():
"""Advice that does not apply is worse than none: it sends the operator to
change a setting that is already right."""
assert H.remedy_for({"load_in_4bit": True, "offload_strategy": "auto"},
"spill") == ""
def test_no_card_at_all_yields_no_advice():
assert H.remedy_for(None) == ""
assert H.remedy_for({}) == ""
# ---------------------------------------------------------------------------
# the manager gate
# ---------------------------------------------------------------------------
@pytest.fixture
def gate(monkeypatch):
"""The real method, on a manager whose measurements are ours."""
from codai.models.manager import multi_model_manager as m
monkeypatch.setattr(m, "models", {}, raising=False)
monkeypatch.setattr(m, "config", {}, raising=False)
monkeypatch.setattr(m, "_ram_admission_enabled", staticmethod(lambda: True))
monkeypatch.setattr(m, "_ram_admission_floor_gb", staticmethod(lambda: 4.0))
monkeypatch.setattr(m, "_evict_idle_on_ram_enabled", staticmethod(lambda: False))
monkeypatch.setattr(m, "_get_free_vram_gb", lambda *a, **k: 2.0)
monkeypatch.setattr(m, "_get_model_used_vram_gb", lambda *a, **k: 17.6)
monkeypatch.setattr(m, "_ram_cap_gb", staticmethod(lambda: 43.0))
monkeypatch.setattr(m, "_get_process_ram_gb", staticmethod(lambda: 40.0))
return m
def test_the_gate_refuses_the_load_that_killed_the_engine(gate, monkeypatch):
monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: 12.0)
gate.config["image:Z"] = {"component_quantization": {"text_encoder": "4bit"}}
with pytest.raises(H.HostRamExhausted) as ei:
gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo", "Z-Image-Turbo")
# Named by what the caller ASKED for, not by the internal key.
assert ei.value.model == "Z-Image-Turbo"
assert "add 'transformer'" in ei.value.remedy
def test_the_gate_lets_a_load_through_when_there_is_room(gate, monkeypatch):
monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: 80.0)
monkeypatch.setattr(gate, "_ram_cap_gb", staticmethod(lambda: None))
gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo")
def test_a_resident_model_is_never_refused(gate, monkeypatch):
"""It is not about to allocate anything — and asking "is there room for it?"
while it IS the memory is the self-eviction bug the VRAM path already had."""
monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: 0.0)
for key in ("image:Z", "Tongyi-MAI/Z-Image-Turbo", "Z-Image-Turbo"):
gate.models.clear()
gate.models[key] = object()
gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo", "Z-Image-Turbo")
def test_turning_admission_off_restores_the_old_behaviour(gate, monkeypatch):
"""An operator who would rather take the risk keeps the right to."""
monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: 0.0)
monkeypatch.setattr(gate, "_ram_admission_enabled", staticmethod(lambda: False))
gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo")
def test_an_unsizeable_model_is_admitted_rather_than_blocked(gate, monkeypatch):
monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: 7.0)
monkeypatch.setattr(gate, "_get_model_used_vram_gb", lambda *a, **k: 0.0)
gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo")
def test_the_gate_frees_host_ram_before_it_refuses(gate, monkeypatch):
"""Refusal is the last step, not the first: the same ladder RAM pressure
already uses runs first, and a load that fits afterwards is admitted."""
freed = []
avail = [6.0]
monkeypatch.setattr("codai.models.proc_mem.host_available_gb", lambda: avail[0])
monkeypatch.setattr(gate, "_evict_idle_on_ram_enabled", staticmethod(lambda: True))
monkeypatch.setattr(gate, "_ram_cap_gb", staticmethod(lambda: None))
monkeypatch.setattr(gate, "_get_model_used_vram_gb", lambda *a, **k: 10.0)
def _unload(idle_seconds):
freed.append(idle_seconds)
avail[0] = 40.0
monkeypatch.setattr(gate, "unload_idle_models", _unload)
gate.ensure_host_ram_for("image:Z", "Tongyi-MAI/Z-Image-Turbo")
assert freed, "eviction was never attempted before deciding"
assert freed[0] >= 1.0, (
"an idle threshold near zero pulls a model that is merely between its "
"own requests")
# ---------------------------------------------------------------------------
# wiring — the gate has to be ON the load paths, not merely importable
# ---------------------------------------------------------------------------
def _src(rel):
import io
import os
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
return io.open(os.path.join(root, rel), encoding="utf-8").read()
def test_every_vram_admission_point_also_admits_on_host_ram():
"""The two checks answer the same question about different memory. A load
path that reserves VRAM and not host RAM is exactly the gap that let the
kernel decide."""
src = _src("codai/models/manager.py")
body = src.split("def request_model(", 1)[1] if "def request_model(" in src else src
assert body.count("self.ensure_vram_for(") == body.count("self.ensure_host_ram_for(")
def test_the_image_path_admits_on_host_ram():
"""It does its own eviction instead of going through request_model, so it
needs its own call — and it is the path the failing cards are on."""
assert "ensure_host_ram_for(" in _src("codai/api/images.py")
def test_the_refusal_is_reported_as_503_and_not_a_500():
app_src = _src("codai/api/app.py")
assert "HostRamExhausted" in app_src
assert "status_code=503" in app_src.split("HostRamExhausted", 1)[1]
def test_the_setting_reaches_the_running_server():
"""A config field nothing plumbs is a field that does nothing: the cap it
guards is read live from global args."""
assert "ram_admission" in _src("codai/config.py")
assert "global_args.ram_admission" in _src("codai/main.py")
assert "ram_admission" in _src("codai/admin/routes.py")
# ---------------------------------------------------------------------------
# the auto-offload sizing step that was skipping itself
# ---------------------------------------------------------------------------
def test_the_kv_types_are_read_before_the_offload_sizing_uses_them():
"""The regression that took the host down three times in five hours.
``_kv_quantized = bool(_ck or _cv)`` sat in the auto-offload sizing block,
while ``_ck``/``_cv`` were bound ~110 lines LATER, where llama_kwargs is
assembled. Reading them first raised UnboundLocalError on every load that
reached the branch; the handler printed "(auto-offload sizing skipped: …)",
which reads like a decision; nothing reduced n_gpu_layers; and llama.cpp
answered a 35B that did not fit with a 43,562 MiB CPU KV buffer.
Asserted on source ORDER rather than line numbers, so moving the block does
not break the test -- only reintroducing the bug does.
"""
src = _src("codai/backends/vulkan.py")
use = src.index("_kv_quantized = bool(_ck or _cv)")
first_bind = min(src.index("_ck = kwargs.get("),
src.index("_cv = kwargs.get("))
assert first_bind < use, (
"_ck/_cv are used by the auto-offload sizing before they are bound — "
"UnboundLocalError, swallowed as 'sizing skipped', model loads unsized")
def test_a_failed_sizing_step_does_not_read_as_a_completed_one():
"""'skipped' invites the reader to assume nothing needed doing. The message
has to say the load is unsized, or the next person reads past it again."""
src = _src("codai/backends/vulkan.py")
assert "auto-offload sizing FAILED" in src
assert "this load is UNSIZED" in src
# And the consequence has to be in the printed line itself, not only in a
# comment next to it: a print is what the operator sees.
printed = [ln for ln in src.splitlines()
if "auto-offload sizing FAILED" in ln and not ln.strip().startswith("#")]
assert printed, "the warning is not in a print, only in a comment"
tests/test_remote_code_allowlist.py
View file @ fdbc7701
...@@ -129,7 +129,11 @@ def test_a_permission_error_is_not_swallowed_by_the_loader(): ...@@ -129,7 +129,11 @@ def test_a_permission_error_is_not_swallowed_by_the_loader():
a refusal must not be one of them.""" a refusal must not be one of them."""
src = open("codai/api/images.py").read() src = open("codai/api/images.py").read()
head = src.index("_img_remote = {}") head = src.index("_img_remote = {}")
body = src[head:head + 900] # Bounded by the next statement after the block, not by a character count:
# a fixed window silently stops covering the code it was written for the
# moment anything is added inside it, which is how this test broke when the
# declared-class check went in.
body = src[head:src.index("_img_comp_models", head)]
assert "except PermissionError:" in body and "raise" in body assert "except PermissionError:" in body and "raise" in body
...@@ -147,3 +151,64 @@ def test_such_a_model_is_not_tried_as_sdxl_first(): ...@@ -147,3 +151,64 @@ def test_such_a_model_is_not_tried_as_sdxl_first():
def test_the_field_is_savable_from_the_admin_page(): def test_the_field_is_savable_from_the_admin_page():
assert '"custom_pipeline",' in open("codai/admin/routes.py").read() assert '"custom_pipeline",' in open("codai/admin/routes.py").read()
# ---------------------------------------------------------------------------
# A repo that declares its own class without any card saying so
# ---------------------------------------------------------------------------
def test_a_repo_declaring_an_unavailable_class_is_detected(monkeypatch):
"""MOSS-Image and Kromoss, exactly as they failed: no card named a
custom_pipeline, so nothing fired, and the load died on
'module diffusers has no attribute MOSSImagePipeline'."""
from codai.models import remote_code as R
monkeypatch.setattr(R, "declared_pipeline_class",
lambda repo: "MOSSImagePipeline")
kw = R.kwargs_for_declared_class(_cfg(["openmoss-team/*"]),
"OpenMOSS-Team/MOSS-Image-SFT")
assert kw["trust_remote_code"] is True
assert kw["custom_pipeline"] == "OpenMOSS-Team/MOSS-Image-SFT"
assert kw["_declared_class"] == "MOSSImagePipeline"
def test_such_a_repo_is_refused_by_name_when_not_allow_listed(monkeypatch):
"""The refusal has to name the setting — that is the whole reason the
allow-list is spelled out rather than implied."""
from codai.models import remote_code as R
monkeypatch.setattr(R, "declared_pipeline_class",
lambda repo: "MOSSImagePipeline")
with pytest.raises(PermissionError) as ei:
R.kwargs_for_declared_class(_cfg([]), "lodestones/Kromoss")
msg = str(ei.value)
assert "models.trust_remote_code" in msg
assert "lodestones/*" in msg
assert "MOSSImagePipeline" in msg
def test_an_ordinary_repo_pays_nothing(monkeypatch):
"""The class diffusers already has must not trigger any of this: every
normal image model goes through this code path."""
from codai.models import remote_code as R
monkeypatch.setattr(R, "declared_pipeline_class",
lambda repo: "StableDiffusionXLPipeline")
assert R.kwargs_for_declared_class(_cfg([]), "any/sdxl") == {}
def test_a_repo_that_declares_nothing_is_left_alone(monkeypatch):
from codai.models import remote_code as R
monkeypatch.setattr(R, "declared_pipeline_class", lambda repo: None)
assert R.kwargs_for_declared_class(_cfg([]), "any/model") == {}
def test_the_marker_never_reaches_from_pretrained():
"""_declared_class is ours, for the log line. Passing it on would be a
TypeError from diffusers at the worst possible moment."""
src = open("codai/api/images.py").read()
head = src.index("_img_remote = {}")
body = src[head:src.index("_img_comp_models", head)]
assert '_img_remote.pop("_declared_class"' in body
def test_the_loader_asks_the_repository_when_no_card_did():
src = open("codai/api/images.py").read()
assert "kwargs_for_declared_class(" in src
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment