measure a model's memory from the model, and release what it leaves behind

Six findings from one 21h30m production log (548,687 lines, 51 loads of 12
models). They share a cause: nothing measured a model directly, so every
decision downstream of a measurement was made on a number that was wrong.

1. The footprint was a process-RSS delta
---------------------------------------
`measured_ram = self._get_own_ram_gb() - ram_before`, where _get_own_ram_gb
sums RSS over the engine and every child. RSS counts mmap'd weight-file pages
the loader just walked; summed over a tree it counts pages SHARED between those
processes once per process; and as a delta it absorbs anything else that moved
in the window. Against ground truth:

    gemma-2-9b-it.Q4_K_M   5.37 GB on disk  ->  reported 30.81 GB   (5.7x)
    LTX-2.5, six loads, one unchanged config:
        22.11, 22.11, 23.28, 4.50, 19.02, 29.80 GB                  (6.6x)

Every other model sat at 1.0-1.2x, so this was not noise. Wan2.2's 3.4x spread
was the real quantization fix and is the one that belonged there.

codai/models/footprint.py measures the object instead. Quantization needs no
correction factor, which is the point: a bitsandbytes Params4bit IS a uint8
tensor, so numel x element_size already gives packed bytes. What those formats
hide is counted explicitly -- bnb's quant_state (absmax, code book, nested
double-quant state) lives outside parameters(), and GPTQ/AWQ keep
qweight/qzeros/scales in BUFFERS, so a parameters-only walk reports a 4-bit
model as nearly empty. Tied weights are deduped by storage pointer; accelerate
meta placeholders are sized from their dtype and flagged as declared, keeping
the placement-independence the field already documents; a diffusers pipeline is
walked through its components, since it is not an nn.Module.

Context is costed properly: KV from the KEY/VALUE head count, not the attention
head count (4x on a GQA model), with sliding-window layers sized to their
window. But a computed KV is only a fallback and says so -- this codebase
already learned that "predicting KV from GGUF metadata is architecture-specific
... and was off by 5x in BOTH directions", so measured_kv_gb for the context in
question always wins. A GGUF, which no walk can reach, falls back to its file
size; it is loaded as stored, so the file IS the weights.

2. A failed load overwrote the good measurement
-----------------------------------------------
    Device map: cpu: 1 layers
    Measured footprint: 4.50 GB total
    Saved measured_vram_gb=4.505 (force_vram_update)
    generation FAILED: StopIteration interacts badly with generators
    Full-GPU load looks viable (4.5 GB peak need)   <- 4 seconds later

A load that placed one layer and raised still reached the recorder, and
force_vram_update wrote it to disk as truth. Now a measurement that is
implausible on its own (~0 GB) or that collapses to under half of what the SAME
quantization signature previously measured is treated as a partial load and
discarded. Wan2.2's genuine 40.04 -> 11.73 GB drop is let through because it
arrived WITH a signature change, which is exactly what tells the two apart.
The rejection names which of the two checks fired.

The ratio guard alone was not enough. Walking a pipeline and finding that NO
component holds any weights is a failed load outright, and there is nothing to
fall back to -- a size taken from process memory measures everything except
the model in question. That case is now refused before any estimate is
considered, and the refusal is recorded in the history with its reason rather
than left looking like a successful load. The hole showed up as one of the new
tests passing alone and failing in the full suite: the fallback delta grows
with whatever else the process has loaded, so the figure it invented depended
on test order.

3. The teardown-leak breaker never looked at host RAM
-----------------------------------------------------
It registers under `if _tens.is_cuda` and sweeps under `_obj.is_cuda`. A
pipeline loaded with enable_model_cpu_offload keeps its weights in HOST RAM,
and 17 of 21 video loads took that path, so none of those weights were tracked
and nothing looked for them. With nothing loaded and 482 MiB of VRAM in use:

    coderai-nvidia   33.14 GB PSS_Anon
    30.85 GB anon in 6588 mappings; 6565 of them under 100 MB (20.25 GB)

Thousands of mid-sized allocations is retained tensors, not allocator slack --
which is why gc and malloc_trim ran every 15 s for hours with the figure never
moving, and why the ladder then evicted healthy models to chase it:

    RAM eviction: unloading 'video:Lightricks/LTX-2.5-Diffusers' to free host
    RAM (RSS 40.0 GB > cap target 36.5 GB)

codai/models/teardown.py adds the host arm, by weakref rather than by storage
pointer: a freed host address is readily handed back out by malloc, so a
pointer match could null a live tensor that by then belongs to another model. A
referent still alive after teardown IS this pipeline's leak, identity-exact. It
runs before the allocator trim, since trimming while the weights are still
referenced returns nothing.

4. The RAM cap counted shared pages and page cache
--------------------------------------------------
codai/models/proc_mem.py reads Pss_Anon + Pss_Shmem from smaps_rollup, which
has already divided shared pages by their sharers and excludes reclaimable page
cache. Measured correction on a 5-engine host: 40.06 GB (RSS) -> 37.91 GB
(PSS). Modest -- the leak above was the real problem -- but a cap should chase
memory that freeing an allocation can actually release. Falls back to the old
reading rather than to 0.0, which would read as "no memory in use".

5. A model nobody used stayed loaded
------------------------------------
    Ondemand mode - keeping 'audio_gen:ACE-Step' in VRAM alongside new model

It held 7.9 GB of a 3090 and 2.1 GB of host RAM for hours after its last
request. offload.idle_unload_seconds (default 1800) unloads a model untouched
that long. Deliberately generous: reloads are the expensive event -- that day
spent 2h04m of 21h30m inside loads -- so a short timeout would trade a VRAM
saving nobody asked for against a reload everybody waits on. A key that was
never stamped is left alone rather than assumed idle. It runs before the
leak-watch gate, so turning that off does not silently disable this.

6. The log left no room for the history anyone needed
-----------------------------------------------------
32,633 of 33,009 requests were embeddings, and one engine produced 426,682 of
the 548,687 lines -- 78% -- filling the 2,000-line view in about four minutes.
The per-item endpoints are now summarised to one line (model, count, shape)
instead of dumped, which also stops a caller's content being written to a
shared log. Prompts are deliberately NOT truncated: a 4,000-character script
prompt is what identified the music defect below.

GET /v1/models/history replaces inferring swaps from a log that rotates: time,
event, model, engine, plus seconds, vram_gb, offloaded_gb, strategy for a load
and reason, freed_gb for an eviction. Eviction reasons are passed at the call
sites. The front merges across engines, because a swap is a cross-engine event
-- one engine evicts so a co-located one can load -- so one engine's copy is
half the story.

Also: ACE-Step was asked for a song with no words
-------------------------------------------------
Motus sends a music_prompt and no lyrics, and empty lyrics do not mean
instrumental: the pipeline then conditions on a single padding token
(lyric_token_idx = [0], lyric_mask = [0]) and the model still tries to sing,
which comes out as babbling over the backing track. ACE-Step's own UI documents
the way to ask -- "Use [instrumental] or [inst] to generate instrumental
music" -- and every genre preset it ships ends in either "male vocals, raw
vocals" or "instrumental". Both are now sent when no lyrics are given.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.15"
+__version__ = "0.5.16"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/app.py

@@ -937,6 +937,24 @@ async def list_models():
     return ModelList(data=models)
 
 
+@app.get("/v1/models/history", summary="Model load / eviction history", tags=["Core"])
+async def models_history(limit: int = 200, event: str = None):
+    """When models were loaded and evicted on this engine, and why.
+
+    Answers the question a slow render actually raises -- which swaps happened,
+    what they cost, what triggered them -- without having to reconstruct it from
+    the log, which rotates: on one production host 78% of the log was
+    per-request debug and the 2,000-line view covered about four minutes.
+
+    Each entry carries ``time``, ``event`` ('load' | 'evict'), ``model`` and
+    ``engine``, plus whatever the call site knew: ``seconds`` (how long the load
+    took), ``vram_gb`` (measured footprint), ``offloaded_gb``, ``strategy``, and
+    for an eviction ``reason`` and ``freed_gb``. Newest first, in-memory and
+    bounded, so it is empty after a restart.
+    """
+    return JSONResponse({"history": multi_model_manager.get_history(limit, event)})
+
+
 def _admin_scoped_token(request) -> str:
     """The caller's bearer token when it carries the admin scope, else "".
 

codai/api/audio_gen.py

@@ -407,9 +407,23 @@ def _generate_audio(pipe, model_name: str, request: AudioGenerationRequest, task
         # for. A field left out falls back to ACE-Step's own default; a field
         # explicitly set to 10 still means 10.
         _sent = getattr(request, 'model_fields_set', set())
+        # ACE-Step is a SONG model: the prompt carries comma-separated tags and
+        # the lyrics carry the words. Handing it empty lyrics does not mean
+        # "instrumental" -- the pipeline then conditions on a single padding
+        # token (lyric_token_idx = [0], lyric_mask = [0]) and the model still
+        # tries to sing, which comes out as babbling over the backing track.
+        # Its own UI documents the way to ask: "Use [instrumental] or [inst] to
+        # generate instrumental music", and every genre preset it ships ends in
+        # either "male vocals, raw vocals" or "instrumental".
+        _lyrics = (getattr(request, 'lyrics', None) or "").strip()
+        _prompt = request.prompt or ""
+        if not _lyrics:
+            _lyrics = "[instrumental]"
+            if "instrumental" not in _prompt.lower():
+                _prompt = (_prompt + ", instrumental") if _prompt else "instrumental"
         wav = pipe.generate(
-            prompt=request.prompt,
-            lyrics=getattr(request, 'lyrics', None) or "",
+            prompt=_prompt,
+            lyrics=_lyrics,
             duration=(float(request.duration) if 'duration' in _sent
                       and request.duration else 60.0),
             steps=int(getattr(request, 'steps', None) or 60),

codai/api/log.py

@@ -54,6 +54,32 @@ def _redact_blobs(obj):
     return obj
 
 
+def _summarise(parsed) -> str:
+    """One line describing a request, for the high-volume paths.
+
+    Says what is needed to follow what the server was asked to do -- the model,
+    how many items, the shape -- without reproducing the caller's content.
+    """
+    if not isinstance(parsed, dict):
+        return "(no JSON body)"
+    bits = []
+    model = parsed.get("model")
+    if model:
+        bits.append(f"model={model}")
+    inp = parsed.get("input")
+    if isinstance(inp, list):
+        bits.append(f"inputs={len(inp)}")
+        chars = sum(len(x) for x in inp if isinstance(x, str))
+        if chars:
+            bits.append(f"chars={chars}")
+    elif isinstance(inp, str):
+        bits.append(f"inputs=1 chars={len(inp)}")
+    for key in ("dimensions", "encoding_format"):
+        if parsed.get(key) is not None:
+            bits.append(f"{key}={parsed[key]}")
+    return " ".join(bits) or "(empty body)"
+
+
 def get_recent_activity():
     return list(_activity)
 
@@ -119,6 +145,25 @@ def _durable_request_debug(method, url, parsed):
         pass
 
 
+# NOTE: long free text is deliberately NOT truncated here. A prompt is the most
+# useful thing in the log -- a 4,000-character script prompt was what identified
+# a music defect -- and capping it would trade the diagnosis for bytes. The
+# volume problem was one endpoint called per item rather than per user action,
+# which _SUMMARY_ONLY_PATHS addresses without touching anything else.
+
+#: Paths whose bodies are SUMMARISED rather than dumped, even under debug.
+#:
+#: These are called per item, not per user action, so a full dump is the log
+#: rather than a part of it. One production day: 32,633 embedding requests out
+#: of 33,009 total, and the radeon engine alone produced 426,682 of the log's
+#: 548,687 lines -- about 78% -- which filled the 2,000-line view in roughly
+#: four minutes and left no record of the model loads and evictions anyone was
+#: trying to diagnose. The activity tracking below is unaffected: it is one
+#: dict entry per request and is what the admin page reads.
+_SUMMARY_ONLY_PATHS = {
+    "/v1/embeddings",
+}
+
 _TRACKED_PATHS = {
     "/v1/chat/completions": "chat",
     "/v1/completions": "completion",
@@ -175,7 +220,9 @@ async def log_requests(request: Request, call_next):
                 parsed = json.loads(body_str)
                 model = parsed.get("model", "—")
 
-                if global_debug:
+                if global_debug and path in _SUMMARY_ONLY_PATHS:
+                    print(f"[req] {request.method} {path} {_summarise(parsed)}")
+                elif global_debug:
                     print(f"\n{'='*80}")
                     print(f"=== FULL REQUEST DEBUG ===")
                     print(f"Method: {request.method}  URL: {request.url}")

codai/api/video.py

@@ -940,6 +940,21 @@ def _free_pipeline_vram(pipe) -> None:
                         continue
         except Exception:
             pass
+    # The pointer pass above is CUDA-only, by design and by necessity: a freed
+    # HOST address is readily handed back out by malloc, so matching host tensors
+    # by storage pointer could null a live tensor that by then belongs to another
+    # model. Track the host-resident ones by weak reference instead, so a referent
+    # still alive after teardown IS this pipeline's leaked tensor. Without this,
+    # a 'model CPU offload' pipeline -- 17 of 21 video loads on one production day
+    # -- had NONE of its weights tracked, and the engine sat on 33 GB of anonymous
+    # host RAM with an empty GPU. See codai/models/teardown.py.
+    _host_refs = []
+    try:
+        from codai.models import teardown as _td
+        _host_refs = _td.track_host_tensors(pipe)
+    except Exception:
+        _host_refs = []
+
     try:
         if pipe is not None:
             _comps = getattr(pipe, 'components', {}) or {}
@@ -1083,6 +1098,22 @@ def _free_pipeline_vram(pipe) -> None:
                           f"{_vram_used0:.1f} → {_vram_used1:.1f} GB")
         except Exception as _lbe:
             print(f"  [video] teardown-leak breaker failed: {_lbe}")
+    # Host-side arm of the same breaker. Runs BEFORE the allocator trim: trimming
+    # while the weights are still referenced returns nothing, which is what the
+    # RAM mitigation ladder was doing every 15 s for hours without the figure
+    # moving.
+    if _host_refs:
+        try:
+            from codai.models import teardown as _td
+            _hr = _td.break_host_leaks(_host_refs)
+            if _hr["broken"] or _hr["survived_gb"] > 0.5:
+                print(f"  [video] host teardown-leak breaker: "
+                      f"{_hr['survived_gb']:.1f} GB of this pipeline's host-resident "
+                      f"tensors survived teardown; nulled {_hr['broken']} external "
+                      f"reference(s)"
+                      + (f"; holders: {_hr['holders']}" if _hr["holders"] else ""))
+        except Exception as _he:
+            print(f"  [video] host teardown-leak breaker failed: {_he}")
     try:
         from codai.models.manager import _trim_cpu_ram
         _trim_cpu_ram()

codai/config.py

@@ -292,6 +292,17 @@ class OffloadConfig:
     ram_watch_poll_seconds: float = 15.0    # how often the watcher samples RSS
     ram_watch_soft_fraction: float = 0.90   # mitigate at/above this fraction of the cap
     ram_watch_cuda: bool = True             # allow mitigation to call CUDA empty_cache()
+    # Unload a model that has gone untouched for this long, in seconds (0 = never).
+    #
+    # Eviction already frees a model when something else needs the memory, so this
+    # is for the case where nothing does: a 3.5B music model sat on 7.9 GB of a
+    # 3090 and 2.1 GB of host RAM for hours after its last request, because the
+    # on-demand path keeps a model resident whenever there is room for it.
+    #
+    # Kept generous on purpose. Reloads are the expensive event -- one production
+    # day spent 2h04m of 21h30m inside model loads -- so a short timeout trades a
+    # VRAM saving nobody asked for against a reload everybody waits on.
+    idle_unload_seconds: float = 1800.0
     # Cross-backend GPU pooling. OFF by default: each engine uses only its own
     # backend's GPUs (CUDA for an NVIDIA engine, Vulkan for a Radeon engine) and a
     # model still splits across multiple SAME-backend cards (e.g. 2× 3090). When ON,

codai/frontproxy/app.py

@@ -3100,6 +3100,33 @@ def build_app(config, config_dir=None) -> FastAPI:
                             media_type=val.headers.get("content-type"))
         return JSONResponse(val)
 
+    # /v1/models/history is the MERGE across engines, newest first: a swap is a
+    # cross-engine event (one engine evicts so a co-located one can load), so
+    # reading a single engine's copy tells only half the story. Registered before
+    # the catch-all so it is aggregated rather than proxied to one engine.
+    @app.get("/v1/models/history", include_in_schema=False)
+    async def _models_history(request: Request):
+        headers = front._filter_headers(request.headers, _DROP_REQ)
+        merged = []
+        for e in front.registry.all():
+            try:
+                r = await front._short.get(e.url + "/v1/models/history",
+                                           headers=headers,
+                                           params=request.query_params)
+                if r.status_code == 200:
+                    rows = (r.json() or {}).get("history") or []
+                    for row in rows:
+                        row.setdefault("engine", getattr(e, "name", "") or "")
+                    merged.extend(rows)
+            except Exception:
+                continue          # an engine that is busy or down just contributes nothing
+        merged.sort(key=lambda r: r.get("time") or 0, reverse=True)
+        try:
+            limit = max(1, min(500, int(request.query_params.get("limit") or 200)))
+        except (TypeError, ValueError):
+            limit = 200
+        return JSONResponse({"history": merged[:limit]})
+
     # Engine management (front-owned: it runs the supervisor). Registered before
     # the catch-all so they aren't proxied to an engine.
     @app.get("/admin/api/engines", include_in_schema=False)

codai/main.py

@@ -1216,6 +1216,7 @@ def main():
     # Global host-RAM cap + leak watch (read live by hf_loading, manager, ram_monitor).
     global_args.max_ram_gb = config.offload.max_ram_gb
     global_args.evict_idle_on_ram = config.offload.evict_idle_on_ram
+    global_args.idle_unload_seconds = config.offload.idle_unload_seconds
     global_args.ram_leak_watch = config.offload.ram_leak_watch
     global_args.ram_watch_poll_seconds = config.offload.ram_watch_poll_seconds
     global_args.ram_watch_soft_fraction = config.offload.ram_watch_soft_fraction

codai/models/proc_mem.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""How much host memory the server fleet actually holds.
+
+The host-RAM cap used to be enforced against the sum of RSS over the front, the
+engines and every worker. That number cannot be driven down by evicting models,
+for two reasons:
+
+* **Shared pages are counted once per process.** The interpreter, torch, the
+  CUDA runtime and libc are mapped into every engine. Summing RSS adds each of
+  those pages as many times as there are processes.
+* **File-backed pages count as RSS.** Walking a mmap'd GGUF or safetensors file
+  maps its pages into the process; they are page cache, reclaimed by the kernel
+  on demand, and dropping a model does not release them.
+
+Measured on a production host, 2026-10-29. The mitigation ladder ran every
+~15 s for hours with the figure never moving:
+
+    RAM watch: RSS 40.0/43.0 GB (93%) -- mitigation [gc+trim+evict_idle] -> 39.1 GB
+    RAM watch: RSS 39.1/43.0 GB (91%) -- mitigation [gc+trim+evict_idle] -> 39.1 GB
+    RAM watch: RSS 39.1/43.0 GB (91%) -- mitigation [gc+trim+evict_idle] -> 39.1 GB
+
+Before and after are the same value, 291 identical repetitions for one figure
+alone, while the host itself had 14 GB available and 11 GB of reclaimable
+cache. It was not merely noisy -- it evicted working models to chase it:
+
+    RAM eviction: unloading 'video:Lightricks/LTX-2.5-Diffusers' to free host
+    RAM (RSS 40.0 GB > cap target 36.5 GB)
+
+The fix is to measure proportional set size instead. ``Pss`` divides each
+shared page by the number of processes sharing it, so summing PSS across a tree
+counts every page exactly once, and the ``Pss_Anon``/``Pss_File`` split
+separates anonymous memory (which only freeing an allocation releases) from
+page cache (which the kernel reclaims by itself). A cap should chase the
+former.
+"""
+
+from __future__ import annotations
+
+import os
+
+#: Fields of ``/proc/<pid>/smaps_rollup`` we care about, in kB.
+_WANTED = ("Pss", "Pss_Anon", "Pss_File", "Pss_Shmem", "Rss")
+
+
+def read_rollup(pid) -> dict:
+    """Parse ``/proc/<pid>/smaps_rollup`` into a dict of bytes, or ``{}``.
+
+    Cheap: the kernel aggregates the whole address space into one record, so
+    this is a single short read regardless of how many mappings a process has
+    (unlike ``smaps``, which emits a stanza per mapping and is what makes
+    naive memory accounting expensive).
+    """
+    try:
+        with open(f"/proc/{int(pid)}/smaps_rollup") as fh:
+            out = {}
+            for line in fh:
+                name, _, rest = line.partition(":")
+                if name in _WANTED:
+                    try:
+                        out[name] = int(rest.split()[0]) * 1024
+                    except (IndexError, ValueError):
+                        continue
+            return out
+    except (OSError, ValueError, TypeError):
+        return {}
+
+
+def process_held_bytes(pid) -> int:
+    """Memory this one process holds that only it can release.
+
+    ``Pss_Anon`` is anonymous memory -- allocations, model weights that were
+    copied rather than mapped -- shared pages already divided by their sharers.
+    ``Pss_Shmem`` is shared-memory segments, which are not backed by a file on
+    disk and so are not reclaimable either. ``Pss_File`` is deliberately
+    excluded: those are mapped file pages, i.e. page cache.
+
+    Falls back to ``Pss`` and then ``Rss`` on a kernel or permission setup that
+    does not expose the breakdown, so the caller always gets a number -- a
+    coarse one, which is still better than none.
+    """
+    r = read_rollup(pid)
+    if not r:
+        return 0
+    if "Pss_Anon" in r:
+        return r.get("Pss_Anon", 0) + r.get("Pss_Shmem", 0)
+    if "Pss" in r:
+        return r["Pss"]
+    return r.get("Rss", 0)
+
+
+def tree_pids(root_pid=None) -> list:
+    """The root process and all its descendants."""
+    try:
+        import psutil
+    except Exception:
+        return [int(root_pid)] if root_pid else [os.getpid()]
+    try:
+        proc = psutil.Process(int(root_pid)) if root_pid else psutil.Process()
+    except Exception:
+        try:
+            proc = psutil.Process()
+        except Exception:
+            return []
+    pids = [proc.pid]
+    try:
+        pids += [c.pid for c in proc.children(recursive=True)]
+    except Exception:
+        pass
+    return pids
+
+
+def tree_held_gb(root_pid=None) -> float:
+    """Unreclaimable host memory held by a process tree, in GB.
+
+    Summing PSS across the tree is sound precisely because PSS has already
+    divided shared pages by their sharers: a page mapped into five engines
+    contributes a fifth from each, totalling one.
+    """
+    total = 0
+    for pid in tree_pids(root_pid):
+        total += process_held_bytes(pid)
+    return total / 1e9
+
+
+def tree_rss_gb(root_pid=None) -> float:
+    """Summed RSS of the tree, in GB -- the old reading, kept for comparison.
+
+    Reported alongside the PSS figure so the gap between them is visible in
+    diagnostics rather than inferred: on a multi-engine host it is large, and
+    it is entirely double-counted shared pages plus page cache.
+    """
+    total = 0
+    for pid in tree_pids(root_pid):
+        r = read_rollup(pid)
+        total += r.get("Rss", 0)
+    return total / 1e9
+
+
+def host_available_gb() -> float:
+    """What the kernel says is actually available, including reclaimable cache.
+
+    ``MemAvailable`` is the kernel's own estimate of what can be allocated
+    without swapping, and it already accounts for the page cache it is willing
+    to drop. A cap that disagrees with this is measuring the wrong thing.
+    """
+    try:
+        with open("/proc/meminfo") as fh:
+            for line in fh:
+                if line.startswith("MemAvailable:"):
+                    return int(line.split()[1]) / (1024 ** 2)
+    except OSError:
+        pass
+    try:
+        import psutil
+        return psutil.virtual_memory().available / (1024 ** 3)
+    except Exception:
+        return 0.0

codai/models/ram_monitor.py

@@ -48,6 +48,17 @@ def get_status() -> Dict[str, Any]:
         return dict(_state)
 
 
+def _idle_unload_seconds() -> float:
+    """``offload.idle_unload_seconds`` (0 = never), via the global args."""
+    try:
+        from codai.api.state import get_global_args
+        ga = get_global_args()
+        v = getattr(ga, "idle_unload_seconds", None) if ga else None
+        return float(v) if v else 0.0
+    except Exception:
+        return 0.0
+
+
 def _cap_gb() -> Optional[float]:
     try:
         from codai.api.state import get_global_args
@@ -211,6 +222,19 @@ def _loop():
     while True:
         try:
             time.sleep(_poll_seconds())
+            # Idle unload is independent of the leak watch and of the RAM cap: a
+            # model nobody has used for half an hour is not holding its memory
+            # for any reason, and waiting for pressure is how a 3.5B music model
+            # came to sit on 7.9 GB of a 3090 with the card otherwise idle. Run
+            # it BEFORE the _watch_enabled() gate so turning the leak watch off
+            # does not silently disable it too.
+            try:
+                _idle_limit = _idle_unload_seconds()
+                if _idle_limit > 0:
+                    from codai.models.manager import multi_model_manager as _mmm
+                    _mmm.unload_idle_models(_idle_limit)
+            except Exception as _iue:
+                _log.debug("idle unload sweep failed: %s", _iue)
             if not _watch_enabled():
                 continue
             cap = _cap_gb()

codai/models/teardown.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""Finding a pipeline's weights that outlived its teardown -- on the HOST side.
+
+``codai/api/video.py`` already carries a teardown-leak breaker, and it works:
+on a production host it went from reporting "16.9 GB of this pipeline's tensors
+survived teardown; nulled 7581 external reference(s)" to 0.0 GB once the
+injected components were released in the right order. But both halves of it are
+CUDA-only -- it records ``_orig_cuda_ptrs`` under ``if _tens.is_cuda`` and
+sweeps under ``if not (isinstance(_obj, Tensor) and _obj.is_cuda): continue``.
+
+A pipeline loaded with ``enable_model_cpu_offload`` keeps its weights in HOST
+RAM and streams modules to the GPU for each forward. Seventeen of twenty-one
+video loads in one production day took that path. None of those weights were
+ever tracked, so nothing looked for them after teardown. Measured on the same
+host, 2026-10-29, with nothing loaded and 482 MiB of VRAM in use:
+
+    coderai-nvidia   33.14 GB PSS_Anon
+    total anon:      30.85 GB in 6588 mappings
+      >= 1 GB:          2
+      100 MB - 1 GB:   21   (3.49 GB)
+      < 100 MB:     6,565   (20.25 GB)
+
+Thousands of separate mid-sized allocations is retained tensors, not allocator
+slack -- which is why the mitigation ladder's gc + ``malloc_trim`` could run
+every 15 s for hours without the figure moving, and why it then evicted healthy
+models to chase it.
+
+Identity, not addresses
+-----------------------
+The CUDA breaker matches surviving tensors by storage pointer. That is sound
+enough on the GPU, but a freed *host* address is readily handed back out by
+malloc, so a pointer that once belonged to this pipeline may by then belong to
+another model's tensor -- and nulling its referrers would corrupt a live model.
+So this tracks :mod:`weakref` references to the tensor objects themselves. A
+referent that is still alive after teardown IS the leaked tensor, with no
+address-reuse ambiguity, and a referent that has gone is proof the teardown
+worked.
+"""
+
+from __future__ import annotations
+
+import gc
+import weakref
+
+
+def _component_tensors(pipe):
+    """Every parameter and buffer of every component of a pipeline."""
+    comps = getattr(pipe, "components", None)
+    if not isinstance(comps, dict):
+        comps = {}
+    for comp in comps.values():
+        if comp is None or not hasattr(comp, "parameters"):
+            continue
+        try:
+            params = list(comp.parameters(recurse=True))
+        except Exception:
+            params = []
+        try:
+            bufs = list(comp.buffers(recurse=True))
+        except Exception:
+            bufs = []
+        for t in params + bufs:
+            yield t
+            # A bnb 4-bit parameter keeps its quantized payload on `.data`,
+            # which can be a different tensor object from the parameter.
+            d = getattr(t, "data", None)
+            if d is not None and d is not t:
+                yield d
+
+
+def track_host_tensors(pipe) -> list:
+    """Weak references to this pipeline's HOST-resident tensors.
+
+    Call before teardown. GPU-resident tensors are left to the existing CUDA
+    breaker, so the two do not fight over the same objects.
+    """
+    refs = []
+    seen = set()
+    for t in _component_tensors(pipe):
+        try:
+            if getattr(t, "is_cuda", False) or t.device.type != "cpu":
+                continue
+            if t.numel() == 0:
+                continue
+        except Exception:
+            continue
+        if id(t) in seen:
+            continue
+        seen.add(id(t))
+        try:
+            refs.append(weakref.ref(t))
+        except TypeError:
+            continue            # not weak-referenceable; nothing we can do
+    return refs
+
+
+def _null_referrers(obj) -> int:
+    """Clear the slots in lists/dicts that still point at ``obj``.
+
+    The same approach the CUDA breaker uses: a weight pinned by a stray entry
+    in some cache dict or list cannot be freed by dropping the pipeline, and
+    these two container kinds are what actually holds them in practice
+    (``_modules`` dicts, hook state, our own caches).
+    """
+    broken = 0
+    for ref in gc.get_referrers(obj):
+        try:
+            if isinstance(ref, list):
+                for i, item in enumerate(ref):
+                    if item is obj:
+                        ref[i] = None
+                        broken += 1
+            elif isinstance(ref, dict):
+                for k, v in list(ref.items()):
+                    if v is obj:
+                        ref[k] = None
+                        broken += 1
+        except Exception:
+            continue
+    return broken
+
+
+def break_host_leaks(refs, *, min_report_gb: float = 0.05) -> dict:
+    """Null external references to host tensors that survived teardown.
+
+    ``refs`` is what :func:`track_host_tensors` returned. Returns a report with
+    the surviving size, how many references were broken, and the biggest
+    holders, mirroring what the CUDA breaker prints so the two read alike.
+
+    Only tensors we tracked are touched, and each by identity, so another
+    model's memory is never at risk.
+    """
+    out = {"survived_gb": 0.0, "tensors": 0, "broken": 0, "holders": []}
+    if not refs:
+        return out
+    gc.collect()
+    for ref in refs:
+        t = ref()
+        if t is None:
+            continue                     # freed as it should have been
+        try:
+            gb = t.numel() * t.element_size() / 1e9
+        except Exception:
+            continue
+        out["survived_gb"] += gb
+        out["tensors"] += 1
+        out["broken"] += _null_referrers(t)
+        if gb >= min_report_gb and len(out["holders"]) < 6:
+            try:
+                out["holders"].append(f"{gb:.2f} GB {tuple(t.shape)} {t.dtype}")
+            except Exception:
+                pass
+        del t
+    gc.collect()
+    out["survived_gb"] = round(out["survived_gb"], 3)
+    return out
+
+
+def trim_host_allocator() -> bool:
+    """Ask glibc to return freed heap to the OS.
+
+    Worth doing only AFTER the references are broken: ``malloc_trim`` can
+    return nothing while the memory is still referenced, which is exactly what
+    the mitigation ladder was doing every 15 s for hours.
+    """
+    try:
+        import ctypes
+        return bool(ctypes.CDLL("libc.so.6").malloc_trim(0))
+    except Exception:
+        return False

docs/README.md

@@ -33,6 +33,7 @@ reference; these are the deep dives.
 | [zimage-lora-training.md](zimage-lora-training.md) | LoRA training for image models |
 | [text-to-music.md](text-to-music.md) | `POST /v1/audio/generate` with ACE-Step: style prompt + sung lyrics → a 48 kHz stereo track, the python-3.12 worker it needs, and the two image variants (`--with-acestep`) |
 | [dtype-auto-selection.md](dtype-auto-selection.md) | How model dtype is chosen automatically |
+| [memory-accounting.md](memory-accounting.md) | Where `measured_vram_gb` comes from: walking the loaded model (quantization needs no factor), costing the KV cache from the real GQA/sliding-window geometry, refusing a measurement from a failed load, the PSS-based host-RAM cap, idle unload, and `GET /v1/models/history` |
 | [model-defaults-and-sources.md](model-defaults-and-sources.md) | A model's own `defaults` (steps/guidance/sampler/modes/licence), the sampler names and the flow-matching boundary, and where weights come from: repo, single file, `org/repo:file` or `civitai:<versionId>` |
 
 ## Deployment

docs/memory-accounting.md

+# How coderai works out what a model needs
+
+Every placement decision — resident on the GPU, accelerate CPU offload, how
+much to evict before a load, whether to spill to disk — is made against one
+number per model. This is where that number comes from, and what it does and
+does not include.
+
+## The model is measured, not the process
+
+`measured_vram_gb` is a model's **full footprint**, and it is deliberately
+placement-independent: offload moves weights between GPU and host, it does not
+change how much the model needs. The split is reported separately.
+
+It is obtained by walking the loaded object and adding up the bytes its tensors
+really occupy (`codai/models/footprint.py`), plus the KV cache for the
+configured context, plus a runtime reserve for activations and the decode
+spike.
+
+Quantization needs no correction factor, and that is the whole reason for
+measuring the object rather than estimating from config:
+
+* a bitsandbytes `Params4bit` **is** a `uint8` tensor, so `numel ×
+  element_size` already yields the packed size;
+* GPTQ and AWQ keep `qweight` / `qzeros` / `scales` in **buffers**, so buffers
+  are walked as well as parameters — a parameters-only walk reports a 4-bit
+  model as very nearly empty;
+* bitsandbytes keeps `absmax`, the code book and (under double quantization) a
+  nested second-level state in `quant_state`, outside `parameters()`. That is
+  real memory and is counted explicitly.
+
+Tied weights — an embedding shared with the LM head — are one allocation and
+are deduplicated by storage pointer. An accelerate `meta` placeholder, left
+where the real weights were moved to host RAM or disk, is sized from its
+declared dtype and flagged, because the model still needs those bytes.
+
+A diffusers pipeline is not an `nn.Module`; it is walked through its
+components, skipping the schedulers and tokenizers that hold nothing. A
+text/vision model is registered as a `ModelManager` wrapper, so its real module
+and its config are reached through `.backend.model`.
+
+## Context is costed from the geometry
+
+The KV cache is sized from the number of **key/value** heads, not attention
+heads. Under GQA many query heads share one KV head, so sizing by attention
+heads overstates the cache by the grouping factor — fourfold on a 32/8 model.
+Sliding-window layers are sized to their window rather than the full context
+when `swa_full` is off: a model that alternates a windowed layer with a global
+one would otherwise be overstated on half its layers.
+
+**A computed KV figure is an estimate and is labelled as one.** Per-layer
+KV-head arrays, shared KV and architecture-specific cache layouts are not
+modelled, and predicting this from GGUF metadata has been wrong by 5× in both
+directions on these very models. A measured `measured_kv_gb` for the context in
+question — llama.cpp reports its own KV buffer size exactly — always wins.
+
+## What a GGUF needs
+
+A GGUF is loaded as stored, with no dtype conversion, so its weights **are** the
+file, byte for byte. llama.cpp owns that memory and there is no torch object to
+walk, so the file size is used, plus the KV cache and the reserve. A companion
+vision projector (`mmproj`) loads on top and is added.
+
+## A measurement that cannot be trusted is refused
+
+`force_vram_update` refreshes these fields on every load, so a bad reading
+becomes authoritative and is planned against. Three checks stand in the way:
+
+1. **An empty load.** If the object is walkable and holds no weights at all,
+   the load did not complete. There is nothing to fall back to — a size taken
+   from process memory measures everything *except* this model — so nothing is
+   recorded, and the refusal is written to the history with its reason.
+2. **An implausible size.** No real model is under ~10 MB.
+3. **A collapse against the previous reading.** Less than half of what the
+   *same quantization signature* measured before is a partial load. A genuine
+   drop arrives together with a signature change — adding 4-bit quantization
+   to a model's transformers took one from 40.04 GB to 11.73 GB — and is let
+   through. The signature covers precision, `load_in_4bit`, `load_in_8bit` and
+   `component_quantization`.
+
+The log names which check fired, because the three call for opposite responses.
+
+## Why not process memory
+
+The footprint used to be a process-RSS delta across the load. RSS is the wrong
+instrument: it counts file-backed pages that the kernel maps when the loader
+reads a mmap'd safetensors or GGUF, and summed over a process tree it counts
+pages **shared** between those processes once per process — the interpreter,
+torch and the CUDA runtime are mapped into every engine. As a delta it also
+absorbs anything else that loaded, evicted or faulted pages in the same window.
+
+Measured against ground truth on a 5-engine host:
+
+```
+gemma-2-9b-it.Q4_K_M    5.37 GB on disk  ->  reported 30.81 GB   (5.7x)
+LTX-2.5, six loads, one unchanged config:
+    22.11, 22.11, 23.28, 4.50, 19.02, 29.80 GB                   (6.6x)
+```
+
+Every other model on that host sat at 1.0–1.2×. The 4.50 GB reading came from a
+load that placed a single layer and then raised; `Full-GPU load looks viable
+(4.5 GB peak need)` was planned off it four seconds later.
+
+## The host-RAM cap
+
+`offload.max_ram_gb` is enforced against proportional set size
+(`Pss_Anon + Pss_Shmem`, from `/proc/<pid>/smaps_rollup`), summed over the
+front, the engines and their workers. PSS has already divided each shared page
+by the number of processes sharing it, so summing it across a tree counts every
+page exactly once; `Pss_File` is excluded because mapped file pages are page
+cache that the kernel reclaims by itself and that evicting a model cannot free.
+
+The cap can only be met by releasing memory something actually holds, so the
+metric has to be one that releasing memory moves.
+
+## Idle unload
+
+Eviction frees a model when something else needs the memory.
+`offload.idle_unload_seconds` (default 1800) covers the case where nothing
+does: the on-demand path keeps a model resident whenever there is room, which
+left a 3.5B music model on 7.9 GB of a 24 GB card for hours after its last
+request.
+
+It is deliberately generous. Reloads are the expensive event — one production
+day spent 2h04m of 21h30m inside model loads — so a short timeout trades a VRAM
+saving nobody asked for against a reload everybody waits on. A model whose LRU
+clock was never stamped is left alone rather than assumed idle.
+
+## Seeing it happen
+
+`GET /v1/models/history` returns what was loaded and evicted and why, newest
+first, merged across engines — a swap is a cross-engine event, since one engine
+evicts so a co-located one can load.
+
+```
+time, event ('load' | 'evict'), model, engine
+  load:  seconds, vram_gb, offloaded_gb, strategy, measured
+  evict: reason, freed_gb, expected_gb
+```
+
+It is in-memory and bounded (500 entries), so it is empty after a restart. It
+exists because the alternative was reconstructing swaps from the log, and the
+log rotates: on one host 78% of it was per-request debug, which filled the
+2,000-line view in about four minutes.

docs/text-to-music.md

@@ -17,7 +17,7 @@ curl -s http://localhost:8000/v1/audio/generate \
 | field | ACE-Step default | notes |
 |---|---|---|
 | `prompt` | — | the *style*, not the words: genre, instruments, tempo, mood |
-| `lyrics` | `""` | supports its `[verse]` / `[chorus]` tags; `[inst]` for an instrumental |
+| `lyrics` | `[instrumental]` | supports its `[verse]` / `[chorus]` tags; send words to get vocals |
 | `duration` | 60 s | the whole track is generated at once, not stitched |
 | `steps` | 60 | diffusion steps |
 | `cfg_coef` | 15.0 | guidance; ACE-Step wants far more of it than MusicGen |
@@ -31,6 +31,23 @@ them out gets **ACE-Step's** defaults instead, because a 10-second song at a
 guidance it was not tuned for is not a useful reading of "unspecified". A value
 you actually send is always used, including `duration: 10`.
 
+## No lyrics means instrumental, and it has to be said
+
+ACE-Step is a *song* model: the prompt carries the style and the lyrics carry
+the words. Leaving `lyrics` empty does **not** ask for an instrumental — the
+pipeline then conditions on a single padding token (`lyric_token_idx = [0]`,
+`lyric_mask = [0]`) and the model still tries to sing, which comes out as
+babbling over an otherwise reasonable backing track. ACE-Step's own interface
+says how to ask: *"Use `[instrumental]` or `[inst]` to generate instrumental
+music"*, and every genre preset it ships ends in either `male vocals, raw
+vocals` or `instrumental`.
+
+So a request with no `lyrics` is sent as `[instrumental]`, and `instrumental`
+is appended to the style prompt when it is not already there. Background music
+for a video — the common case, where the caller has a mood to describe and no
+words — therefore comes out instrumental by default. To get vocals, send
+lyrics.
+
 ## Why it runs in its own interpreter
 
 ACE-Step ships its own `acestep` package, which pins `transformers==4.50` and

tests/test_host_teardown_leak.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""Host-resident pipeline weights must be released on teardown too.
+
+The existing teardown-leak breaker in video.py is CUDA-only on both sides: it
+registers under ``if _tens.is_cuda`` and sweeps under ``if not (... and
+_obj.is_cuda): continue``. A pipeline loaded with ``enable_model_cpu_offload``
+keeps its weights in HOST RAM, and 17 of 21 video loads in one production day
+took that path, so none of those weights were tracked and nothing looked for
+them afterwards.
+
+Measured 2026-10-29 with nothing loaded and 482 MiB of VRAM in use:
+
+    coderai-nvidia  33.14 GB PSS_Anon
+    30.85 GB anon in 6588 mappings; 6565 of them under 100 MB (20.25 GB)
+
+Thousands of mid-sized allocations is retained tensors, not allocator slack --
+so gc + malloc_trim every 15 s never moved it, and the ladder evicted healthy
+models instead.
+"""
+
+import gc
+import io
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from codai.models import teardown as T
+
+torch = pytest.importorskip("torch")
+nn = torch.nn
+
+
+def _pipe(**comps):
+    p = type("Pipe", (), {})()
+    p.components = comps
+    return p
+
+
+def test_host_tensors_are_tracked():
+    pipe = _pipe(transformer=nn.Linear(64, 64), scheduler=object(), tokenizer=None)
+    refs = T.track_host_tensors(pipe)
+    assert refs, "no host tensors tracked for a CPU-resident pipeline"
+
+
+def test_gpu_tensors_are_left_to_the_cuda_breaker():
+    """The two arms must not both chase the same objects."""
+    if not torch.cuda.is_available():
+        pytest.skip("needs CUDA")
+    pipe = _pipe(transformer=nn.Linear(64, 64).cuda())
+    assert T.track_host_tensors(pipe) == []
+
+
+def test_a_clean_teardown_reports_nothing_surviving():
+    pipe = _pipe(transformer=nn.Linear(256, 256))
+    refs = T.track_host_tensors(pipe)
+    del pipe
+    gc.collect()
+    r = T.break_host_leaks(refs)
+    assert r["survived_gb"] == 0.0
+    assert r["tensors"] == 0
+    assert r["broken"] == 0
+
+
+def test_a_tensor_pinned_by_a_dict_is_found_and_released():
+    """The real shape of the leak: a stray cache entry outlives the pipeline
+    and pins the weights, so dropping the pipeline frees nothing."""
+    pipe = _pipe(transformer=nn.Linear(512, 512))
+    stray_cache = {"weight": pipe.components["transformer"].weight}
+    refs = T.track_host_tensors(pipe)
+    del pipe
+    gc.collect()
+
+    r = T.break_host_leaks(refs)
+    assert r["tensors"] >= 1, "the surviving host tensor was not detected"
+    assert r["broken"] >= 1, "the reference pinning it was not broken"
+    assert stray_cache["weight"] is None, "the cache still pins the weight"
+
+
+def test_a_tensor_pinned_by_a_list_is_released():
+    pipe = _pipe(transformer=nn.Linear(256, 256))
+    stray = [pipe.components["transformer"].weight]
+    refs = T.track_host_tensors(pipe)
+    del pipe
+    gc.collect()
+    T.break_host_leaks(refs)
+    assert stray[0] is None
+
+
+def test_the_surviving_size_is_reported():
+    pipe = _pipe(transformer=nn.Linear(1024, 1024, bias=False).to(torch.float32))
+    held = {"w": pipe.components["transformer"].weight}
+    refs = T.track_host_tensors(pipe)
+    del pipe
+    gc.collect()
+    r = T.break_host_leaks(refs)
+    # reported to 3 dp, i.e. 1 MB granularity
+    assert r["survived_gb"] == pytest.approx(1024 * 1024 * 4 / 1e9, abs=0.001)
+    assert held["w"] is None
+
+
+def test_only_tracked_tensors_are_touched():
+    """Another model's memory must never be at risk. This is why the host arm
+    tracks weakrefs to the objects instead of matching storage addresses: a
+    freed host address is readily reused by malloc, and a pointer match would
+    then null a live tensor belonging to something else."""
+    victim = nn.Linear(128, 128)
+    bystander = {"w": victim.weight}
+    other = _pipe(transformer=nn.Linear(128, 128))
+    refs = T.track_host_tensors(other)
+    del other
+    gc.collect()
+    T.break_host_leaks(refs)
+    assert bystander["w"] is victim.weight, "an untracked tensor was nulled"
+
+
+def test_buffers_are_tracked_not_just_parameters():
+    """Pre-quantized formats keep their payload in buffers, so a
+    parameters-only sweep would leave a 4-bit model's weights pinned."""
+    m = nn.Module()
+    m.register_buffer("qweight", torch.empty((256, 64), dtype=torch.int32))
+    pipe = _pipe(transformer=m)
+    held = {"q": m.qweight}
+    refs = T.track_host_tensors(pipe)
+    del pipe, m
+    gc.collect()
+    r = T.break_host_leaks(refs)
+    assert r["tensors"] >= 1
+    assert held["q"] is None
+
+
+def test_the_cuda_arm_still_only_tracks_cuda():
+    """Guard the division of labour in video.py: if the CUDA registration ever
+    starts taking host tensors too, the two arms would double-handle them."""
+    src = io.open(os.path.join(os.path.dirname(os.path.dirname(
+        os.path.abspath(__file__))), "codai/api/video.py"), encoding="utf-8").read()
+    assert "_orig_cuda_ptrs.add" in src
+    reg = src.split("_orig_cuda_ptrs.add", 1)[0][-400:]
+    assert "is_cuda" in reg, (
+        "the CUDA breaker no longer gates registration on is_cuda")
+
+
+def test_trim_is_available_and_safe_to_call():
+    assert T.trim_host_allocator() in (True, False)

tests/test_idle_unload_history_and_log_volume.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""Four things a production day asked for.
+
+Measured on one host over 21h30m, 2026-10-29:
+
+* a 3.5B music model held 7.9 GB of a 3090 and 2.1 GB of host RAM for hours
+  after its last request, because the on-demand path keeps a model resident
+  whenever there is room ("Ondemand mode - keeping 'audio_gen:ACE-Step' in VRAM
+  alongside new model");
+* 78% of the log (426,682 of 548,687 lines) was per-request debug from one
+  engine, filling the 2,000-line view in about four minutes and leaving no
+  record of the loads and evictions behind a slow render;
+* those loads and evictions could only be inferred from that log;
+* ACE-Step was asked for a song with no lyrics instead of an instrumental.
+"""
+
+import io
+import os
+import sys
+import time
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+# ------------------------------------------------------------ idle unload ---
+
+class _FakeManager:
+    """Only what unload_idle_models touches."""
+
+    from codai.models.manager import MultiModelManager as _MM
+    unload_idle_models = _MM.unload_idle_models
+    record_history = _MM.record_history
+    get_history = _MM.get_history
+
+    def __init__(self):
+        from collections import deque
+        self.models = {}
+        self._last_used = {}
+        self._keep_resident = set()
+        self._history = deque(maxlen=500)
+        self.evicted = []
+        self.busy = set()
+
+    def _lru_order(self):
+        return sorted(self.models, key=lambda k: self._last_used.get(k, 0.0))
+
+    def _is_key_busy(self, key):
+        return key in self.busy
+
+    def _evict_one(self, key, reason=""):
+        self.evicted.append((key, reason))
+        self.models.pop(key, None)
+        self._last_used.pop(key, None)
+
+
+def _mgr(**ages):
+    m = _FakeManager()
+    now = time.monotonic()
+    for key, age in ages.items():
+        m.models[key] = object()
+        m._last_used[key] = now - age
+    return m
+
+
+def test_an_idle_model_is_unloaded():
+    m = _mgr(**{"audio_gen:ACE-Step": 3600.0})
+    freed = m.unload_idle_models(1800.0)
+    assert freed == ["audio_gen:ACE-Step"]
+    assert m.evicted and "idle" in m.evicted[0][1]
+
+
+def test_a_recently_used_model_is_kept():
+    m = _mgr(**{"video:X": 60.0})
+    assert m.unload_idle_models(1800.0) == []
+    assert m.models
+
+
+def test_a_busy_model_is_never_pulled():
+    m = _mgr(**{"video:X": 9999.0})
+    m.busy.add("video:X")
+    assert m.unload_idle_models(1800.0) == []
+
+
+def test_keep_resident_is_respected():
+    m = _mgr(**{"tts:Kokoro": 9999.0})
+    m._keep_resident.add("tts:Kokoro")
+    assert m.unload_idle_models(1800.0) == []
+
+
+def test_an_unstamped_model_is_left_alone():
+    """A key with no _last_used is indistinguishable from one loaded a second
+    ago by a path that forgot to stamp it, so unloading it would be the
+    opposite of what is wanted."""
+    m = _FakeManager()
+    m.models["video:X"] = object()          # never stamped
+    assert m.unload_idle_models(1800.0) == []
+
+
+def test_zero_disables_it():
+    m = _mgr(**{"video:X": 99999.0})
+    assert m.unload_idle_models(0) == []
+    assert m.unload_idle_models(None) == []
+
+
+def test_the_setting_exists_and_is_plumbed():
+    from codai.config import OffloadConfig
+    assert hasattr(OffloadConfig(), "idle_unload_seconds")
+    main = io.open(os.path.join(_ROOT, "codai/main.py"), encoding="utf-8").read()
+    assert "global_args.idle_unload_seconds" in main, (
+        "the setting never reaches the running server")
+    rm = io.open(os.path.join(_ROOT, "codai/models/ram_monitor.py"), encoding="utf-8").read()
+    assert "unload_idle_models" in rm, "nothing ever runs the idle sweep"
+
+
+# ---------------------------------------------------------------- history ---
+
+def test_history_records_a_load_with_what_it_cost():
+    m = _FakeManager()
+    m.record_history("load", "video:X", seconds=44.2, vram_gb=15.68,
+                     strategy="resident")
+    row = m.get_history()[0]
+    assert row["event"] == "load" and row["model"] == "video:X"
+    assert row["seconds"] == 44.2 and row["vram_gb"] == 15.68
+    assert row["time"] > 0
+
+
+def test_history_records_why_a_model_was_evicted():
+    m = _FakeManager()
+    m.record_history("evict", "video:X", reason="cosite: a co-located engine "
+                                                "needed the GPU", freed_gb=11.1)
+    row = m.get_history()[0]
+    assert "co-located" in row["reason"]
+    assert row["freed_gb"] == 11.1
+
+
+def test_history_is_newest_first_and_filterable():
+    m = _FakeManager()
+    m.record_history("load", "a")
+    m.record_history("evict", "b")
+    assert [r["model"] for r in m.get_history()] == ["b", "a"]
+    assert [r["model"] for r in m.get_history(event="load")] == ["a"]
+
+
+def test_history_is_bounded():
+    """It must not become the leak it is there to help diagnose."""
+    m = _FakeManager()
+    for i in range(1200):
+        m.record_history("load", f"m{i}")
+    assert len(m._history) == 500
+
+
+def test_eviction_reasons_are_passed_at_the_call_sites():
+    src = io.open(os.path.join(_ROOT, "codai/models/manager.py"), encoding="utf-8").read()
+    for marker in ("reason=f\"vram:", "reason=\"cosite:", "reason=f\"idle for"):
+        assert marker in src, f"no reason given at the {marker!r} call site"
+
+
+def test_the_endpoints_exist():
+    eng = io.open(os.path.join(_ROOT, "codai/api/app.py"), encoding="utf-8").read()
+    assert '"/v1/models/history"' in eng
+    front = io.open(os.path.join(_ROOT, "codai/frontproxy/app.py"), encoding="utf-8").read()
+    assert '"/v1/models/history"' in front, (
+        "the front does not aggregate it, so a caller sees one engine's half of "
+        "a cross-engine swap")
+    # It must be registered BEFORE the catch-all, or it is proxied to one engine.
+    assert front.index('"/v1/models/history"') < front.index("/admin/api/engines")
+
+
+# ------------------------------------------------------------- log volume ---
+
+def test_embeddings_are_summarised_not_dumped():
+    from codai.api import log as L
+    assert "/v1/embeddings" in L._SUMMARY_ONLY_PATHS
+    assert "/v1/embeddings" in L._TRACKED_PATHS, (
+        "summarising must not stop the admin page counting them")
+
+
+def test_the_summary_says_what_was_asked_without_the_content():
+    from codai.api import log as L
+    secret = "a client's private property listing " * 20
+    line = L._summarise({"model": "Qwen3-Embedding-4B", "input": [secret, secret],
+                         "dimensions": 1024})
+    assert "model=Qwen3-Embedding-4B" in line
+    assert "inputs=2" in line
+    assert "dimensions=1024" in line
+    assert "property listing" not in line, "the caller's content is still in the log"
+
+
+def test_a_prompt_is_left_intact():
+    """Deliberately NOT truncated: a prompt is the most useful thing in the log
+    -- a 4,000-character script prompt was what identified a music defect --
+    and the volume problem was one per-item endpoint, not prompts."""
+    from codai.api import log as L
+    # Real prose, not a run of base64-legal characters: a long unbroken token
+    # IS treated as a blob, and correctly so.
+    prompt = ("Write a 81-second video script about the rally. " * 100)
+    out = L._redact_blobs({"prompt": prompt})
+    assert out["prompt"] == prompt, "a script prompt is being truncated"
+    # A base64 payload is still redacted.
+    blob = L._redact_blobs({"init_image": "A" * 5000})["init_image"]
+    assert "truncated" in blob
+
+
+def test_a_short_value_is_untouched():
+    from codai.api import log as L
+    assert L._redact_blobs({"model": "Krea-2-Turbo"}) == {"model": "Krea-2-Turbo"}
+
+
+# ------------------------------------------------------- acestep lyrics ----
+
+def test_an_instrumental_request_says_so():
+    """ACE-Step is a SONG model. Empty lyrics leave it conditioning on a single
+    padding token (lyric_token_idx = [0], lyric_mask = [0]) and it still tries
+    to sing. Its own UI documents the way to ask: "Use [instrumental] or [inst]
+    to generate instrumental music"."""
+    src = io.open(os.path.join(_ROOT, "codai/api/audio_gen.py"), encoding="utf-8").read()
+    block = src.split("elif model_type == 'ace_step':", 1)[1][:2000]
+    assert "[instrumental]" in block, (
+        "an empty-lyrics request is still sent as a song with no words")
+    assert "lyrics=_lyrics" in block
+    assert "instrumental" in block.split("_prompt")[1][:400], (
+        "the tag prompt is not told it is instrumental either")

tests/test_ltx2_audio_and_engines.py

@@ -80,7 +80,11 @@ def test_stable_audio_and_ace_step_each_have_their_own_loader():
     assert "_load_musicgen" not in load
     # ...and the generate branch returns the worker's wav rather than falling
     # through to the array-to-int16 writer, which would see no audio_np at all.
-    gen = src[src.index("elif model_type == 'ace_step':"):][:1400]
+    # Slice to the NEXT branch rather than a character count: a fixed window
+    # silently stops covering the branch as soon as anything is added to it.
+    _start = src.index("elif model_type == 'ace_step':")
+    _end = src.index("elif model_type ==", _start + 10)
+    gen = src[_start:_end]
     assert "return wav, 'wav'" in gen