Six findings from one 21h30m production log (548,687 lines, 51 loads of 12
models). They share a cause: nothing measured a model directly, so every
decision downstream of a measurement was made on a number that was wrong.

1. The footprint was a process-RSS delta
---------------------------------------
`measured_ram = self._get_own_ram_gb() - ram_before`, where _get_own_ram_gb
sums RSS over the engine and every child. RSS counts mmap'd weight-file pages
the loader just walked; summed over a tree it counts pages SHARED between those
processes once per process; and as a delta it absorbs anything else that moved
in the window. Against ground truth:

    gemma-2-9b-it.Q4_K_M   5.37 GB on disk  ->  reported 30.81 GB   (5.7x)
    LTX-2.5, six loads, one unchanged config:
        22.11, 22.11, 23.28, 4.50, 19.02, 29.80 GB                  (6.6x)

Every other model sat at 1.0-1.2x, so this was not noise. Wan2.2's 3.4x spread
was the real quantization fix and is the one that belonged there.

codai/models/footprint.py measures the object instead. Quantization needs no
correction factor, which is the point: a bitsandbytes Params4bit IS a uint8
tensor, so numel x element_size already gives packed bytes. What those formats
hide is counted explicitly -- bnb's quant_state (absmax, code book, nested
double-quant state) lives outside parameters(), and GPTQ/AWQ keep
qweight/qzeros/scales in BUFFERS, so a parameters-only walk reports a 4-bit
model as nearly empty. Tied weights are deduped by storage pointer; accelerate
meta placeholders are sized from their dtype and flagged as declared, keeping
the placement-independence the field already documents; a diffusers pipeline is
walked through its components, since it is not an nn.Module.

Context is costed properly: KV from the KEY/VALUE head count, not the attention
head count (4x on a GQA model), with sliding-window layers sized to their
window. But a computed KV is only a fallback and says so -- this codebase
already learned that "predicting KV from GGUF metadata is architecture-specific
... and was off by 5x in BOTH directions", so measured_kv_gb for the context in
question always wins. A GGUF, which no walk can reach, falls back to its file
size; it is loaded as stored, so the file IS the weights.

2. A failed load overwrote the good measurement
-----------------------------------------------
    Device map: cpu: 1 layers
    Measured footprint: 4.50 GB total
    Saved measured_vram_gb=4.505 (force_vram_update)
    generation FAILED: StopIteration interacts badly with generators
    Full-GPU load looks viable (4.5 GB peak need)   <- 4 seconds later

A load that placed one layer and raised still reached the recorder, and
force_vram_update wrote it to disk as truth. Now a measurement that is
implausible on its own (~0 GB) or that collapses to under half of what the SAME
quantization signature previously measured is treated as a partial load and
discarded. Wan2.2's genuine 40.04 -> 11.73 GB drop is let through because it
arrived WITH a signature change, which is exactly what tells the two apart.
The rejection names which of the two checks fired.

The ratio guard alone was not enough. Walking a pipeline and finding that NO
component holds any weights is a failed load outright, and there is nothing to
fall back to -- a size taken from process memory measures everything except
the model in question. That case is now refused before any estimate is
considered, and the refusal is recorded in the history with its reason rather
than left looking like a successful load. The hole showed up as one of the new
tests passing alone and failing in the full suite: the fallback delta grows
with whatever else the process has loaded, so the figure it invented depended
on test order.

3. The teardown-leak breaker never looked at host RAM
-----------------------------------------------------
It registers under `if _tens.is_cuda` and sweeps under `_obj.is_cuda`. A
pipeline loaded with enable_model_cpu_offload keeps its weights in HOST RAM,
and 17 of 21 video loads took that path, so none of those weights were tracked
and nothing looked for them. With nothing loaded and 482 MiB of VRAM in use:

    coderai-nvidia   33.14 GB PSS_Anon
    30.85 GB anon in 6588 mappings; 6565 of them under 100 MB (20.25 GB)

Thousands of mid-sized allocations is retained tensors, not allocator slack --
which is why gc and malloc_trim ran every 15 s for hours with the figure never
moving, and why the ladder then evicted healthy models to chase it:

    RAM eviction: unloading 'video:Lightricks/LTX-2.5-Diffusers' to free host
    RAM (RSS 40.0 GB > cap target 36.5 GB)

codai/models/teardown.py adds the host arm, by weakref rather than by storage
pointer: a freed host address is readily handed back out by malloc, so a
pointer match could null a live tensor that by then belongs to another model. A
referent still alive after teardown IS this pipeline's leak, identity-exact. It
runs before the allocator trim, since trimming while the weights are still
referenced returns nothing.

4. The RAM cap counted shared pages and page cache
--------------------------------------------------
codai/models/proc_mem.py reads Pss_Anon + Pss_Shmem from smaps_rollup, which
has already divided shared pages by their sharers and excludes reclaimable page
cache. Measured correction on a 5-engine host: 40.06 GB (RSS) -> 37.91 GB
(PSS). Modest -- the leak above was the real problem -- but a cap should chase
memory that freeing an allocation can actually release. Falls back to the old
reading rather than to 0.0, which would read as "no memory in use".

5. A model nobody used stayed loaded
------------------------------------
    Ondemand mode - keeping 'audio_gen:ACE-Step' in VRAM alongside new model

It held 7.9 GB of a 3090 and 2.1 GB of host RAM for hours after its last
request. offload.idle_unload_seconds (default 1800) unloads a model untouched
that long. Deliberately generous: reloads are the expensive event -- that day
spent 2h04m of 21h30m inside loads -- so a short timeout would trade a VRAM
saving nobody asked for against a reload everybody waits on. A key that was
never stamped is left alone rather than assumed idle. It runs before the
leak-watch gate, so turning that off does not silently disable this.

6. The log left no room for the history anyone needed
-----------------------------------------------------
32,633 of 33,009 requests were embeddings, and one engine produced 426,682 of
the 548,687 lines -- 78% -- filling the 2,000-line view in about four minutes.
The per-item endpoints are now summarised to one line (model, count, shape)
instead of dumped, which also stops a caller's content being written to a
shared log. Prompts are deliberately NOT truncated: a 4,000-character script
prompt is what identified the music defect below.

GET /v1/models/history replaces inferring swaps from a log that rotates: time,
event, model, engine, plus seconds, vram_gb, offloaded_gb, strategy for a load
and reason, freed_gb for an eviction. Eviction reasons are passed at the call
sites. The front merges across engines, because a swap is a cross-engine event
-- one engine evicts so a co-located one can load -- so one engine's copy is
half the story.

Also: ACE-Step was asked for a song with no words
-------------------------------------------------
Motus sends a music_prompt and no lyrics, and empty lyrics do not mean
instrumental: the pipeline then conditions on a single padding token
(lyric_token_idx = [0], lyric_mask = [0]) and the model still tries to sing,
which comes out as babbling over the backing track. ACE-Step's own UI documents
the way to ask -- "Use [instrumental] or [inst] to generate instrumental
music" -- and every genre preset it ships ends in either "male vocals, raw
vocals" or "instrumental". Both are now sent when no lyrics are given.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC