quantize what load_in_4bit was asked to quantize, and say why a load spills

component_quantization replaced the global flag instead of refining it, so
naming ONE component switched load_in_4bit off for every other one. Measured
2026-10-29 on LTX-2.5 (load_in_4bit=True, map naming transformer and
text_encoder): ten of its twelve components loaded at full precision, among
them connectors at 11.82 GB and prompt_enhancer at 9.51 GB -- 21.3 GB of a
25.3 GB card before the quantized transformer arrived, which is why the
resident attempt OOMed on exactly those components.

The candidate list was the second half of it. Written for unet/SD-era
pipelines, it matched only transformer*/unet/text_encoder*, so connectors and
prompt_enhancer could not be reached by the global flag either. A modern video
pipeline carries several multi-GB Linear-heavy parts under other names.

Both builders are fixed together -- build_pipeline_quant_config and the
incremental component cache -- because they had the same shape, and a cached
build that quantizes differently from a direct load is its own bug. An
explicit map entry still wins, including "none"/"full", which is how a
component is opted out.

Wan2.2-TI2V-5B shows the size of this: load_in_4bit=True with a map naming
only text_encoder and vae, so its transformer was never quantized at all --
a 5B model measuring 40.0 GB on a 25.3 GB card.

Separately, the message printed when eviction cannot make room claimed
"Remaining models are busy or VRAM is held elsewhere" without having
established either, and it was usually false. Over one production day, 10 of
12 of these were a model asking for more than the whole card (25.4 to 40.0 GB
against 25.3 GB), four of them printing "freed 0.0 GB (now 25.3 GB free)" --
the card was already empty. "Skipping eviction of ... still busy" never
appeared once: eviction was never blocked by a busy model.

That wording sent an investigation at the scheduler when the causes were a
model larger than the GPU and components that were never quantized. It now
names which of the three cases applies, and for the common one points at the
two things that actually help.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/models/hf_loading.py

@@ -150,6 +150,28 @@ def _discover_components(model_name: str) -> Dict[str, Any]:
     return out
 
 
+#: Components worth quantizing beyond the unet/SD-era shortlist.
+#: A modern video pipeline carries several multi-GB, Linear-heavy parts whose
+#: names start with neither "transformer" nor "text_encoder", so they were
+#: never candidates. Measured 2026-10-29 on LTX-2.5: connectors (11.82 GB) and
+#: prompt_enhancer (9.51 GB) between them outweigh its quantized transformer,
+#: and both loaded at full precision -- 21.3 GB of a 25.3 GB card before the
+#: transformer arrived.
+_ALSO_QUANTIZABLE = ('connectors', 'prompt_enhancer', 'image_encoder',
+                     'text_encoder_2', 'text_encoder_3')
+
+
+def _is_quantizable_component(name: str) -> bool:
+    """Whether this component is a candidate for the GLOBAL quantization flag.
+
+    An explicit entry in ``component_quantization`` always wins over this; the
+    list only decides what ``load_in_4bit`` reaches on its own.
+    """
+    n = (name or '').lower()
+    return (n.startswith('transformer') or n == 'unet'
+            or n.startswith('text_encoder') or n in _ALSO_QUANTIZABLE)
+
+
 def build_pipeline_quant_config(model_name: str, cfg: Optional[Dict[str, Any]], dtype):
     """Build a diffusers PipelineQuantizationConfig from a per-model config.
 
@@ -279,11 +301,20 @@ def build_pipeline_quant_config(model_name: str, cfg: Optional[Dict[str, Any]],
             if cfg_obj is not None:
                 quant_mapping[name] = cfg_obj
                 descs.append(f"{name}:{mode}")
-    else:
+    # The map REFINES the global flag; it does not replace it. Listing one
+    # component used to silence load_in_4bit for every OTHER component, which
+    # is the opposite of what setting both plainly means. Measured 2026-10-29
+    # on LTX-2.5 (load_in_4bit=True, map naming transformer + text_encoder):
+    # ten of its twelve components loaded at full precision, among them
+    # connectors at 11.82 GB and prompt_enhancer at 9.51 GB. An explicit
+    # "none"/"full" in the map still opts a component out.
+    if global_4 or global_8:
         mode = '4bit' if global_4 else '8bit'
-        targets = [n for n in comp_lib if _is_heavy(n)] or \
+        targets = [n for n in comp_lib if _is_quantizable_component(n)] or \
             ['transformer', 'transformer_2', 'text_encoder', 'unet']
         for name in targets:
+            if name in quant_mapping or name in comp_q:
+                continue          # the map already had its say for this one
             if _bnb_incompatible(name):
                 continue
             cfg_obj = _mk(comp_lib.get(name, 'diffusers'), mode)
@@ -1190,10 +1221,15 @@ def build_cached_components(model_name: str, cfg: Optional[Dict[str, Any]], dtyp
             mode = _normalize_quant_mode(raw)  # gguf/none/2bit -> not bnb here
             if mode in ('4bit', '8bit') and not _bnb_incompatible(name) and name in specs:
                 targets[name] = mode
-    else:
+    # Same rule as build_pipeline_quant_config: the map refines the global
+    # flag rather than replacing it, so a cached build quantizes exactly what
+    # a direct load would.
+    if global_4 or global_8:
         mode = '4bit' if global_4 else '8bit'
         for name in specs:
-            if _is_heavy(name) and not _bnb_incompatible(name):
+            if name in targets or name in comp_q:
+                continue
+            if _is_quantizable_component(name) and not _bnb_incompatible(name):
                 targets[name] = mode
     if not targets:
         return {}, ''

codai/models/manager.py

@@ -5533,10 +5533,37 @@ class MultiModelManager:
         # Report what we actually freed so a failure to free is visible.
         _free_after = self._get_free_vram_gb()
         if _free_after < needed_gb:
+            # Say WHICH of the three it was. The old wording blamed "busy
+            # models or VRAM held elsewhere" unconditionally, and that was
+            # usually false: measured over one production day, 10 of 12 of
+            # these were a model asking for more than the entire card (25.4 to
+            # 40.0 GB against 25.3 GB usable) and FOUR of them printed
+            # "freed 0.0 GB (now 25.3 GB free)" -- i.e. the card was already
+            # completely empty. Not one was blocked by a busy model.
+            #
+            # That matters because the message is the first thing anyone reads
+            # when a model lands on the CPU, and it pointed at the scheduler
+            # when the real causes were an under-declared size and a model
+            # larger than the GPU.
+            _total_gb = 0.0
+            try:
+                _total_gb = self._total_vram_gb()
+            except Exception:
+                _total_gb = 0.0
+            if _total_gb and needed_gb > _total_gb:
+                _why = (f"this model needs more than the whole card "
+                        f"({needed_gb:.1f} GB needed, {_total_gb:.1f} GB "
+                        f"total) — eviction cannot help; quantize it further "
+                        f"or correct used_vram_gb if that figure is wrong")
+            elif _free_after >= _free_before + 0.1:
+                _why = (f"freed what it could; {needed_gb - _free_after:.1f} GB "
+                        f"short, held by a busy model or another process")
+            else:
+                _why = ("nothing could be freed; either the card is already "
+                        "empty or every other owner is outside this engine")
             print(f"  ⚠ Eviction freed {_free_after - _free_before:.1f} GB "
-                  f"(now {_free_after:.1f} GB free, needed {needed_gb:.1f} GB). "
-                  f"Remaining models are busy or VRAM is held elsewhere — the new "
-                  f"model will load with CPU/disk offload.")
+                  f"(now {_free_after:.1f} GB free, needed {needed_gb:.1f} GB): "
+                  f"{_why} — the new model will load with CPU/disk offload.")
 
     def release_idle_vram(self, needed_gb: float = 0.0,
                           wait_for_busy: bool = False,

tests/test_eviction_message_names_the_cause.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""When a load falls back to CPU, say WHY it really did.
+
+The message printed whenever eviction cannot make room is the first thing
+anyone reads when a model lands on the CPU, and it used to blame "remaining
+models are busy or VRAM is held elsewhere" unconditionally.
+
+Measured over one production day, that was wrong 10 times out of 12: the model
+had asked for more than the entire card (25.4 to 40.0 GB against 25.3 GB
+usable), and FOUR of those printed "freed 0.0 GB (now 25.3 GB free)" -- the
+card was already completely empty. Not one of the twelve was blocked by a busy
+model; "Skipping eviction of ... still busy" never appeared at all.
+
+The wording sent the investigation at the scheduler when the real causes were
+an under-declared used_vram_gb and a model bigger than the GPU.
+"""
+
+import io
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+_SRC = io.open(os.path.join(os.path.dirname(os.path.dirname(
+    os.path.abspath(__file__))), "codai/models/manager.py"), encoding="utf-8").read()
+
+_BLOCK = _SRC.split("⚠ Eviction freed", 1)[0][-3000:] + \
+    _SRC.split("⚠ Eviction freed", 1)[1][:1500]
+
+
+def test_it_no_longer_blames_busy_models_unconditionally():
+    tail = _SRC.split("⚠ Eviction freed", 1)[1][:1200]
+    assert "Remaining models are busy or VRAM is held elsewhere — the new" not in tail, (
+        "the message still asserts a cause it has not established")
+
+
+def test_a_model_bigger_than_the_card_is_named_as_such():
+    """The dominant real case: eviction cannot possibly help."""
+    assert "needs more than the whole card" in _SRC, (
+        "the commonest cause -- a model larger than the GPU -- is not reported")
+
+
+def test_the_three_cases_are_distinguished():
+    before = _SRC.split("⚠ Eviction freed", 1)[0][-2000:]
+    assert "if _total_gb and needed_gb > _total_gb:" in before
+    assert "elif _free_after >= _free_before" in before
+    assert "else:" in before
+
+
+def test_it_uses_a_total_vram_accessor_that_exists():
+    """A misspelled accessor would fall into the except and silently report
+    the wrong case every time -- the exact failure this message is meant to
+    stop."""
+    assert "self._total_vram_gb()" in _SRC
+    assert "self._get_total_vram_gb()" not in _SRC, (
+        "that method does not exist; the call would always raise and the "
+        "branch would never be taken")
+    import ast
+    tree = ast.parse(_SRC)
+    names = {n.name for n in ast.walk(tree)
+             if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))}
+    assert "_total_vram_gb" in names
+
+
+def test_it_suggests_something_actionable():
+    """A model larger than the card is fixed by quantization or by correcting
+    a wrong declared size -- not by waiting longer in a queue."""
+    tail = _SRC.split("needs more than the whole card", 1)[1][:400]
+    assert "quantize" in tail and "used_vram_gb" in tail

tests/test_quantization_map_refines.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""component_quantization refines load_in_4bit; it does not cancel it.
+
+Naming one component in the map used to switch the global flag off for every
+OTHER component, which is the opposite of what setting both plainly means.
+
+Measured 2026-10-29 on LTX-2.5, configured load_in_4bit=True with a map naming
+only transformer and text_encoder: ten of its twelve components loaded at full
+precision, including connectors (11.82 GB) and prompt_enhancer (9.51 GB) --
+21.3 GB of a 25.3 GB card before the quantized transformer arrived. The
+resident attempt then OOMed on exactly those components.
+
+The old candidate list was also written for unet/SD-era pipelines and matched
+only transformer*/unet/text_encoder*, so those two were never candidates for
+the global flag either.
+"""
+
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from codai.models.hf_loading import _is_quantizable_component
+
+
+def test_the_classic_heavy_components_still_qualify():
+    for n in ("transformer", "transformer_2", "unet", "text_encoder"):
+        assert _is_quantizable_component(n), n
+
+
+def test_the_multi_gb_parts_that_were_missed_now_qualify():
+    """connectors and prompt_enhancer outweigh LTX-2.5's quantized
+    transformer between them."""
+    for n in ("connectors", "prompt_enhancer"):
+        assert _is_quantizable_component(n), (
+            f"{n} is still not a candidate for the global quantization flag")
+
+
+def test_small_or_conv_only_components_are_not_swept_in():
+    for n in ("vae", "audio_vae", "scheduler", "tokenizer", "vocoder",
+              "duration_head"):
+        assert not _is_quantizable_component(n), n
+
+
+def test_both_builders_apply_the_global_flag_alongside_the_map():
+    """The pipeline builder and the incremental cache builder must agree, or a
+    cached build quantizes differently from a direct load."""
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    src = open(os.path.join(root, "codai/models/hf_loading.py"),
+               encoding="utf-8").read()
+    assert src.count("if global_4 or global_8:") >= 2, (
+        "one of the two builders still treats the map as replacing the "
+        "global flag")
+    # And neither may fall back to the old narrow list.
+    assert "if _is_heavy(name) and not _bnb_incompatible(name)" not in src
+
+
+def test_an_explicit_map_entry_is_not_overridden():
+    """The map wins for the components it names -- including an explicit
+    'none', which is how an operator opts one component out."""
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    src = open(os.path.join(root, "codai/models/hf_loading.py"),
+               encoding="utf-8").read()
+    assert "if name in quant_mapping or name in comp_q:" in src, (
+        "the global flag can overwrite a component the operator configured")
+    assert "if name in targets or name in comp_q:" in src