give a picture enough tokens to be described

A GGUF vision model answered every image question fluently and wrongly. On
Qwen3.8-27B, with its own matching projector loaded and llama.cpp reporting
"has vision encoder":

  * a 448x448 image added 16 tokens to the prompt (61 -> 77). The same image
    through the HF path added 156.
  * two completely different pictures -- a yellow field with a blue triangle
    and a green square, and three white circles on black -- produced
    byte-identical descriptions at temperature 0, neither resembling either
    image ("the image shows a series of exclamation marks").
  * the same prompt with NO image answered differently, so the pictures were
    reaching the model and only their content was not.

llama.cpp had been printing the reason on every single load:

    load_hparams: projector: qwen3vl_merger
    load_hparams: Qwen-VL models require at minimum 1024 image tokens to
                  function correctly on grounding tasks
    load_hparams: if you encounter problems with accuracy, try adding
                  --image-min-tokens 1024

mtmd_context_params carries image_min_tokens and image_max_tokens, and nothing
set either, so the count came from the image's own size alone: sixteen tokens,
which cannot describe a photograph. The model did the only thing it could and
answered from the prompt, with complete confidence. A 200 carrying a fluent
wrong answer is the worst failure shape there is, because no caller can tell it
from a right one -- and this one survived a full capability sweep, reported as
"answers don't match the picture".

So the floor is set: from the card's image_min_tokens when it says, and
otherwise from the projector's own documented requirement, read from the mmproj
GGUF's clip.projector_type rather than guessed from a filename (an mmproj is
routinely renamed to sit beside the model it serves). Projectors with no
documented floor are left exactly as they were, because forcing an upscale
costs prefill on every request. The figure is printed at load, since it changes
what every image costs.

Upstream builds its context from mtmd_context_params_default() and sets four
fields on the result, with no hook in between and the token bounds not among
them. Rather than copy that method -- which would silently stop tracking any
field upstream later adds -- the handler wraps the default-params factory for
the duration of that one call, so the struct upstream receives already carries
the bounds and everything upstream does to it still happens.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.31"
+__version__ = "0.5.32"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -2985,6 +2985,9 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                 "gpu_split", "tensor_split", "split_strategy", "split_secondary_cap_gb",
                 "turboquant", "engine", "engine_fallback",
                 "quant_backend", "kv_cache_budget_mb", "kv_cache_slots", "mmproj",
+                # How many tokens a picture may become through the mmproj. Left
+                # unset, it is taken from the projector's own requirement.
+                "image_min_tokens", "image_max_tokens",
                 "auto_compact", "auto_compact_pct", "auto_compact_strategy",
                 "auto_compact_model", "auto_compact_target_pct",
                 "auto_compact_head_messages", "auto_compact_head_tokens",

codai/backends/mtmd_vision.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""How many tokens a picture is worth to a GGUF vision model.
+
+A GGUF text model plus an ``mmproj`` projector answers image questions through
+llama.cpp's mtmd handler. The handler decides how many tokens an image becomes
+from the image's own size, and for some projectors that number is far too small
+to carry a picture -- llama.cpp says so itself, on every load:
+
+    load_hparams: projector: qwen3vl_merger
+    load_hparams: Qwen-VL models require at minimum 1024 image tokens to
+                  function correctly on grounding tasks
+    load_hparams: if you encounter problems with accuracy, try adding
+                  --image-min-tokens 1024
+
+Nothing set that floor, and ``mtmd_context_params.image_min_tokens`` stayed at
+its default. What that looked like from outside, measured on Qwen3.8-27B with
+its own matching projector loaded and reporting ``has vision encoder``:
+
+* a 448x448 picture added **16** tokens to the prompt (61 -> 77), where the
+  same image through the HF path added 156;
+* two completely different pictures -- a yellow field with a blue triangle and
+  a green square, and three white circles on black -- produced byte-identical
+  descriptions at temperature 0, neither resembling either image ("a series of
+  exclamation marks");
+* the same prompt with NO image answered differently, so the pictures were
+  reaching the model; only their content was not.
+
+Sixteen tokens cannot describe a photograph, so the model answered from the
+prompt and sounded perfectly confident doing it. That is the worst failure
+shape there is: a 200 with a fluent wrong answer, which no caller can tell from
+a right one.
+
+So the floor is set here: from the card when it says, and otherwise from what
+the projector itself needs.
+"""
+
+from __future__ import annotations
+
+import os
+import threading
+from typing import Optional
+
+#: Projector families that need a minimum, and what llama.cpp says it is.
+#: Keyed by substring of ``clip.projector_type`` as the GGUF records it.
+#: Only families with a documented floor belong here -- a projector that is
+#: happy with what the image gives it must not be forced to upscale, which
+#: costs prefill time for nothing.
+PROJECTOR_MIN_TOKENS = {
+    "qwen3vl": 1024,     # qwen3vl_merger — llama.cpp ggml-org/llama.cpp#16842
+    "qwen2vl": 1024,     # same merger lineage, same grounding requirement
+    "qwen25vl": 1024,
+}
+
+#: Guards the brief window in which the upstream default-params factory is
+#: wrapped (see :class:`ImageTokenFloorHandler`). Loads are serialised in
+#: practice; this makes it true rather than likely.
+_PARAMS_LOCK = threading.Lock()
+
+
+def projector_type(mmproj_path: str) -> str:
+    """``clip.projector_type`` from an mmproj GGUF, lowercased, or "".
+
+    Read from the file rather than guessed from its name: an mmproj is often
+    renamed to sit beside the model it serves, and the name of the base model
+    says nothing about which merger was exported.
+    """
+    try:
+        from gguf import GGUFReader
+        path = os.path.expanduser(str(mmproj_path or ""))
+        if not os.path.isfile(path):
+            return ""
+        reader = GGUFReader(path)
+        field = reader.fields.get("clip.projector_type")
+        if field is None:
+            return ""
+        val = field.parts[field.data[0]] if field.data else None
+        if val is None:
+            return ""
+        try:
+            return bytes(val).decode("utf-8", "replace").strip().lower()
+        except (TypeError, ValueError):
+            return str(val).strip().lower()
+    except Exception:
+        # No gguf reader, an unreadable file, or a metadata layout this does
+        # not know: fall back to no floor, which is how it behaved before.
+        return ""
+
+
+def min_image_tokens(mmproj_path: str, configured=None) -> tuple:
+    """``(min_tokens, why)`` for this projector.
+
+    A card's own ``image_min_tokens`` always wins, including an explicit 0 for
+    an operator who wants the old behaviour back.
+    """
+    if configured is not None:
+        try:
+            want = int(configured)
+        except (TypeError, ValueError):
+            want = 0
+        return max(0, want), "set on the model's card"
+    proj = projector_type(mmproj_path)
+    if not proj:
+        return 0, ""
+    key = proj.replace("-", "").replace("_", "").replace(".", "")
+    for name, floor in PROJECTOR_MIN_TOKENS.items():
+        if name in key:
+            return floor, (f"{proj} needs at least {floor} image tokens to "
+                           f"describe a picture")
+    return 0, ""
+
+
+def handler(mmproj_path: str, *, use_gpu: bool = True, verbose: bool = False,
+            image_min_tokens: int = 0, image_max_tokens: int = 0):
+    """An mtmd chat handler that honours an image-token floor.
+
+    Returns the plain upstream handler when no floor applies, so the ordinary
+    case runs upstream code with nothing of ours in the way.
+    """
+    from llama_cpp.llama_chat_format import MTMDChatHandler
+    if not (image_min_tokens or image_max_tokens):
+        return MTMDChatHandler(clip_model_path=mmproj_path, verbose=verbose,
+                               use_gpu=use_gpu)
+    return ImageTokenFloorHandler(
+        clip_model_path=mmproj_path, verbose=verbose, use_gpu=use_gpu,
+        image_min_tokens=image_min_tokens, image_max_tokens=image_max_tokens)
+
+
+class _LazyBase:
+    """Deferred base so importing this module never imports llama_cpp."""
+
+
+def _make_handler_class():
+    from llama_cpp.llama_chat_format import MTMDChatHandler
+
+    class ImageTokenFloorHandler(MTMDChatHandler):
+        """:class:`MTMDChatHandler` with ``image_min/max_tokens`` applied.
+
+        Upstream builds its context from
+        ``mtmd_cpp.mtmd_context_params_default()`` and sets four fields on the
+        result; the token bounds are not among them and there is no hook in
+        between. Rather than copy that method -- which would silently stop
+        tracking any field upstream adds -- this wraps the default-params
+        factory for the duration of the one call, so the struct upstream
+        receives already carries the bounds and everything else upstream does
+        to it still happens.
+        """
+
+        def __init__(self, *args, image_min_tokens: int = 0,
+                     image_max_tokens: int = 0, **kwargs):
+            self._image_min_tokens = max(0, int(image_min_tokens or 0))
+            self._image_max_tokens = max(0, int(image_max_tokens or 0))
+            super().__init__(*args, **kwargs)
+
+        def _init_mtmd_context(self, llama_model):
+            if getattr(self, "mtmd_ctx", None) is not None:
+                return
+            mtmd = self._mtmd_cpp
+            original = mtmd.mtmd_context_params_default
+
+            def _with_bounds():
+                params = original()
+                if self._image_min_tokens:
+                    params.image_min_tokens = self._image_min_tokens
+                if self._image_max_tokens:
+                    params.image_max_tokens = self._image_max_tokens
+                return params
+
+            with _PARAMS_LOCK:
+                mtmd.mtmd_context_params_default = _with_bounds
+                try:
+                    return super()._init_mtmd_context(llama_model)
+                finally:
+                    mtmd.mtmd_context_params_default = original
+
+    return ImageTokenFloorHandler
+
+
+def __getattr__(name):
+    # ImageTokenFloorHandler is built on first use so that `import
+    # mtmd_vision` costs nothing on a server with no vision model.
+    if name == "ImageTokenFloorHandler":
+        cls = _make_handler_class()
+        globals()[name] = cls
+        return cls
+    raise AttributeError(name)

codai/backends/vulkan.py

@@ -1884,14 +1884,32 @@ class VulkanBackend(ModelBackend):
                     _mmproj_path = _cand
             if os.path.isfile(_mmproj_path):
                 try:
-                    from llama_cpp.llama_chat_format import MTMDChatHandler
-                    llama_kwargs['chat_handler'] = MTMDChatHandler(
-                        clip_model_path=_mmproj_path,
+                    from codai.backends import mtmd_vision
+                    # How many tokens a picture is allowed to become. Left
+                    # unset, mtmd takes it from the image's size alone, and for
+                    # a Qwen-VL merger that is nowhere near enough: a 448x448
+                    # image became 16 tokens, two different pictures produced
+                    # identical descriptions, and the model answered fluently
+                    # from the prompt. llama.cpp prints the requirement on every
+                    # load ("requires at minimum 1024 image tokens"); this reads
+                    # it from the projector instead of hoping someone saw it.
+                    _min_tok, _why = mtmd_vision.min_image_tokens(
+                        _mmproj_path, kwargs.get('image_min_tokens',
+                                                 _raw_cfg.get('image_min_tokens')))
+                    _max_tok = kwargs.get('image_max_tokens',
+                                          _raw_cfg.get('image_max_tokens')) or 0
+                    llama_kwargs['chat_handler'] = mtmd_vision.handler(
+                        _mmproj_path,
                         verbose=False,
                         use_gpu=(self.n_gpu_layers != 0),
+                        image_min_tokens=_min_tok,
+                        image_max_tokens=int(_max_tok or 0),
                     )
                     self.supports_vision = True
                     print(f"  mmproj       : {os.path.basename(_mmproj_path)} (vision enabled)")
+                    if _min_tok:
+                        print(f"  image tokens : at least {_min_tok} per image"
+                              + (f" — {_why}" if _why else ""))
                 except Exception as _e:
                     print(f"  mmproj       : failed to load projector ({_e}); continuing text-only")
             else:

tests/test_gguf_vision_image_tokens.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""How many tokens a picture is worth to a GGUF vision model.
+
+The failure these exist for, measured on Qwen3.8-27B with its own matching
+projector loaded and llama.cpp reporting ``has vision encoder``:
+
+* a 448x448 image added 16 tokens to the prompt (61 -> 77); the same image on
+  the HF path added 156;
+* two completely different pictures produced byte-identical descriptions at
+  temperature 0, neither resembling either image;
+* the same prompt with no image answered differently, so the pictures arrived
+  and only their content did not.
+
+llama.cpp had printed the reason on every load -- ``qwen3vl_merger`` needs at
+least 1024 image tokens -- and ``mtmd_context_params.image_min_tokens`` was
+never set. A 200 with a fluent wrong answer is the worst outcome available, so
+these tests are about the floor being applied, and about not applying it where
+it is not needed.
+"""
+
+import io
+import os
+
+import pytest
+
+from codai.backends import mtmd_vision as M
+
+_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _src(rel):
+    return io.open(os.path.join(_ROOT, rel), encoding="utf-8").read()
+
+
+# ---------------------------------------------------------------------------
+# the floor itself
+# ---------------------------------------------------------------------------
+
+@pytest.mark.parametrize("declared", [
+    "qwen3vl_merger", "QWEN3VL_MERGER", "qwen3vl-merger", "qwen2vl_merger",
+])
+def test_a_qwen_vl_merger_gets_the_floor_llama_cpp_asks_for(monkeypatch, declared):
+    monkeypatch.setattr(M, "projector_type", lambda p: declared.lower())
+    tokens, why = M.min_image_tokens("/any/mmproj.gguf")
+    assert tokens == 1024
+    assert "image tokens" in why
+
+
+def test_a_projector_with_no_documented_floor_is_left_alone(monkeypatch):
+    """Forcing an upscale costs prefill time on every request — a projector
+    happy with what the image gives it must not pay for this."""
+    monkeypatch.setattr(M, "projector_type", lambda p: "mlp")
+    assert M.min_image_tokens("/any/mmproj.gguf") == (0, "")
+
+
+def test_an_unreadable_projector_behaves_exactly_as_before(monkeypatch):
+    monkeypatch.setattr(M, "projector_type", lambda p: "")
+    assert M.min_image_tokens("/any/mmproj.gguf") == (0, "")
+
+
+def test_the_card_always_wins(monkeypatch):
+    monkeypatch.setattr(M, "projector_type", lambda p: "qwen3vl_merger")
+    tokens, why = M.min_image_tokens("/any/mmproj.gguf", 2048)
+    assert tokens == 2048
+    assert "card" in why
+
+
+def test_an_explicit_zero_on_the_card_restores_the_old_behaviour(monkeypatch):
+    """An operator who would rather have the previous behaviour keeps it, and
+    0 must not be read as "unset" — which is the usual way a deliberate zero
+    gets ignored."""
+    monkeypatch.setattr(M, "projector_type", lambda p: "qwen3vl_merger")
+    assert M.min_image_tokens("/any/mmproj.gguf", 0)[0] == 0
+
+
+def test_nonsense_on_the_card_does_not_raise(monkeypatch):
+    monkeypatch.setattr(M, "projector_type", lambda p: "qwen3vl_merger")
+    assert M.min_image_tokens("/any/mmproj.gguf", "lots")[0] == 0
+    assert M.min_image_tokens("/any/mmproj.gguf", -5)[0] == 0
+
+
+def test_the_projector_type_is_read_from_the_file_not_the_name():
+    """An mmproj is routinely renamed to sit beside the model it serves, and
+    the base model's name says nothing about which merger was exported."""
+    assert M.projector_type("/does/not/exist.gguf") == ""
+    src = _src("codai/backends/mtmd_vision.py")
+    assert "clip.projector_type" in src
+    assert "GGUFReader" in src
+
+
+# ---------------------------------------------------------------------------
+# the handler
+# ---------------------------------------------------------------------------
+
+def test_no_floor_means_the_plain_upstream_handler(monkeypatch):
+    """Nothing of ours in the way for the ordinary case."""
+    built = {}
+
+    class _Fake:
+        def __init__(self, **kw):
+            built.update(kw)
+
+    import sys
+    import types
+    mod = types.ModuleType("llama_cpp.llama_chat_format")
+    mod.MTMDChatHandler = _Fake
+    pkg = types.ModuleType("llama_cpp")
+    pkg.llama_chat_format = mod
+    monkeypatch.setitem(sys.modules, "llama_cpp", pkg)
+    monkeypatch.setitem(sys.modules, "llama_cpp.llama_chat_format", mod)
+    out = M.handler("/m.gguf", use_gpu=True, image_min_tokens=0)
+    assert isinstance(out, _Fake)
+    assert built == {"clip_model_path": "/m.gguf", "verbose": False,
+                     "use_gpu": True}
+
+
+def test_the_bounds_reach_the_params_upstream_builds(monkeypatch):
+    """Upstream sets four fields on mtmd_context_params_default() and offers no
+    hook in between; the token bounds are not among them. Wrapping the factory
+    for the duration of that one call means the struct upstream receives
+    already carries them AND everything upstream does to it still happens —
+    copying the method instead would stop tracking any field upstream adds."""
+    import sys
+    import types
+
+    class _Params:
+        def __init__(self):
+            self.use_gpu = None
+            self.image_min_tokens = 0
+            self.image_max_tokens = 0
+
+    seen = {}
+
+    class _Mtmd:
+        @staticmethod
+        def mtmd_context_params_default():
+            return _Params()
+
+    class _Upstream:
+        def __init__(self, **kw):
+            self.mtmd_ctx = None
+            self._mtmd_cpp = _Mtmd()
+
+        def _init_mtmd_context(self, llama_model):
+            # Stand-in for upstream: build params, set its own field, record.
+            params = self._mtmd_cpp.mtmd_context_params_default()
+            params.use_gpu = True
+            seen["params"] = params
+
+    mod = types.ModuleType("llama_cpp.llama_chat_format")
+    mod.MTMDChatHandler = _Upstream
+    pkg = types.ModuleType("llama_cpp")
+    pkg.llama_chat_format = mod
+    monkeypatch.setitem(sys.modules, "llama_cpp", pkg)
+    monkeypatch.setitem(sys.modules, "llama_cpp.llama_chat_format", mod)
+
+    cls = M._make_handler_class()
+    h = cls(clip_model_path="/m.gguf", image_min_tokens=1024,
+            image_max_tokens=4096)
+    h._init_mtmd_context(object())
+    assert seen["params"].image_min_tokens == 1024
+    assert seen["params"].image_max_tokens == 4096
+    # Upstream's own field survived our wrapper.
+    assert seen["params"].use_gpu is True
+    # And the factory is restored, so nothing else sees our values.
+    assert h._mtmd_cpp.mtmd_context_params_default().image_min_tokens == 0
+
+
+# ---------------------------------------------------------------------------
+# wiring
+# ---------------------------------------------------------------------------
+
+def test_the_backend_sets_the_floor_when_it_loads_a_projector():
+    src = _src("codai/backends/vulkan.py")
+    head = src.index("_mmproj = kwargs.get('mmproj'")
+    body = src[head:head + 2500]
+    assert "mtmd_vision.min_image_tokens(" in body
+    assert "mtmd_vision.handler(" in body
+    # The old unconditional construction is gone.
+    assert "MTMDChatHandler(\n" not in body
+
+
+def test_the_floor_is_announced_in_the_log():
+    """It changes how much prefill every image costs; an operator reading the
+    load log should see it without having to know this file exists."""
+    src = _src("codai/backends/vulkan.py")
+    assert "image tokens : at least" in src
+
+
+def test_the_field_is_savable_from_the_admin_page():
+    assert '"image_min_tokens", "image_max_tokens",' in _src("codai/admin/routes.py")