load the vision tower a checkpoint asks for, and refuse the pictures we cannot read

A model listed as able to read images has to read them, or say it cannot.

`Qwen3.5-9B` was listed with `image_to_text` and answered every photograph
with "a blank white image with no visible content" — nine real pictures of
one video, indexed as blank. Two causes, one symptom:

  AutoModelForCausalLM        -> Qwen3_5ForCausalLM            (text only)
  AutoModelForImageTextToText -> Qwen3_5ForConditionalGeneration

The engine asked for the first, so the tower was never built, and with no
projector the chat path flattened the picture to the literal text
"[image_url content]". The model was answering the prompt it was given.

Now the class comes from the checkpoint's own config, every load path in the
CUDA backend goes through that resolver, a vision model also loads the
processor that makes a picture promptable, and an image request takes its own
generation path (no KV prefix reuse — image tokens come from the pixels).
Where a model genuinely cannot see, the request is a 400 rather than a
confident wrong answer, and a GGUF with no mmproj stops advertising the
capability it cannot serve.

Two things found while measuring the image path on a live fleet:

* every image load printed "Proactive eviction skipped: name 'request' is not
  defined". `_load_diffusers_pipeline` has no `request`; the NameError was
  swallowed as a skip, so the minimum-eviction block never ran and each load
  fell through to the blunt eviction that unloads models which would have
  fitted alongside.
* `Z-Image-Turbo`, distilled for 8 steps at CFG 1, was served at 30 steps and
  CFG 7.5 because its card carried no `defaults` — four times the work for an
  over-guided picture, ~72 s a still. A card that says nothing now falls back
  to what the model's own name says before the generic floor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.16"
+__version__ = "0.5.17"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/images.py

@@ -647,6 +647,12 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
 
     _mc = model_config or {}
 
+    # The id the CALLER asked for, kept before `model_name` is rewritten to the
+    # checkpoint path below. Sizing needs it: two cards can share one
+    # checkpoint with different quantization and offload, and the one that was
+    # asked for is the one whose footprint applies.
+    _requested_id = model_name
+
     # A multi-config sibling is registered under its alias, so `model_name` may
     # BE that alias rather than something from_pretrained can open. The config
     # carries the checkpoint it is a config of; prefer it when present. For a
@@ -845,9 +851,15 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
                 # requested_id: same-path siblings (one checkpoint, two cards
                 # with different settings) must be sized by the card that was
                 # ASKED for, not by whichever one registered the path.
+                # `request` does not exist in this function. Reading it raised
+                # NameError on EVERY image load, and the except below swallowed
+                # it as "Proactive eviction skipped: name 'request' is not
+                # defined" -- so the whole minimum-eviction block never ran and
+                # every image load fell through to the blunt eviction further
+                # down, which unloads a model that would have fitted alongside.
                 _need_gb = _mmm._get_model_used_vram_gb(
                     _key or model_name, model_name,
-                    requested_id=getattr(request, 'model', None))
+                    requested_id=_requested_id)
                 if _need_gb > 0:
                     if _free_gb < _need_gb:
                         print(f"Image model needs {_need_gb:.1f} GB, {_free_gb:.1f} GB free "
@@ -1558,13 +1570,36 @@ async def _generate_with_diffusers(pipeline, request, global_args, http_request=
     _d_cfg = _dflt.get("guidance_scale")
     if _d_cfg is None:
         _d_cfg = _dflt.get("cfg_scale")
+    # A distilled checkpoint whose card says nothing. 30 steps at CFG 7.5 is
+    # the wrong answer for a model distilled for 8 at CFG 1 -- four times the
+    # work AND an over-guided, burnt-looking picture. Below the card's own
+    # defaults and below a fused distill LoRA, because both know more; above
+    # the generic fallback, because the model's own name knows more than that.
+    _distilled = None
+    if not _d_steps and _accel_steps is None:
+        try:
+            from codai.models.acceleration import distilled_defaults_for
+            _distilled = distilled_defaults_for(
+                str((model_config or {}).get('path') or '')
+                or str(getattr(request, 'model', '') or ''))
+        except Exception:
+            _distilled = None
+        if _distilled and not (request.steps and request.guidance_scale):
+            print(f"  '{_distilled['token']}' in the model id: this checkpoint is "
+                  f"distilled, so {_distilled['steps']} steps at CFG "
+                  f"{_distilled['guidance_scale']} (its card carries no defaults; "
+                  f"the generic fallback would be 30 at CFG 7.5)")
+
     num_steps = request.steps if request.steps else (
         _accel_steps if _accel_steps else
-        (int(_d_steps) if _d_steps else (30 if quality == "standard" else 50)))
+        (int(_d_steps) if _d_steps else
+         (int(_distilled["steps"]) if _distilled else
+          (30 if quality == "standard" else 50))))
     cfg_scale = request.guidance_scale if request.guidance_scale else (
         _accel_cfg if _accel_cfg is not None else
         (float(_d_cfg) if _d_cfg is not None else
-         (getattr(global_args, 'image_cfg_scale', 7.5) if quality == "standard" else 9.0))
+         (float(_distilled["guidance_scale"]) if _distilled else
+          (getattr(global_args, 'image_cfg_scale', 7.5) if quality == "standard" else 9.0)))
     )
     # The sampler: what the caller asked for, else what the card recommends.
     # A bad name from the CALLER is a 400 -- they can fix it -- while a bad one

codai/api/text.py

@@ -1620,6 +1620,31 @@ async def chat_completions(request: ChatCompletionRequest, http_request: Request
     # flattened to a text placeholder, so the multimodal handler can see it.
     _vision_ok = bool(getattr(getattr(current_manager, 'backend', None), 'supports_vision', False))
 
+    # A picture sent to a model that cannot see it must not be answered.
+    #
+    # Measured 2026-10-29: Qwen3.5-9B is listed with `image_to_text`, was
+    # loaded without its vision tower, and answered every photo with "a blank
+    # white image with no visible content" -- because the image had been
+    # replaced, below, by the literal text "[image_url content]". The model was
+    # answering the prompt it got. A caller cannot tell that answer from a real
+    # one, so it indexed nine photographs as blank. Refusing is the only honest
+    # outcome: a client can fall back to a model that can see, and cannot fall
+    # back from a confident wrong answer.
+    if not _vision_ok:
+        for _m in (request.messages or []):
+            _c = getattr(_m, 'content', None)
+            if not isinstance(_c, list):
+                continue
+            if any(isinstance(_it, dict) and _it.get('type') in ('image_url', 'input_image')
+                   for _it in _c):
+                _release_instance()
+                raise HTTPException(status_code=400, detail=(
+                    f"Model '{requested_model}' cannot read images: it is served "
+                    f"without a vision tower (an HF checkpoint with no "
+                    f"vision_config or processor, or a GGUF with no mmproj). "
+                    f"Send the image to a model whose capabilities list "
+                    f"image_to_text, or configure an mmproj for this one."))
+
     # Inject system prompt if --system-prompt flag was provided
     messages = request.messages
     global_system_prompt = get_global_system_prompt()

codai/backends/cuda.py

@@ -175,6 +175,13 @@ class NvidiaBackend(ModelBackend):
         self.tokenizer = None
         self.model_name = None
         self.device = None
+        # A vision-language checkpoint also loads its processor, and only then
+        # is it able to answer about a picture. Both are set by
+        # _init_vision() after the weights are in; `supports_vision` is what
+        # the chat path reads to decide whether an image may be passed through
+        # instead of flattened to a text placeholder.
+        self.processor = None
+        self.supports_vision = False
         self.use_flash_attn = False
         self.flash_attn_available = False
         # Multi-slot KV prefix cache. Each slot keeps a pristine prefill cache for
@@ -583,14 +590,73 @@ class NvidiaBackend(ModelBackend):
         
         return max_memory
     
+    def _auto_model_class(self, model_name: str):
+        """The transformers auto-class this checkpoint should be loaded with.
+
+        A vision-language checkpoint loaded with ``AutoModelForCausalLM`` gets
+        its text decoder and nothing else -- no vision tower, no projector --
+        and then answers every picture as though it were blank, because the
+        chat path has nowhere to put an image and sends a text placeholder
+        instead. Asking the config which class fits is what makes a model
+        advertised as ``image_to_text`` actually able to see.
+
+        Falls back to the text class for anything it cannot determine, so a
+        checkpoint that is not a vision model loads exactly as before.
+        """
+        try:
+            from codai.backends import hf_vision
+            cls, is_vision = hf_vision.auto_model_class(model_name)
+            if is_vision:
+                print(f"  Vision checkpoint: loading with {cls.__name__} "
+                      f"(text-only classes drop the vision tower)")
+            return cls
+        except Exception as exc:
+            print(f"  Warning: could not resolve a model class for {model_name} "
+                  f"({exc}); using AutoModelForCausalLM")
+            from transformers import AutoModelForCausalLM
+            return AutoModelForCausalLM
+
+    def _init_vision(self, model_name: str) -> None:
+        """Load the processor for a vision model, after the weights are in.
+
+        ``supports_vision`` is only set when BOTH halves are present: a model
+        whose class holds a vision tower and a processor that can preprocess a
+        picture for it. Either one missing means the model cannot answer about
+        an image, and the chat path must say so rather than answer blind.
+        """
+        self.processor = None
+        self.supports_vision = False
+        try:
+            from codai.backends import hf_vision
+            if not hf_vision.is_vision_checkpoint(model_name):
+                return
+            # The loaded class decides, not the config: a vision config that
+            # fell back to the text class (unknown family, failed load) is a
+            # text model at runtime whatever the card says.
+            if "ForConditionalGeneration" not in type(self.model).__name__ and \
+                    not hasattr(self.model, "get_image_features"):
+                print(f"  Vision: {model_name} was loaded as "
+                      f"{type(self.model).__name__}, which has no vision tower "
+                      f"— images will be refused")
+                return
+            proc = hf_vision.load_processor(model_name)
+            if proc is None:
+                return
+            self.processor = proc
+            self.supports_vision = True
+            print(f"  Vision: processor loaded — this model can read images")
+        except Exception as exc:
+            print(f"  Warning: vision init failed for {model_name} ({exc}); "
+                  f"images will be refused")
+
     def _try_load_model(self, model_name: str, load_kwargs: dict, device: str):
         """Try to load model with given settings."""
         import torch
-        from transformers import AutoModelForCausalLM
+        _auto = self._auto_model_class(model_name)
         
         try:
             load_kwargs = self._strip_invalid_native_quant_config(model_name, load_kwargs)
-            model = AutoModelForCausalLM.from_pretrained(model_name, **load_kwargs)
+            model = _auto.from_pretrained(model_name, **load_kwargs)
             if device == "cpu" and load_kwargs.get('device_map') is None:
                 model = model.to(device)
             return model
@@ -609,7 +675,7 @@ class NvidiaBackend(ModelBackend):
                     retry_kwargs.pop('load_in_4bit', None)
                     retry_kwargs.pop('load_in_8bit', None)
                     try:
-                        model = AutoModelForCausalLM.from_pretrained(model_name, **retry_kwargs)
+                        model = _auto.from_pretrained(model_name, **retry_kwargs)
                         if device == "cpu" and retry_kwargs.get('device_map') is None:
                             model = model.to(device)
                         print("Model loaded successfully without quantization")
@@ -873,7 +939,7 @@ class NvidiaBackend(ModelBackend):
     def load_model(self, model_name: str, **kwargs) -> None:
         """Load the model using HuggingFace Transformers with automatic OOM handling."""
         import torch
-        from transformers import AutoModelForCausalLM, AutoTokenizer
+        from transformers import AutoTokenizer
 
         # Re-evaluate KV-prefix support for the model about to be loaded.
         self._kv_prefix_ok = None
@@ -1079,10 +1145,12 @@ class NvidiaBackend(ModelBackend):
             
             try:
                 load_kwargs = self._strip_invalid_native_quant_config(model_name, load_kwargs)
-                model = AutoModelForCausalLM.from_pretrained(model_name, **load_kwargs)
+                model = self._auto_model_class(model_name).from_pretrained(
+                    model_name, **load_kwargs)
                 self.model = model
                 self.model.eval()
                 self.model_name = model_name
+                self._init_vision(model_name)
                 
                 print(f"\n--no-ram: Model loaded successfully on {cuda_device}")
                 print(f"Model device: {next(self.model.parameters()).device}")
@@ -1138,7 +1206,8 @@ class NvidiaBackend(ModelBackend):
             
             try:
                 load_kwargs = self._strip_invalid_native_quant_config(model_name, load_kwargs)
-                model = AutoModelForCausalLM.from_pretrained(model_name, **load_kwargs)
+                model = self._auto_model_class(model_name).from_pretrained(
+                    model_name, **load_kwargs)
             except Exception as e:
                 raise RuntimeError(
                     f"--offload-strategy none: Failed to load model entirely on GPU ({cuda_device}). "
@@ -1375,6 +1444,7 @@ class NvidiaBackend(ModelBackend):
         _apply_text_loras(self.model, kwargs.get('model_config') or kwargs)
         self.model.eval()
         self.model_name = model_name
+        self._init_vision(model_name)
 
         print(f"\nModel loaded successfully")
         print(f"Model device: {next(self.model.parameters()).device}")
@@ -2366,6 +2436,123 @@ class NvidiaBackend(ModelBackend):
                 pass
         return self._format_messages_to_str(messages)
 
+    def _vision_request(self, messages) -> bool:
+        """Whether this request needs the vision path.
+
+        Both halves have to be true: the request carries a picture, and this
+        model loaded a tower and a processor for one. A picture sent to a model
+        without them is refused by the API layer, not answered blind.
+        """
+        if not (self.supports_vision and self.processor is not None):
+            return False
+        try:
+            from codai.backends import hf_vision
+            return hf_vision.has_images(messages)
+        except Exception:
+            return False
+
+    def _vision_inputs(self, messages):
+        """Processor inputs for a chat with pictures, on the model's device."""
+        from codai.backends import hf_vision
+        inputs, images = hf_vision.build_inputs(self.processor, messages)
+        inputs = inputs.to(self.model.device)
+        return inputs, images
+
+    def _vision_gen_kwargs(self, max_tokens, temperature, top_p):
+        temperature, top_p, do_sample = self._validate_params(temperature, top_p)
+        tok = getattr(self.processor, "tokenizer", None) or self.tokenizer
+        gen_kwargs = dict(
+            max_new_tokens=max_tokens,
+            temperature=temperature if do_sample else None,
+            top_p=top_p if do_sample else None,
+            do_sample=do_sample,
+            pad_token_id=getattr(tok, "pad_token_id", None),
+            eos_token_id=self._eos_token_ids(),
+            use_cache=True,
+        )
+        _therm = _make_thermal_criteria()
+        if _therm is not None:
+            from transformers import StoppingCriteriaList
+            gen_kwargs["stopping_criteria"] = StoppingCriteriaList([_therm])
+        return gen_kwargs
+
+    def _generate_vision_chat(self, messages, max_tokens, temperature, top_p) -> str:
+        """Answer a chat that contains pictures.
+
+        Deliberately separate from the text path: the KV prefix cache keys on a
+        token prefix, and image tokens are expanded by the processor from the
+        pictures themselves, so a cached prefix from another request would be
+        reused against the wrong pixels. An image request is also rare enough
+        that the cache buys nothing.
+        """
+        import torch
+        inputs, images = self._vision_inputs(messages)
+        prompt_len = int(inputs["input_ids"].shape[1])
+        gen_kwargs = self._vision_gen_kwargs(max_tokens, temperature, top_p)
+        with self._gen_lock, torch.no_grad():
+            outputs = self.model.generate(**inputs, **gen_kwargs)
+        new_tokens = outputs[0][prompt_len:]
+        tok = getattr(self.processor, "tokenizer", None) or self.tokenizer
+        text = tok.decode(new_tokens, skip_special_tokens=True)
+        self._last_usage = {
+            'prompt_tokens': prompt_len,
+            'completion_tokens': int(new_tokens.shape[0]),
+            'cached_tokens': 0,
+            'images': len(images),
+        }
+        return text
+
+    async def _generate_vision_chat_stream(self, messages, max_tokens,
+                                           temperature, top_p):
+        """The streaming form of :meth:`_generate_vision_chat`."""
+        import asyncio
+        import torch
+        from transformers import TextIteratorStreamer
+        from threading import Thread
+
+        inputs, images = self._vision_inputs(messages)
+        prompt_len = int(inputs["input_ids"].shape[1])
+        tok = getattr(self.processor, "tokenizer", None) or self.tokenizer
+        streamer = TextIteratorStreamer(tok, skip_prompt=True, skip_special_tokens=True)
+        gen_kwargs = self._vision_gen_kwargs(max_tokens, temperature, top_p)
+        gen_kwargs["streamer"] = streamer
+        err = [None]
+
+        def _run():
+            try:
+                with self._gen_lock, torch.no_grad():
+                    self.model.generate(**inputs, **gen_kwargs)
+            except Exception as exc:
+                err[0] = str(exc)
+                print(f"Error during vision stream generation: {exc}")
+            finally:
+                # generate() only ends the streamer when it finished; ending it
+                # here too means a failure cannot deadlock the consumer.
+                try:
+                    streamer.end()
+                except Exception:
+                    pass
+
+        Thread(target=_run, daemon=True).start()
+        loop = asyncio.get_event_loop()
+        _SENT = object()
+        produced = 0
+        while True:
+            chunk = await loop.run_in_executor(
+                None, lambda: next(iter(streamer), _SENT))
+            if chunk is _SENT:
+                break
+            produced += 1
+            yield chunk
+        if err[0] and produced == 0:
+            raise GenerationFailed(f"vision generation failed: {err[0]}")
+        self._last_usage = {
+            'prompt_tokens': prompt_len,
+            'completion_tokens': produced,
+            'cached_tokens': 0,
+            'images': len(images),
+        }
+
     def generate_chat(self, messages, max_tokens=None, temperature=0.7,
                       top_p=1.0, stop=None, tools=None, response_format=None,
                       enable_thinking=False) -> str:
@@ -2380,6 +2567,9 @@ class NvidiaBackend(ModelBackend):
         if max_tokens is None:
             max_tokens = 512
 
+        if self._vision_request(messages):
+            return self._generate_vision_chat(messages, max_tokens, temperature, top_p)
+
         full_prompt = self._build_chat_prompt(messages, enable_thinking=enable_thinking,
                                               add_generation_prompt=True, tools=tools)
         total_input_ids = self.tokenizer(full_prompt, return_tensors="pt")['input_ids']
@@ -2518,6 +2708,12 @@ class NvidiaBackend(ModelBackend):
         if max_tokens is None:
             max_tokens = 512
 
+        if self._vision_request(messages):
+            async for _chunk in self._generate_vision_chat_stream(
+                    messages, max_tokens, temperature, top_p):
+                yield _chunk
+            return
+
         full_prompt = self._build_chat_prompt(messages, enable_thinking=enable_thinking,
                                               add_generation_prompt=True, tools=tools)
         total_input_ids = self.tokenizer(full_prompt, return_tensors="pt")['input_ids']
@@ -2825,6 +3021,10 @@ class NvidiaBackend(ModelBackend):
         if self.tokenizer is not None:
             del self.tokenizer
             self.tokenizer = None
+        if self.processor is not None:
+            del self.processor
+        self.processor = None
+        self.supports_vision = False
         # Force Python GC before emptying the CUDA allocator pool so that all
         # Python-held tensor references (closures, local vars, etc.) are dropped.
         for _ in range(3):

codai/backends/hf_vision.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""Vision for a transformers-served chat model.
+
+A vision-language checkpoint holds two things: a text decoder and a vision
+tower that turns pixels into the embeddings the decoder reads. ``AutoModelFor-
+CausalLM`` loads only the first of them. Measured on transformers 5.12 with
+``Qwen/Qwen3.5-9B``, whose config carries a ``vision_config`` and an
+``image_token_id``::
+
+    AutoModelForCausalLM      -> Qwen3_5ForCausalLM            (text only)
+    AutoModelForImageTextToText -> Qwen3_5ForConditionalGeneration
+
+The engine asked for the first class, so the tower was never instantiated and
+the weights for it were never read. The model then answered every picture with
+a description of nothing -- "a blank white image with no visible content" --
+because the request had reached it as the literal text ``[image_url content]``:
+with no projector loaded the chat path has nowhere to put an image, so it
+flattens multipart content to a placeholder. The model was not wrong; it was
+answering the prompt it was given.
+
+Two halves, both here:
+
+* :func:`auto_model_class` picks the class from the checkpoint's own config, so
+  a vision checkpoint loads its tower. It is used by every load path in the
+  CUDA backend, which is why it must never raise -- an unreadable config falls
+  back to the text class and the model keeps working as it did before.
+* :func:`build_inputs` renders a chat with its images through the model's
+  ``AutoProcessor``, which is the only component that knows how many image
+  placeholder tokens this family expands one picture into.
+
+What a caller sends is normalised by :func:`codai.api.text._normalize_vision_content`
+before it gets here, so images arrive as ``{"type": "image_url", "image_url":
+{"url": ...}}`` with either an http(s) link or a ``data:`` URI.
+"""
+
+import base64
+import binascii
+import io
+import os
+import re
+from typing import Any, Dict, List, Optional, Tuple
+
+#: How long to wait for an image the caller referenced by URL. A chat request
+#: is already waiting on this, so it is short on purpose.
+_FETCH_TIMEOUT = 20
+
+_DATA_URI = re.compile(r"^data:(?P<mime>[^;,]*)(?P<b64>;base64)?,(?P<data>.*)$",
+                       re.IGNORECASE | re.DOTALL)
+
+
+def _config_for(model_name: str):
+    """The checkpoint's config, or None when it cannot be read.
+
+    ``trust_remote_code`` matches what the loaders use: a config that needs its
+    own code to be read at all would otherwise report "no vision" and the model
+    would load without its tower again.
+    """
+    try:
+        from transformers import AutoConfig
+        return AutoConfig.from_pretrained(model_name, trust_remote_code=True)
+    except Exception:
+        return None
+
+
+def is_vision_checkpoint(model_name: str, config=None) -> bool:
+    """Whether this checkpoint carries a vision tower.
+
+    Decided from the config, not from the model id: ``Qwen3.5-9B`` has no
+    "vl"/"vision" in its name and is a vision model, while plenty of models
+    with "vision" in the name are not image-text-to-text at all. The two
+    signals are a nested ``vision_config`` and an ``image_token_id`` -- a
+    decoder that has somewhere to put an image.
+    """
+    cfg = config if config is not None else _config_for(model_name)
+    if cfg is None:
+        return False
+    if getattr(cfg, "vision_config", None) is None and \
+            getattr(cfg, "image_token_id", None) is None:
+        return False
+    # The config says there is a tower; the installed transformers must also
+    # have a class that loads it. Without this check a brand-new family would
+    # pick the vision class and fail the load outright, which is worse than
+    # serving it as text.
+    try:
+        from transformers import AutoModelForImageTextToText
+        return AutoModelForImageTextToText._model_mapping.get(type(cfg)) is not None
+    except Exception:
+        return False
+
+
+def auto_model_class(model_name: str):
+    """``(class, is_vision)`` for loading this checkpoint.
+
+    Never raises. The text class is the fallback for every uncertainty, so a
+    checkpoint this function cannot make sense of loads exactly as it did
+    before vision existed here.
+    """
+    from transformers import AutoModelForCausalLM
+    try:
+        if is_vision_checkpoint(model_name):
+            from transformers import AutoModelForImageTextToText
+            return AutoModelForImageTextToText, True
+    except Exception:
+        pass
+    return AutoModelForCausalLM, False
+
+
+def load_processor(model_name: str):
+    """The checkpoint's ``AutoProcessor``, or None.
+
+    The processor -- not the tokenizer -- owns the image preprocessing and the
+    placeholder expansion, so without it a vision model cannot be prompted with
+    a picture even though its tower is loaded. A model whose processor cannot be
+    built is therefore not vision-capable, and the backend says so rather than
+    answering blind.
+    """
+    try:
+        from transformers import AutoProcessor
+        proc = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
+    except Exception as exc:
+        print(f"[vision] no AutoProcessor for {model_name}: {exc}")
+        return None
+    # A tokenizer-only "processor" cannot take images. transformers happily
+    # returns one for some text checkpoints, and it would pass every duck-type
+    # check until the first picture arrived.
+    if not hasattr(proc, "image_processor") and not hasattr(proc, "feature_extractor"):
+        print(f"[vision] {model_name}: AutoProcessor has no image processor")
+        return None
+    return proc
+
+
+def _decode_data_uri(url: str):
+    m = _DATA_URI.match(url)
+    if not m:
+        return None
+    data = m.group("data")
+    if m.group("b64"):
+        try:
+            return base64.b64decode(data, validate=False)
+        except (binascii.Error, ValueError) as exc:
+            raise ValueError(f"image data URI is not valid base64: {exc}")
+    from urllib.parse import unquote_to_bytes
+    return unquote_to_bytes(data)
+
+
+def load_image(url: str):
+    """One ``image_url`` as a PIL image in RGB.
+
+    Accepts a ``data:`` URI, an http(s) link and a local path, which is the
+    same set the llama.cpp multimodal path accepts -- a client cannot be
+    expected to know which backend is serving the model it asked for.
+    """
+    from PIL import Image
+
+    raw: Optional[bytes] = None
+    if url.startswith("data:"):
+        raw = _decode_data_uri(url)
+    elif url.startswith(("http://", "https://")):
+        import requests
+        resp = requests.get(url, timeout=_FETCH_TIMEOUT)
+        resp.raise_for_status()
+        raw = resp.content
+    elif os.path.exists(url):
+        with open(url, "rb") as fh:
+            raw = fh.read()
+    if raw is None:
+        raise ValueError(f"unsupported image reference: {url[:64]!r}")
+    img = Image.open(io.BytesIO(raw))
+    # Convert while the file handle is still open: PIL is lazy, and a converted
+    # copy is also what detaches the image from the BytesIO we are about to
+    # drop. RGB because a palette or RGBA image reaches the tower as the wrong
+    # channel count.
+    return img.convert("RGB")
+
+
+def has_images(messages) -> bool:
+    """Whether any turn carries a picture."""
+    for m in messages or []:
+        content = m.get("content") if isinstance(m, dict) else getattr(m, "content", None)
+        if not isinstance(content, list):
+            continue
+        for part in content:
+            if isinstance(part, dict) and part.get("type") in ("image_url", "input_image", "image"):
+                return True
+    return False
+
+
+def _part_url(part: Dict[str, Any]) -> Optional[str]:
+    t = part.get("type")
+    holder = part.get("image_url") if t == "image_url" else part.get("image")
+    if holder is None:
+        holder = part.get("url")
+    url = holder.get("url") if isinstance(holder, dict) else holder
+    return url if isinstance(url, str) and url else None
+
+
+def to_processor_messages(messages) -> Tuple[List[dict], List[Any]]:
+    """``(messages, images)`` in the shape a processor chat template wants.
+
+    Images are replaced by ``{"type": "image"}`` markers and handed over
+    separately, in the order they appear, because that is the contract every
+    HF image-text-to-text processor shares: the template emits one placeholder
+    run per marker and ``processor(text=…, images=[…])`` fills them in order.
+    A tool/assistant turn keeps its text; roles are passed through untouched so
+    the model's own template decides what to do with them.
+    """
+    out: List[dict] = []
+    images: List[Any] = []
+    for m in messages or []:
+        if isinstance(m, dict):
+            role = m.get("role") or "user"
+            content = m.get("content")
+        else:
+            role = getattr(m, "role", "user") or "user"
+            content = getattr(m, "content", None)
+        if not isinstance(content, list):
+            out.append({"role": role,
+                        "content": [{"type": "text", "text": "" if content is None else str(content)}]})
+            continue
+        parts: List[dict] = []
+        for part in content:
+            if not isinstance(part, dict):
+                parts.append({"type": "text", "text": str(part)})
+                continue
+            if part.get("type") in ("image_url", "input_image", "image"):
+                url = _part_url(part)
+                if not url:
+                    continue
+                images.append(load_image(url))
+                parts.append({"type": "image"})
+            elif "text" in part:
+                parts.append({"type": "text", "text": str(part.get("text") or "")})
+        if not parts:
+            parts = [{"type": "text", "text": ""}]
+        out.append({"role": role, "content": parts})
+    return out, images
+
+
+def build_inputs(processor, messages, add_generation_prompt: bool = True):
+    """Tokenized model inputs for a chat that contains pictures.
+
+    Rendered in two steps -- template to text, then ``processor(text, images)``
+    -- rather than the one-step ``apply_chat_template(tokenize=True)``, because
+    the two-step form is the one every family in transformers 5 supports and it
+    keeps the rendered prompt available for logging. The one-step form is the
+    fallback for a processor whose template refuses to render without its
+    images (some remote-code processors do).
+    """
+    msgs, images = to_processor_messages(messages)
+    try:
+        text = processor.apply_chat_template(
+            msgs, tokenize=False, add_generation_prompt=add_generation_prompt)
+        return processor(text=[text], images=images or None, return_tensors="pt"), images
+    except Exception as exc:
+        print(f"[vision] two-step prompt render failed ({exc}); using the "
+              f"processor's own tokenizing template")
+        inputs = processor.apply_chat_template(
+            _embed_images(msgs, images), add_generation_prompt=add_generation_prompt,
+            tokenize=True, return_dict=True, return_tensors="pt")
+        return inputs, images
+
+
+def _embed_images(msgs: List[dict], images: List[Any]) -> List[dict]:
+    """``msgs`` with each image marker carrying its own picture.
+
+    The one-step template reads the picture out of the content part itself,
+    while the two-step form takes them as a separate list. Same order either
+    way, so the markers are filled from the front.
+    """
+    pool = list(images)
+    out = []
+    for m in msgs:
+        parts = []
+        for part in m.get("content") or []:
+            if part.get("type") == "image" and pool:
+                parts.append({"type": "image", "image": pool.pop(0)})
+            else:
+                parts.append(part)
+        out.append({**m, "content": parts})
+    return out

codai/models/acceleration.py

@@ -423,3 +423,60 @@ def accel_call_defaults(accel: Optional[dict]) -> dict:
     if accel.get("guidance_scale") is not None:
         out["guidance_scale"] = float(accel["guidance_scale"])
     return out
+
+
+# A distilled checkpoint says so in its own name. These are the families whose
+# whole point is a low step count, with the step/guidance pair each one was
+# distilled for. The numbers are the conservative end of each family's
+# published range: an 8-step model given 8 steps is right, and a 4-step model
+# given 8 is slower than it needs to be but still correct, while the generic
+# 30-step default is wrong for both.
+#
+# `boundary` marks the tokens that are words rather than fragments, so
+# "turbovision" or "lcmx" cannot match on a substring.
+_DISTILLED_NAMES = (
+    # (token, boundary, steps, guidance_scale)
+    ("turbo",     True,  8, 1.0),
+    ("lightning", False, 8, 1.0),
+    ("lightx2v",  False, 8, 1.0),
+    ("hyper-sd",  False, 8, 1.0),
+    ("hyper_sd",  False, 8, 1.0),
+    ("schnell",   False, 4, 1.0),
+    ("lcm",       True,  6, 1.5),
+    ("tcd",       True,  8, 1.0),
+    ("dmd2",      False, 4, 1.0),
+    ("distill",   False, 8, 1.0),
+    ("sdxs",      True,  2, 1.0),
+)
+
+
+def distilled_defaults_for(model_id: str) -> Optional[dict]:
+    """``{steps, guidance_scale, token}`` when this id names a distilled model.
+
+    The last resort before the generic 30-step / CFG-7.5 fallback, and only
+    reached when the operator's card carries neither ``defaults`` nor an
+    ``acceleration`` block.
+
+    Measured on zeiss, 2026-10-29: ``Z-Image-Turbo`` -- a model distilled for 8
+    steps at CFG 1 -- was served at 30 steps and CFG 7.5 because its card had
+    no defaults, which is roughly four times the work for a picture that also
+    comes out over-guided. Both halves matter: a distilled model at CFG 7.5
+    looks burnt, so this is not only a speed fix.
+
+    A card's own `defaults` still win, because an operator who wrote a number
+    knows something this does not.
+    """
+    n = (model_id or "").lower().replace("\\", "/").rsplit("/", 1)[-1]
+    if not n:
+        return None
+    for token, boundary, steps, cfg in _DISTILLED_NAMES:
+        at = n.find(token)
+        while at != -1:
+            if not boundary:
+                return {"steps": steps, "guidance_scale": cfg, "token": token}
+            before = n[at - 1] if at > 0 else ""
+            after = n[at + len(token):at + len(token) + 1]
+            if not before.isalnum() and not after.isalnum():
+                return {"steps": steps, "guidance_scale": cfg, "token": token}
+            at = n.find(token, at + 1)
+    return None

codai/models/manager.py

@@ -6860,6 +6860,21 @@ class MultiModelManager:
                     caps = ModelCapabilities()      # drop the wrong guess
                 setattr(caps, TYPE_MIN_CAP[model_type], True)
             resolved_type = model_type or (caps.to_list()[0].split("_")[0] if caps.to_list() else "text")
+            # A GGUF text model with no projector cannot read an image, whatever
+            # its card says. llama.cpp needs an mmproj file to turn pixels into
+            # embeddings; without one the request reaches the model as a text
+            # placeholder and the answer describes nothing. Advertising
+            # `image_to_text` there sends clients to a model that will now
+            # (correctly) refuse them, so the listing has to agree with the
+            # runtime. Only local GGUF entries are judged here: an HF checkpoint
+            # loads its own vision tower when it has one, and a remote/pod-served
+            # model is the remote's business.
+            if caps.image_to_text and resolved_type == "text":
+                _p = str(meta.get("path") or meta.get("model_path") or "")
+                _backend = str(meta.get("backend") or "").lower()
+                if (_p.lower().endswith(".gguf") and not meta.get("mmproj")
+                        and _backend not in ("vllm", "runpod", "remote", "colibri")):
+                    caps.image_to_text = False
             # A video model carries its geometry: the frame rate it was trained at,
             # the frame counts its VAE can decode, how much one render holds and
             # whether it continues natively. Without it every client plans video by

docs/README.md

@@ -33,6 +33,7 @@ reference; these are the deep dives.
 | [zimage-lora-training.md](zimage-lora-training.md) | LoRA training for image models |
 | [text-to-music.md](text-to-music.md) | `POST /v1/audio/generate` with ACE-Step: style prompt + sung lyrics → a 48 kHz stereo track, the python-3.12 worker it needs, and the two image variants (`--with-acestep`) |
 | [dtype-auto-selection.md](dtype-auto-selection.md) | How model dtype is chosen automatically |
+| [vision-chat.md](vision-chat.md) | Images in a chat request: loading the vision tower the checkpoint's config asks for, the processor that makes a picture promptable, and why a model that cannot see refuses instead of answering blind |
 | [memory-accounting.md](memory-accounting.md) | Where `measured_vram_gb` comes from: walking the loaded model (quantization needs no factor), costing the KV cache from the real GQA/sliding-window geometry, refusing a measurement from a failed load, the PSS-based host-RAM cap, idle unload, and `GET /v1/models/history` |
 | [model-defaults-and-sources.md](model-defaults-and-sources.md) | A model's own `defaults` (steps/guidance/sampler/modes/licence), the sampler names and the flow-matching boundary, and where weights come from: repo, single file, `org/repo:file` or `civitai:<versionId>` |
 

docs/model-defaults-and-sources.md

@@ -36,7 +36,36 @@ Precedence, highest first:
 2. a **fused** distillation preset detected on the pipeline (`acceleration`),
    because it knows the LoRA actually baked in;
 3. `defaults` from the model entry;
-4. the generic floor (30 steps / CFG 7.5 for images, 25 / 7.5 for video).
+4. **what the model's own name says**, when nothing above answered (images);
+5. the generic floor (30 steps / CFG 7.5 for images, 25 / 7.5 for video).
+
+### 4: the name, when the card says nothing
+
+A card with no `defaults` and no `acceleration` block used to fall straight to
+the floor. Measured on a production fleet, 2026-10-29: `Z-Image-Turbo` — a
+checkpoint distilled for 8 steps at CFG 1 — was served at 30 steps and CFG
+7.5, about four times the work for a picture that also comes out over-guided,
+because a distilled model given real classifier-free guidance looks burnt.
+Roughly 72 s per 768×1344 still, for a model whose whole point is to take
+seconds.
+
+So when nothing better is available, the id itself is read:
+
+| in the id | steps | CFG |
+|---|---|---|
+| `turbo`, `lightning`, `lightx2v`, `hyper-sd`, `tcd`, `distill` | 8 | 1.0 |
+| `lcm` | 6 | 1.5 |
+| `schnell`, `dmd2` | 4 | 1.0 |
+| `sdxs` | 2 | 1.0 |
+
+Only the last path segment is read, so an org named `turbo-labs` proves
+nothing, and the word must stand on its own: `turbovision-xl` and `lcmx` do
+not match. The choice is logged with the number it replaced, because an
+unexplained step count is indistinguishable from a bug when a picture comes
+out wrong.
+
+This is a floor, not a recommendation: writing `defaults` on the card is still
+the right thing to do, and it wins.
 
 `/v1/models` publishes `defaults`, `modes` and `license` so a UI can show them
 instead of inventing them. `modes` is read from `defaults.modes`, a top-level

docs/vision-chat.md

+# Images in a chat request
+
+A model that is listed as able to read images has to read them. If it cannot,
+the request is refused — it is never answered blind.
+
+## What used to happen
+
+`/v1/models` listed `Qwen3.5-9B` with `["text_generation", "image_to_text"]`.
+A chat whose user turn carried an `image_url` part came back, for every
+picture, with a description of nothing:
+
+```json
+{"shows": "a blank white image with no visible content"}
+```
+
+Nine photographs of one video were indexed that way. They were real pictures —
+the footage scorer rated them 0.53–0.59 — and the model was not hallucinating.
+It was answering the prompt it had been given, which was the literal text
+`[image_url content]`.
+
+Two causes, measured on transformers 5.12 against `Qwen/Qwen3.5-9B`, whose
+`config.json` carries a `vision_config` and an `image_token_id`:
+
+| auto class | resolves to | holds the vision tower |
+|---|---|---|
+| `AutoModelForCausalLM` | `Qwen3_5ForCausalLM` | no |
+| `AutoModelForImageTextToText` | `Qwen3_5ForConditionalGeneration` | yes |
+
+The engine asked for the first one, so the tower was never built and its
+weights were never read. And with no projector loaded, the chat path flattened
+multipart content to a text placeholder, because that is all it could do with
+a picture.
+
+## What happens now
+
+**The class comes from the checkpoint's config, not from its name.**
+`codai/backends/hf_vision.py` reads the config: a nested `vision_config` or an
+`image_token_id`, plus an `AutoModelForImageTextToText` mapping in the
+installed transformers, means the model is loaded with the class that holds
+its tower. Every `from_pretrained` in the CUDA backend goes through that
+resolver, and any uncertainty — unreadable config, unknown family, missing
+mapping — falls back to the text class, so a model that is not a vision model
+loads exactly as it did before.
+
+Deciding from the name does not work in either direction: `Qwen3.5-9B` carries
+no "vl" or "vision" and is a vision model, while plenty of ids containing
+"vision" are not image-text-to-text at all.
+
+**A vision model also loads its processor.** The `AutoProcessor` — not the
+tokenizer — owns the image preprocessing and knows how many placeholder tokens
+this family expands one picture into. `supports_vision` is set only when both
+halves are present: a class that holds a tower *and* a processor that can
+preprocess for it. A tokenizer-shaped "processor" with no image processor does
+not count; transformers returns one for some text checkpoints and it would
+pass every duck-type check until the first picture arrived.
+
+**An image request takes its own generation path.** The prompt is rendered by
+the processor's chat template and then fed through `processor(text=…,
+images=[…])`, which yields `pixel_values` and the grid metadata the tower
+needs. Images travel as a separate list in the order they appear in the turn,
+because that order is what binds a picture to its placeholder.
+
+That path deliberately does not use the KV prefix cache. Image tokens are
+expanded from the pixels themselves, so a prefix cached from another request
+would be reused against the wrong picture.
+
+**A picture sent to a model that cannot see is a 400.**
+
+```
+Model 'X' cannot read images: it is served without a vision tower (an HF
+checkpoint with no vision_config or processor, or a GGUF with no mmproj).
+Send the image to a model whose capabilities list image_to_text, or
+configure an mmproj for this one.
+```
+
+A client can fall back from a refusal. It cannot fall back from a confident
+wrong answer, which is why this is an error and not a warning.
+
+**The listing agrees with the runtime.** A local GGUF text entry with no
+`mmproj` no longer advertises `image_to_text`: llama.cpp needs that projector
+file to turn pixels into embeddings, and without it the request would now be
+refused. HF checkpoints keep the capability — they load their own tower when
+they have one — and a remote or pod-served model is the remote's business.
+
+## Configuring vision
+
+* **HF checkpoint** (`text_models`, `vision_models`): nothing to set. If the
+  config has a vision tower and the installed transformers knows the family,
+  it is loaded and used.
+* **GGUF**: set `mmproj` on the entry to the projector file. Without it the
+  model is text-only, and is listed as text-only.
+* **Remote / pod-served** (vLLM, RunPod): vision is the remote's, and is
+  reported as supported.
+
+## Checking it
+
+```bash
+curl -s $HOST/v1/models | jq -r '.data[] | select(.capabilities[]? == "image_to_text") | .id'
+```
+
+Then send one picture to one of those ids, and the same picture to a text
+model. The first answers about the image; the second answers `400`. Neither
+describes a blank page.

tests/test_distilled_model_defaults.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""A turbo model must not be served at 30 steps and CFG 7.5.
+
+Measured on zeiss, 2026-10-29. The card for `Z-Image-Turbo` carried no
+``defaults`` and no ``acceleration`` block, so a request that did not name a
+step count got the generic fallback: 30 steps at CFG 7.5, against a checkpoint
+distilled for 8 steps at CFG 1. That is about four times the work, and the
+picture also comes out over-guided -- a distilled model at CFG 7.5 looks
+burnt. The client's own report said "a turbo, few-step model should take
+seconds" and measured ~72 s per 768x1344 still.
+
+The order of precedence is the point of these tests: the request wins, then a
+fused distill LoRA, then the operator's card, then the model's name, and only
+then the generic number. Each step down is a weaker claim about the same
+model, and nothing weaker may overrule anything stronger.
+"""
+
+import ast
+import io
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from codai.models.acceleration import distilled_defaults_for
+
+_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_SRC = io.open(os.path.join(_ROOT, "codai/api/images.py"), encoding="utf-8").read()
+
+
+def test_the_distilled_families_are_recognised():
+    assert distilled_defaults_for("Tongyi-MAI/Z-Image-Turbo")["steps"] == 8
+    assert distilled_defaults_for("unsloth/Krea-2-Turbo")["steps"] == 8
+    assert distilled_defaults_for("black-forest-labs/FLUX.1-schnell")["steps"] == 4
+    assert distilled_defaults_for("ByteDance/SDXL-Lightning")["steps"] == 8
+    lcm = distilled_defaults_for("latent-consistency/lcm-lora-sdxl")
+    assert (lcm["steps"], lcm["guidance_scale"]) == (6, 1.5), (
+        "LCM is the one family that wants guidance above 1")
+
+
+def test_a_distilled_model_is_never_given_real_cfg():
+    for name in ("Tongyi-MAI/Z-Image-Turbo", "stabilityai/sdxl-turbo",
+                 "black-forest-labs/FLUX.1-schnell", "ByteDance/Hyper-SD"):
+        got = distilled_defaults_for(name)
+        assert got is not None, name
+        assert got["guidance_scale"] <= 1.5, (
+            f"{name} would be served with classifier-free guidance it was "
+            f"distilled to do without")
+
+
+def test_a_non_distilled_model_is_left_alone():
+    """A false positive is the dangerous direction: it would serve a full
+    model at 8 steps, which produces an unfinished picture."""
+    for name in ("Tongyi-MAI/Z-Image", "stabilityai/stable-diffusion-xl-base-1.0",
+                 "black-forest-labs/FLUX.2-klein-4B", "krea/Krea-2"):
+        assert distilled_defaults_for(name) is None, name
+
+
+def test_a_word_inside_another_word_does_not_count():
+    assert distilled_defaults_for("John6666/turbovision-xl") is None
+    assert distilled_defaults_for("some/lcmx-model") is None
+    assert distilled_defaults_for("") is None
+    assert distilled_defaults_for(None) is None
+
+
+def test_only_the_last_path_segment_is_read():
+    """An org called "turbo-labs" does not make its models distilled."""
+    assert distilled_defaults_for("turbo-labs/plain-sdxl-finetune") is None
+
+
+def test_the_card_and_the_request_both_outrank_the_name():
+    """Checked on the source, because the precedence IS the behaviour: the
+    name is a guess, and a number an operator or a caller wrote is not."""
+    block = _SRC.split("_distilled = None", 1)[1][:2200]
+    assert "if not _d_steps and _accel_steps is None:" in block, (
+        "the name heuristic runs even when the card declared its own steps")
+    # In the expression itself, the distilled value must sit below the request,
+    # the fused LoRA and the card.
+    expr = block.split("num_steps = ", 1)[1].split("cfg_scale = ", 1)[0]
+    assert expr.index("request.steps") < expr.index("_accel_steps") < \
+        expr.index("_d_steps") < expr.index('_distilled["steps"]'), (
+        f"precedence is wrong in: {expr.strip()!r}")
+    assert expr.index('_distilled["steps"]') < expr.index("30 if quality"), (
+        "the generic 30-step fallback still wins over the distilled default")
+
+
+def test_the_choice_is_logged_with_what_it_replaced():
+    """An unexplained step count is indistinguishable from a bug when the
+    picture comes out wrong, so the line has to name both numbers."""
+    block = _SRC.split("_distilled = None", 1)[1][:2200]
+    assert "distilled" in block and "30 at CFG 7.5" in block

tests/test_hf_vision_chat.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""A model advertised as able to read images must read them, or refuse.
+
+Measured on 2026-10-29. ``Qwen/Qwen3.5-9B`` was listed with capabilities
+``["text_generation", "image_to_text"]`` and answered every photograph with
+"a blank white image with no visible content" -- nine web pictures of one
+video, all of them real images (the footage scorer rated them 0.53-0.59).
+
+Two defects, one symptom:
+
+* the engine loaded the checkpoint with ``AutoModelForCausalLM``, which for
+  ``Qwen3_5Config`` resolves to ``Qwen3_5ForCausalLM`` -- the text decoder on
+  its own. ``AutoModelForImageTextToText`` resolves to
+  ``Qwen3_5ForConditionalGeneration``, the one that holds the vision tower.
+* with no projector loaded, the chat path flattened the picture to the literal
+  text ``[image_url content]`` and answered that. The model was right; the
+  prompt was blank.
+
+So: load the tower when the checkpoint has one, and where it has none, refuse
+the request instead of answering it blind.
+"""
+
+import ast
+import io
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _src(rel):
+    return io.open(os.path.join(_ROOT, rel), encoding="utf-8").read()
+
+
+# --------------------------------------------------------------------------
+# The request shaping, which is pure and can be exercised directly
+# --------------------------------------------------------------------------
+
+def _png_data_uri(size=(8, 8), colour=(10, 20, 30)):
+    import base64
+    from PIL import Image
+    buf = io.BytesIO()
+    Image.new("RGB", size, colour).save(buf, format="PNG")
+    return "data:image/png;base64," + base64.b64encode(buf.getvalue()).decode()
+
+
+def test_an_image_part_is_recognised_in_any_of_its_spellings():
+    from codai.backends import hf_vision
+    for spelling in ("image_url", "input_image", "image"):
+        msgs = [{"role": "user", "content": [{"type": spelling, "image_url": {"url": "x"}}]}]
+        assert hf_vision.has_images(msgs), spelling
+    assert not hf_vision.has_images(
+        [{"role": "user", "content": [{"type": "text", "text": "no picture here"}]}])
+    assert not hf_vision.has_images([{"role": "user", "content": "plain string"}])
+
+
+def test_a_data_uri_image_is_decoded_to_rgb():
+    from codai.backends import hf_vision
+    img = hf_vision.load_image(_png_data_uri(size=(12, 7)))
+    assert img.size == (12, 7)
+    assert img.mode == "RGB", "a palette/RGBA image reaches the tower with the wrong channels"
+
+
+def test_a_corrupt_data_uri_is_an_error_not_a_blank_image():
+    from codai.backends import hf_vision
+    try:
+        hf_vision.load_image("data:image/png;base64,!!!not base64!!!")
+    except Exception:
+        return
+    raise AssertionError("a broken image decoded to something, which would be "
+                         "answered as though it were a picture")
+
+
+def test_images_are_handed_over_in_the_order_they_appear():
+    """The processor fills placeholders from a list, so order IS the binding.
+    Two pictures swapped means the answer describes the wrong one."""
+    from codai.backends import hf_vision
+    a = _png_data_uri(size=(4, 4))
+    b = _png_data_uri(size=(9, 9))
+    msgs = [
+        {"role": "user", "content": [{"type": "image_url", "image_url": {"url": a}},
+                                     {"type": "text", "text": "and"},
+                                     {"type": "image_url", "image_url": {"url": b}}]},
+    ]
+    out, images = hf_vision.to_processor_messages(msgs)
+    assert [i.size for i in images] == [(4, 4), (9, 9)]
+    kinds = [p["type"] for p in out[0]["content"]]
+    assert kinds == ["image", "text", "image"], kinds
+    # markers carry no pixels on the two-step path: the pictures travel in the
+    # separate list, and a marker that also held one would be counted twice
+    assert all(set(p) == {"type"} for p in out[0]["content"] if p["type"] == "image")
+
+
+def test_the_one_step_fallback_puts_each_picture_back_in_its_marker():
+    from codai.backends import hf_vision
+    msgs = [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": _png_data_uri()}},
+                                         {"type": "image_url", "image_url": {"url": _png_data_uri((5, 5))}}]}]
+    out, images = hf_vision.to_processor_messages(msgs)
+    embedded = hf_vision._embed_images(out, images)
+    got = [p.get("image") for p in embedded[0]["content"]]
+    assert got == images, "the fallback template would render markers with no pixels"
+
+
+def test_a_plain_text_turn_survives_the_reshaping():
+    from codai.backends import hf_vision
+    out, images = hf_vision.to_processor_messages(
+        [{"role": "system", "content": "be brief"},
+         {"role": "user", "content": [{"type": "text", "text": "hello"}]}])
+    assert images == []
+    assert out[0] == {"role": "system", "content": [{"type": "text", "text": "be brief"}]}
+    assert out[1]["content"] == [{"type": "text", "text": "hello"}]
+
+
+def test_a_processor_without_an_image_processor_is_not_vision_capable():
+    """transformers returns a tokenizer-shaped 'processor' for some text
+    checkpoints. It passes every duck-type check until the first picture."""
+    from codai.backends import hf_vision
+
+    class _TokenizerOnly:
+        pass
+
+    import transformers
+    _real = transformers.AutoProcessor
+    try:
+        class _Auto:
+            @staticmethod
+            def from_pretrained(*a, **k):
+                return _TokenizerOnly()
+        transformers.AutoProcessor = _Auto
+        assert hf_vision.load_processor("some/text-model") is None
+    finally:
+        transformers.AutoProcessor = _real
+
+
+# --------------------------------------------------------------------------
+# The load path
+# --------------------------------------------------------------------------
+
+def test_no_load_path_asks_for_the_text_class_directly():
+    """Every from_pretrained in the backend has to go through the resolver;
+    one forgotten call site is one model that silently loses its tower."""
+    src = _src("codai/backends/cuda.py")
+    assert "AutoModelForCausalLM.from_pretrained" not in src, (
+        "a load path still hardcodes the text-only class")
+
+
+def test_the_resolver_falls_back_to_text_on_any_failure():
+    from codai.backends import hf_vision
+    assert hf_vision.auto_model_class("this/does-not-exist-at-all")[1] is False
+    assert hf_vision.is_vision_checkpoint("this/does-not-exist-at-all") is False
+
+
+def test_vision_is_only_claimed_when_both_halves_loaded():
+    """`supports_vision` is what the API layer trusts to pass an image
+    through. A tower with no processor, or a processor with no tower, still
+    cannot answer about a picture."""
+    src = _src("codai/backends/cuda.py")
+    body = src.split("def _init_vision", 1)[1].split("\n    def ", 1)[0]
+    assert "self.supports_vision = False" in body.split("try:", 1)[0], (
+        "_init_vision does not start from 'cannot see'")
+    assert body.index("load_processor") < body.index("self.supports_vision = True"), (
+        "vision is claimed before the processor is known to exist")
+    assert "is_vision_checkpoint" in body
+
+
+def test_unloading_forgets_the_processor():
+    src = _src("codai/backends/cuda.py")
+    assert "self.supports_vision = False" in src.split("del self.tokenizer", 1)[1][:400], (
+        "a swap would leave the next model claiming the old one's vision")
+
+
+def test_the_vision_path_does_not_reuse_a_text_prefix_cache():
+    """Image tokens are expanded from the pixels by the processor, so a KV
+    prefix cached from another request would be reused against the wrong
+    picture. The vision path must not touch the prefix cache at all."""
+    src = _src("codai/backends/cuda.py")
+    body = src.split("def _generate_vision_chat", 1)[1].split("\n    async def ", 1)[0]
+    assert "_reuse_or_seed_prefix" not in body
+    assert "past_key_values" not in body
+
+
+def test_both_chat_entry_points_take_the_vision_branch():
+    src = _src("codai/backends/cuda.py")
+    tree = ast.parse(src)
+    found = {}
+    for node in ast.walk(tree):
+        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and \
+                node.name in ("generate_chat", "generate_chat_stream"):
+            found[node.name] = any(
+                isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute)
+                and n.func.attr == "_vision_request" for n in ast.walk(node))
+    assert found.get("generate_chat"), "non-streaming chat ignores images"
+    assert found.get("generate_chat_stream"), "streaming chat ignores images"
+
+
+# --------------------------------------------------------------------------
+# The refusal
+# --------------------------------------------------------------------------
+
+def test_an_image_to_a_blind_model_is_refused_before_it_is_flattened():
+    src = _src("codai/api/text.py")
+    guard = src.index("if not _vision_ok:")
+    flatten = src.index("[{item.get('type', 'unknown')} content]")
+    assert guard < flatten, (
+        "the placeholder is built before the refusal, so a blind model can "
+        "still answer a picture")
+    body = src[guard:guard + 1400]
+    assert "status_code=400" in body
+    assert "_release_instance()" in body, (
+        "refusing without releasing the instance slot leaks it")
+
+
+def test_a_gguf_without_an_mmproj_does_not_advertise_vision():
+    src = _src("codai/models/manager.py")
+    block = src.split('resolved_type = model_type or', 1)[1][:1600]
+    assert "caps.image_to_text = False" in block
+    assert '.gguf' in block and 'mmproj' in block
+    # a remote/pod-served model is the remote's business, not this listing's
+    assert "vllm" in block and "runpod" in block

tests/test_image_proactive_eviction_scope.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""The minimum-eviction block before an image load has to actually run.
+
+Observed on zeiss, 2026-10-29, on every image load:
+
+    Proactive eviction skipped: name 'request' is not defined
+
+``_load_diffusers_pipeline`` has no ``request`` in scope -- it takes
+``(model_name, global_args, model_config)``. The sizing call read it anyway,
+so the block raised NameError before it could evict anything, and the broad
+``except`` printed the NameError as a skip. Every image load then fell through
+to the blunt eviction further down, which frees by a different rule and
+unloads models that would have fitted alongside.
+
+A name error inside a try/except that reports "skipped" is invisible: the
+feature is simply never there. These tests fail on the shape that allows it.
+"""
+
+import ast
+import io
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+_SRC = io.open(os.path.join(_ROOT, "codai/api/images.py"), encoding="utf-8").read()
+
+
+def _fn(name):
+    for node in ast.walk(ast.parse(_SRC)):
+        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == name:
+            return node
+    raise AssertionError(f"{name} not found")
+
+
+def test_the_loader_does_not_read_a_request_it_never_received():
+    """Every name the function reads must be a parameter, a local, a global or
+    a builtin. `request` is none of those here."""
+    fn = _fn("_load_diffusers_pipeline")
+    params = {a.arg for a in fn.args.args} | {a.arg for a in fn.args.kwonlyargs}
+    assert "request" not in params
+    reads = [n for n in ast.walk(fn)
+             if isinstance(n, ast.Name) and n.id == "request" and isinstance(n.ctx, ast.Load)]
+    assert not reads, (
+        f"_load_diffusers_pipeline reads `request` at line(s) "
+        f"{[n.lineno for n in reads]}, which does not exist in its scope")
+
+
+def test_the_requested_id_is_captured_before_the_path_rewrite():
+    """Sizing needs the id the CALLER asked for: two cards can share one
+    checkpoint with different quantization and offload. `model_name` is
+    rewritten to the checkpoint path early on, so the id has to be kept
+    before that."""
+    fn = _fn("_load_diffusers_pipeline")
+    stores = [n.lineno for n in ast.walk(fn)
+              if isinstance(n, ast.Name) and n.id == "_requested_id"
+              and isinstance(n.ctx, ast.Store)]
+    loads = [n.lineno for n in ast.walk(fn)
+             if isinstance(n, ast.Name) and n.id == "_requested_id"
+             and isinstance(n.ctx, ast.Load)]
+    assert stores and loads, "the requested id is not kept at all"
+    rewrite = _SRC.index("model_name = _cfg_path")
+    rewrite_line = _SRC[:rewrite].count("\n") + 1
+    assert min(stores) < rewrite_line, (
+        f"_requested_id is assigned at line {min(stores)}, after model_name is "
+        f"rewritten to the checkpoint path at {rewrite_line} — it would hold "
+        f"the path, not the id that was asked for")
+    assert min(stores) < min(loads)
+
+
+def test_the_sizing_call_is_still_made_with_it():
+    block = _SRC.split("Proactive VRAM eviction", 1)[1][:2500]
+    assert "requested_id=_requested_id" in block
+    assert "_evict_models_for_vram" in block

tests/test_model_defaults_and_samplers.py

@@ -39,12 +39,20 @@ def test_the_request_beats_the_card_and_the_card_beats_the_floor():
 
     from codai.api import images
     src = inspect.getsource(images._generate_with_diffusers)
+    # The window is the step/guidance decision itself -- from where the card is
+    # read to where the guidance ends -- not a fixed slice. A fixed slice fails
+    # the moment another layer is inserted between the two, which says nothing
+    # about the precedence this test is here to protect.
     i = src.index("_dflt = model_defaults(model_config)")
-    window = src[i:i + 900]
+    window = src[i:src.index("_progress_reset(", i)]
     # request.steps first, then the fused-accel preset, then the card, then 30.
     assert "request.steps if request.steps" in window
     assert "_accel_steps if _accel_steps" in window
     assert "int(_d_steps) if _d_steps" in window
+    assert window.index("request.steps if request.steps") < \
+        window.index("_accel_steps if _accel_steps") < \
+        window.index("int(_d_steps) if _d_steps") < \
+        window.index("30 if quality")
 
 
 def test_video_layers_the_card_under_the_accel_preset_and_over_the_floor():