-
Stefy Lanza (nextime / spora ) authored
A GGUF vision model answered every image question fluently and wrongly. On Qwen3.8-27B, with its own matching projector loaded and llama.cpp reporting "has vision encoder": * a 448x448 image added 16 tokens to the prompt (61 -> 77). The same image through the HF path added 156. * two completely different pictures -- a yellow field with a blue triangle and a green square, and three white circles on black -- produced byte-identical descriptions at temperature 0, neither resembling either image ("the image shows a series of exclamation marks"). * the same prompt with NO image answered differently, so the pictures were reaching the model and only their content was not. llama.cpp had been printing the reason on every single load: load_hparams: projector: qwen3vl_merger load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024 mtmd_context_params carries image_min_tokens and image_max_tokens, and nothing set either, so the count came from the image's own size alone: sixteen tokens, which cannot describe a photograph. The model did the only thing it could and answered from the prompt, with complete confidence. A 200 carrying a fluent wrong answer is the worst failure shape there is, because no caller can tell it from a right one -- and this one survived a full capability sweep, reported as "answers don't match the picture". So the floor is set: from the card's image_min_tokens when it says, and otherwise from the projector's own documented requirement, read from the mmproj GGUF's clip.projector_type rather than guessed from a filename (an mmproj is routinely renamed to sit beside the model it serves). Projectors with no documented floor are left exactly as they were, because forcing an upscale costs prefill on every request. The figure is printed at load, since it changes what every image costs. Upstream builds its context from mtmd_context_params_default() and sets four fields on the result, with no hook in between and the token bounds not among them. Rather than copy that method -- which would silently stop tracking any field upstream later adds -- the handler wraps the default-params factory for the duration of that one call, so the struct upstream receives already carries the bounds and everything upstream does to it still happens. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
6864eae1