Skip to content
Commit 6864eae1 authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files

give a picture enough tokens to be described



A GGUF vision model answered every image question fluently and wrongly. On
Qwen3.8-27B, with its own matching projector loaded and llama.cpp reporting
"has vision encoder":

  * a 448x448 image added 16 tokens to the prompt (61 -> 77). The same image
    through the HF path added 156.
  * two completely different pictures -- a yellow field with a blue triangle
    and a green square, and three white circles on black -- produced
    byte-identical descriptions at temperature 0, neither resembling either
    image ("the image shows a series of exclamation marks").
  * the same prompt with NO image answered differently, so the pictures were
    reaching the model and only their content was not.

llama.cpp had been printing the reason on every single load:

    load_hparams: projector: qwen3vl_merger
    load_hparams: Qwen-VL models require at minimum 1024 image tokens to
                  function correctly on grounding tasks
    load_hparams: if you encounter problems with accuracy, try adding
                  --image-min-tokens 1024

mtmd_context_params carries image_min_tokens and image_max_tokens, and nothing
set either, so the count came from the image's own size alone: sixteen tokens,
which cannot describe a photograph. The model did the only thing it could and
answered from the prompt, with complete confidence. A 200 carrying a fluent
wrong answer is the worst failure shape there is, because no caller can tell it
from a right one -- and this one survived a full capability sweep, reported as
"answers don't match the picture".

So the floor is set: from the card's image_min_tokens when it says, and
otherwise from the projector's own documented requirement, read from the mmproj
GGUF's clip.projector_type rather than guessed from a filename (an mmproj is
routinely renamed to sit beside the model it serves). Projectors with no
documented floor are left exactly as they were, because forcing an upscale
costs prefill on every request. The figure is printed at load, since it changes
what every image costs.

Upstream builds its context from mtmd_context_params_default() and sets four
fields on the result, with no hook in between and the token bounds not among
them. Rather than copy that method -- which would silently stop tracking any
field upstream later adds -- the handler wraps the default-params factory for
the duration of that one call, so the struct upstream receives already carries
the bounds and everything upstream does to it still happens.

Co-Authored-By: default avatarClaude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
parent 56af3f00
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment