• Stefy Lanza (nextime / spora )'s avatar
    give a picture enough tokens to be described · 6864eae1
    Stefy Lanza (nextime / spora ) authored
    A GGUF vision model answered every image question fluently and wrongly. On
    Qwen3.8-27B, with its own matching projector loaded and llama.cpp reporting
    "has vision encoder":
    
      * a 448x448 image added 16 tokens to the prompt (61 -> 77). The same image
        through the HF path added 156.
      * two completely different pictures -- a yellow field with a blue triangle
        and a green square, and three white circles on black -- produced
        byte-identical descriptions at temperature 0, neither resembling either
        image ("the image shows a series of exclamation marks").
      * the same prompt with NO image answered differently, so the pictures were
        reaching the model and only their content was not.
    
    llama.cpp had been printing the reason on every single load:
    
        load_hparams: projector: qwen3vl_merger
        load_hparams: Qwen-VL models require at minimum 1024 image tokens to
                      function correctly on grounding tasks
        load_hparams: if you encounter problems with accuracy, try adding
                      --image-min-tokens 1024
    
    mtmd_context_params carries image_min_tokens and image_max_tokens, and nothing
    set either, so the count came from the image's own size alone: sixteen tokens,
    which cannot describe a photograph. The model did the only thing it could and
    answered from the prompt, with complete confidence. A 200 carrying a fluent
    wrong answer is the worst failure shape there is, because no caller can tell it
    from a right one -- and this one survived a full capability sweep, reported as
    "answers don't match the picture".
    
    So the floor is set: from the card's image_min_tokens when it says, and
    otherwise from the projector's own documented requirement, read from the mmproj
    GGUF's clip.projector_type rather than guessed from a filename (an mmproj is
    routinely renamed to sit beside the model it serves). Projectors with no
    documented floor are left exactly as they were, because forcing an upscale
    costs prefill on every request. The figure is printed at load, since it changes
    what every image costs.
    
    Upstream builds its context from mtmd_context_params_default() and sets four
    fields on the result, with no hook in between and the token bounds not among
    them. Rather than copy that method -- which would silently stop tracking any
    field upstream later adds -- the handler wraps the default-params factory for
    the duration of that one call, so the struct upstream receives already carries
    the bounds and everything upstream does to it still happens.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    6864eae1
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
compose Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pipelines Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
util Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
hf_cache.py Loading commit data...
main.py Loading commit data...
ocr_fleet_workers.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...