make the multimodal path visible in the log

Two completely different pictures sent to a GGUF model with a configured
mmproj produced byte-identical descriptions, while the same prompt with NO
image answered differently. So the images reached the model and their content
did not -- the embeddings carry nothing -- and the log said nothing either way:
projector init, media-marker substitution, bitmap decode and chunk eval all
log under names ('mtmd', 'clip', 'bitmap', 'media', 'projector') that the
load-phase filter did not have, so every one of them was discarded.

Reproduced with two synthetic images (yellow background with a blue triangle
and a green square; black background with three white circles) at temperature
0: identical answers, neither resembling either picture, both unlike the
no-image control. Keeping these lines is what makes the next request say which
step is losing the pixels.
Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
parent da335aeb
...@@ -661,6 +661,15 @@ def _install_layer_log_callback(): ...@@ -661,6 +661,15 @@ def _install_layer_log_callback():
'offload', 'layer', 'buffer size', 'buffer type', 'offload', 'layer', 'buffer size', 'buffer type',
'GPU', 'CUDA', 'Vulkan', 'Metal', 'ROCm', 'SYCL', 'GPU', 'CUDA', 'Vulkan', 'Metal', 'ROCm', 'SYCL',
'CPU', 'VRAM', 'n_layer', 'n_gpu_layers', 'CPU', 'VRAM', 'n_layer', 'n_gpu_layers',
# The multimodal path, which was invisible here. Measured on a GGUF
# model with a configured mmproj: two completely different pictures
# produced byte-identical descriptions, while the same prompt with NO
# image answered differently -- so the images reached the model and
# their content did not. Everything mtmd does to get a bitmap into the
# prompt (projector init, marker substitution, bitmap decode, chunk
# eval) logs under these names and none of them appear above, so there
# was nothing in the log to read either way.
'mtmd', 'clip', 'bitmap', 'media', 'vision', 'projector', 'n_embd',
) )
@_llama_cpp.llama_log_callback @_llama_cpp.llama_log_callback
......
...@@ -316,3 +316,16 @@ def test_the_xtts_worker_recognises_an_org_less_alias(): ...@@ -316,3 +316,16 @@ def test_the_xtts_worker_recognises_an_org_less_alias():
# Already coqui's own four-part naming, and an unrelated model: untouched. # Already coqui's own four-part naming, and an unrelated model: untouched.
assert mod._coqui_name(four) == four assert mod._coqui_name(four) == four
assert mod._coqui_name("some/other-model") == "some/other-model" assert mod._coqui_name("some/other-model") == "some/other-model"
def test_the_multimodal_path_is_visible_in_the_log():
"""Two different pictures produced byte-identical descriptions from a GGUF
model with a configured mmproj, while the same prompt with no image
answered differently — the images reached the model and their content did
not. Nothing mtmd does was printed: projector init, marker substitution,
bitmap decode and chunk eval all log under names the filter did not have."""
src = _src("codai/backends/vulkan.py")
head = src.index("_KEEP = (")
keep = src[head:src.index("\n )", head)]
for word in ("mtmd", "clip", "bitmap", "media", "projector"):
assert f"'{word}'" in keep, word
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment