• Stefy Lanza (nextime / spora )'s avatar
    embeddings: native GGUF Qwen2-VL image embeddings via llama.cpp mtmd · a3e473c6
    Stefy Lanza (nextime / spora ) authored
    A GGUF embedder with a configured `mmproj` now loads as a vision-capable
    'llama-vl' backend: the mtmd image tower feeds an embeddings context
    (pooling LAST) under the GME chat prompt — the same scheme as the HF
    qwenvl backend, so both produce the same vector space (~0.89 cosine
    agreement at Q4). Runs natively on whatever the llama.cpp build targets
    (Vulkan on radeon). Text goes through the same prompt+last-token path
    so both modalities share the GME space; PIL-decoded images are passed
    as raw RGB bitmaps (mtmd's stb can't read AVIF). Serialized on the
    per-model lock; cleanup frees the mtmd ctx.
    
    Reverts the front's image-reroute-to-HF-sibling: a model pinned to an
    engine must be served there, not silently moved.
    
    The gme mmproj was generated from the local HF checkpoint with
    llama.cpp's convert_hf_to_gguf.py --mmproj (F16, 1.3 GB).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
    a3e473c6
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...