embeddings: native GGUF Qwen2-VL image embeddings via llama.cpp mtmd

A GGUF embedder with a configured `mmproj` now loads as a vision-capable
'llama-vl' backend: the mtmd image tower feeds an embeddings context
(pooling LAST) under the GME chat prompt — the same scheme as the HF
qwenvl backend, so both produce the same vector space (~0.89 cosine
agreement at Q4). Runs natively on whatever the llama.cpp build targets
(Vulkan on radeon). Text goes through the same prompt+last-token path
so both modalities share the GME space; PIL-decoded images are passed
as raw RGB bitmaps (mtmd's stb can't read AVIF). Serialized on the
per-model lock; cleanup frees the mtmd ctx.

Reverts the front's image-reroute-to-HF-sibling: a model pinned to an
engine must be served there, not silently moved.

The gme mmproj was generated from the local HF checkpoint with
llama.cpp's convert_hf_to_gguf.py --mmproj (F16, 1.3 GB).
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
parent ffaf2e88
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API # Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here. # metadata and the admin web UI read from here.
__version__ = "0.1.42" __version__ = "0.1.43"
# Configure the CUDA caching allocator BEFORE torch is imported anywhere. # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
# expandable_segments lets the allocator return freed pages to the driver even # expandable_segments lets the allocator return freed pages to the driver even
......
This diff is collapsed.
...@@ -1705,36 +1705,6 @@ class FrontProxy: ...@@ -1705,36 +1705,6 @@ class FrontProxy:
body_bytes = await request.body() body_bytes = await request.body()
model = self._peek_model(body_bytes, request.headers.get("content-type", "")) model = self._peek_model(body_bytes, request.headers.get("content-type", ""))
# Modality-aware embedding reroute: a GGUF embedder (llama.cpp) has no
# image tower, so an `image` request against it would 400. When the SAME
# model is also registered as a non-GGUF (HF) entry, serve the image
# request from that sibling instead — the client keeps one model name;
# text requests stay on the GGUF entry's (e.g. radeon) engine.
if body_bytes is not None and "/embeddings" in path and model:
_gpath = str(self._model_info(model).get("path") or "")
if _gpath.lower().endswith(".gguf"):
try:
import json as _json
import re as _re
_b = _json.loads(body_bytes or b"{}")
except Exception:
_b = None
if isinstance(_b, dict) and _b.get("image"):
_stem = model.lower().split("/")[-1]
if _stem.endswith(".gguf"):
_stem = _stem[:-5]
# strip a trailing quant tag (…-Q4_K_M, ….i1-IQ4_XS, …)
_stem = _re.sub(r'[-.](i1-)?(iq|q)\d[\w-]*$', '', _stem)
_sib = self._model_info(_stem)
_spath = str(_sib.get("path") or "")
if _spath and not _spath.lower().endswith(".gguf"):
_b["model"] = _sib.get("model_id") or _spath
body_bytes = _json.dumps(_b).encode()
model = _b["model"]
print(f"[front] embeddings: image request for a GGUF "
f"(text-only) embedder rerouted to HF sibling "
f"'{model}'", flush=True)
engine = _router.pick_engine( engine = _router.pick_engine(
self.registry, path, method, model, self.registry, path, method, model,
required_cap=self._required_cap(path, model), required_cap=self._required_cap(path, model),
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment