front: reroute image embedding requests off GGUF (text-only) embedders

llama.cpp's embedding API has no image tower, so `image` requests to a
GGUF embedder 400'd. When the same model is also registered as a
non-GGUF HF entry (e.g. gme-Qwen2-VL GGUF on radeon + HF on nvidia),
the front now rewrites the image request to that sibling — one model
name for clients; text stays pinned to the GGUF's engine, images go to
the engine that has the vision tower.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
parent 531ccfba
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API # Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here. # metadata and the admin web UI read from here.
__version__ = "0.1.41" __version__ = "0.1.42"
# Configure the CUDA caching allocator BEFORE torch is imported anywhere. # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
# expandable_segments lets the allocator return freed pages to the driver even # expandable_segments lets the allocator return freed pages to the driver even
......
...@@ -1705,6 +1705,36 @@ class FrontProxy: ...@@ -1705,6 +1705,36 @@ class FrontProxy:
body_bytes = await request.body() body_bytes = await request.body()
model = self._peek_model(body_bytes, request.headers.get("content-type", "")) model = self._peek_model(body_bytes, request.headers.get("content-type", ""))
# Modality-aware embedding reroute: a GGUF embedder (llama.cpp) has no
# image tower, so an `image` request against it would 400. When the SAME
# model is also registered as a non-GGUF (HF) entry, serve the image
# request from that sibling instead — the client keeps one model name;
# text requests stay on the GGUF entry's (e.g. radeon) engine.
if body_bytes is not None and "/embeddings" in path and model:
_gpath = str(self._model_info(model).get("path") or "")
if _gpath.lower().endswith(".gguf"):
try:
import json as _json
import re as _re
_b = _json.loads(body_bytes or b"{}")
except Exception:
_b = None
if isinstance(_b, dict) and _b.get("image"):
_stem = model.lower().split("/")[-1]
if _stem.endswith(".gguf"):
_stem = _stem[:-5]
# strip a trailing quant tag (…-Q4_K_M, ….i1-IQ4_XS, …)
_stem = _re.sub(r'[-.](i1-)?(iq|q)\d[\w-]*$', '', _stem)
_sib = self._model_info(_stem)
_spath = str(_sib.get("path") or "")
if _spath and not _spath.lower().endswith(".gguf"):
_b["model"] = _sib.get("model_id") or _spath
body_bytes = _json.dumps(_b).encode()
model = _b["model"]
print(f"[front] embeddings: image request for a GGUF "
f"(text-only) embedder rerouted to HF sibling "
f"'{model}'", flush=True)
engine = _router.pick_engine( engine = _router.pick_engine(
self.registry, path, method, model, self.registry, path, method, model,
required_cap=self._required_cap(path, model), required_cap=self._required_cap(path, model),
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment