embeddings: DINOv2 image-only backend + native GME-Qwen2-VL multimodal backend
Support the three planned embedding models end-to-end (all verified on GPU):
- 'vision' backend (DINOv2/ViT family): image-only encoders have no text tower
or tokenizer, so none of the existing paths could load them. AutoImageProcessor
+ AutoModel, CLS/pooler embedding, normalized. Text input returns a clear 400.
Verified: facebook/dinov2-large, 1024-d, similar-image 0.966 vs 0.750 cross.
- 'qwenvl' backend (GME-Qwen2-VL…): the repo's remote code (auto_map + custom_st)
pins transformers<4.52 and refuses to load on this stack, so load the plain
Qwen2-VL weights through the CONCRETE native classes (which never consult
auto_map) and reproduce GME's embedding recipe: their chat prompt format and
last-token pooling, fp16, normalized. Shared 1536-d text+image space verified
cross-modally (OCR-style test). No trust_remote_code needed.
- ST image path: custom multimodal ST modules take {"image": …} dicts, not bare
PIL (kept PIL-first for ST's native CLIP wrapper).
- Tighten _supports_images: ST's plain Transformer module also exposes a
`processor` attribute, which made text-only models (Qwen3-Embedding) look
image-capable; now require an actual image-capable processor.
Verified: Qwen/Qwen3-Embedding-4B (ST, 2560-d), facebook/dinov2-large (vision,
1024-d), Alibaba-NLP/gme-Qwen2-VL-2B-Instruct (qwenvl, 1536-d cross-modal).
Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPLnsRpNBzWCHLgkXATqRz
Showing
This diff is collapsed.
Please
register
or
sign in
to comment