embeddings: DINOv2 image-only backend + native GME-Qwen2-VL multimodal backend

Support the three planned embedding models end-to-end (all verified on GPU):

- 'vision' backend (DINOv2/ViT family): image-only encoders have no text tower
  or tokenizer, so none of the existing paths could load them. AutoImageProcessor
  + AutoModel, CLS/pooler embedding, normalized. Text input returns a clear 400.
  Verified: facebook/dinov2-large, 1024-d, similar-image 0.966 vs 0.750 cross.
- 'qwenvl' backend (GME-Qwen2-VL…): the repo's remote code (auto_map + custom_st)
  pins transformers<4.52 and refuses to load on this stack, so load the plain
  Qwen2-VL weights through the CONCRETE native classes (which never consult
  auto_map) and reproduce GME's embedding recipe: their chat prompt format and
  last-token pooling, fp16, normalized. Shared 1536-d text+image space verified
  cross-modally (OCR-style test). No trust_remote_code needed.
- ST image path: custom multimodal ST modules take {"image": …} dicts, not bare
  PIL (kept PIL-first for ST's native CLIP wrapper).
- Tighten _supports_images: ST's plain Transformer module also exposes a
  `processor` attribute, which made text-only models (Qwen3-Embedding) look
  image-capable; now require an actual image-capable processor.

Verified: Qwen/Qwen3-Embedding-4B (ST, 2560-d), facebook/dinov2-large (vision,
1024-d), Alibaba-NLP/gme-Qwen2-VL-2B-Instruct (qwenvl, 1536-d cross-modal).
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPLnsRpNBzWCHLgkXATqRz
parent f7787839
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment