• Stefy Lanza (nextime / spora )'s avatar
    embeddings: DINOv2 image-only backend + native GME-Qwen2-VL multimodal backend · 7980f408
    Stefy Lanza (nextime / spora ) authored
    Support the three planned embedding models end-to-end (all verified on GPU):
    
    - 'vision' backend (DINOv2/ViT family): image-only encoders have no text tower
      or tokenizer, so none of the existing paths could load them. AutoImageProcessor
      + AutoModel, CLS/pooler embedding, normalized. Text input returns a clear 400.
      Verified: facebook/dinov2-large, 1024-d, similar-image 0.966 vs 0.750 cross.
    - 'qwenvl' backend (GME-Qwen2-VL…): the repo's remote code (auto_map + custom_st)
      pins transformers<4.52 and refuses to load on this stack, so load the plain
      Qwen2-VL weights through the CONCRETE native classes (which never consult
      auto_map) and reproduce GME's embedding recipe: their chat prompt format and
      last-token pooling, fp16, normalized. Shared 1536-d text+image space verified
      cross-modally (OCR-style test). No trust_remote_code needed.
    - ST image path: custom multimodal ST modules take {"image": …} dicts, not bare
      PIL (kept PIL-first for ST's native CLIP wrapper).
    - Tighten _supports_images: ST's plain Transformer module also exposes a
      `processor` attribute, which made text-only models (Qwen3-Embedding) look
      image-capable; now require an actual image-capable processor.
    
    Verified: Qwen/Qwen3-Embedding-4B (ST, 2560-d), facebook/dinov2-large (vision,
    1024-d), Alibaba-NLP/gme-Qwen2-VL-2B-Instruct (qwenvl, 1536-d cross-modal).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01EPLnsRpNBzWCHLgkXATqRz
    7980f408
embeddings.py 27.1 KB