• Stefy Lanza (nextime / spora )'s avatar
    embeddings: DINOv2 image-only backend + native GME-Qwen2-VL multimodal backend · 7980f408
    Stefy Lanza (nextime / spora ) authored
    Support the three planned embedding models end-to-end (all verified on GPU):
    
    - 'vision' backend (DINOv2/ViT family): image-only encoders have no text tower
      or tokenizer, so none of the existing paths could load them. AutoImageProcessor
      + AutoModel, CLS/pooler embedding, normalized. Text input returns a clear 400.
      Verified: facebook/dinov2-large, 1024-d, similar-image 0.966 vs 0.750 cross.
    - 'qwenvl' backend (GME-Qwen2-VL…): the repo's remote code (auto_map + custom_st)
      pins transformers<4.52 and refuses to load on this stack, so load the plain
      Qwen2-VL weights through the CONCRETE native classes (which never consult
      auto_map) and reproduce GME's embedding recipe: their chat prompt format and
      last-token pooling, fp16, normalized. Shared 1536-d text+image space verified
      cross-modally (OCR-style test). No trust_remote_code needed.
    - ST image path: custom multimodal ST modules take {"image": …} dicts, not bare
      PIL (kept PIL-first for ST's native CLIP wrapper).
    - Tighten _supports_images: ST's plain Transformer module also exposes a
      `processor` attribute, which made text-only models (Qwen3-Embedding) look
      image-capable; now require an actual image-capable processor.
    
    Verified: Qwen/Qwen3-Embedding-4B (ST, 2560-d), facebook/dinov2-large (vision,
    1024-d), Alibaba-NLP/gme-Qwen2-VL-2B-Instruct (qwenvl, 1536-d cross-modal).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01EPLnsRpNBzWCHLgkXATqRz
    7980f408
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...