-
Stefy Lanza (nextime / spora ) authored
Support the three planned embedding models end-to-end (all verified on GPU): - 'vision' backend (DINOv2/ViT family): image-only encoders have no text tower or tokenizer, so none of the existing paths could load them. AutoImageProcessor + AutoModel, CLS/pooler embedding, normalized. Text input returns a clear 400. Verified: facebook/dinov2-large, 1024-d, similar-image 0.966 vs 0.750 cross. - 'qwenvl' backend (GME-Qwen2-VL…): the repo's remote code (auto_map + custom_st) pins transformers<4.52 and refuses to load on this stack, so load the plain Qwen2-VL weights through the CONCRETE native classes (which never consult auto_map) and reproduce GME's embedding recipe: their chat prompt format and last-token pooling, fp16, normalized. Shared 1536-d text+image space verified cross-modally (OCR-style test). No trust_remote_code needed. - ST image path: custom multimodal ST modules take {"image": …} dicts, not bare PIL (kept PIL-first for ST's native CLIP wrapper). - Tighten _supports_images: ST's plain Transformer module also exposes a `processor` attribute, which made text-only models (Qwen3-Embedding) look image-capable; now require an actual image-capable processor. Verified: Qwen/Qwen3-Embedding-4B (ST, 2560-d), facebook/dinov2-large (vision, 1024-d), Alibaba-NLP/gme-Qwen2-VL-2B-Instruct (qwenvl, 1536-d cross-modal). Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EPLnsRpNBzWCHLgkXATqRz
7980f408
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| admin | ||
| api | ||
| backends | ||
| broker | ||
| frontproxy | ||
| models | ||
| openai | ||
| pydantic | ||
| queue | ||
| tasks | ||
| __init__.py | ||
| cli.py | ||
| config.py | ||
| main.py | ||
| platform_paths.py | ||
| system_app.py |