• Stefy Lanza (nextime / spora )'s avatar
    embeddings: DINOv2 image-only backend + native GME-Qwen2-VL multimodal backend · 7980f408
    Stefy Lanza (nextime / spora ) authored
    Support the three planned embedding models end-to-end (all verified on GPU):
    
    - 'vision' backend (DINOv2/ViT family): image-only encoders have no text tower
      or tokenizer, so none of the existing paths could load them. AutoImageProcessor
      + AutoModel, CLS/pooler embedding, normalized. Text input returns a clear 400.
      Verified: facebook/dinov2-large, 1024-d, similar-image 0.966 vs 0.750 cross.
    - 'qwenvl' backend (GME-Qwen2-VL…): the repo's remote code (auto_map + custom_st)
      pins transformers<4.52 and refuses to load on this stack, so load the plain
      Qwen2-VL weights through the CONCRETE native classes (which never consult
      auto_map) and reproduce GME's embedding recipe: their chat prompt format and
      last-token pooling, fp16, normalized. Shared 1536-d text+image space verified
      cross-modally (OCR-style test). No trust_remote_code needed.
    - ST image path: custom multimodal ST modules take {"image": …} dicts, not bare
      PIL (kept PIL-first for ST's native CLIP wrapper).
    - Tighten _supports_images: ST's plain Transformer module also exposes a
      `processor` attribute, which made text-only models (Qwen3-Embedding) look
      image-capable; now require an actual image-capable processor.
    
    Verified: Qwen/Qwen3-Embedding-4B (ST, 2560-d), facebook/dinov2-large (vision,
    1024-d), Alibaba-NLP/gme-Qwen2-VL-2B-Instruct (qwenvl, 1536-d cross-modal).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01EPLnsRpNBzWCHLgkXATqRz
    7980f408
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...