• Stefy Lanza (nextime / spora )'s avatar
    embeddings: multimodal (text+image) support + correct VRAM tracking/eviction · dd848607
    Stefy Lanza (nextime / spora ) authored
    Wire up real image embeddings and make embedding models first-class in the
    VRAM lifecycle.
    
    - api/embeddings.py: detect CLIP/SigLIP dual encoders and drive them through
      transformers get_text_features/get_image_features so text and images share
      one projected space (ST path kept for repos shipping a native recipe).
      request.image is now actually read (URL/data-URI/path/base64), vectors are
      appended after the text ones, and text-only models return a clear 400.
      Handle the transformers 5.x pooled-output return shape.
    - Wrap the loaded model in _EmbeddingModel (unpacks as (backend, model) but
      exposes cleanup()) and register it via add_model() + record_vram_delta() on
      the request path, so it is measured, LRU-tracked, and cleanly evicted like
      every other model type instead of leaking as a bare tuple.
    - admin: fix the model-load button for embeddings (was routed to the diffusers
      loader) to use _load_embedding_model, matching the request path.
    - admin: backfill used_vram_gb after a download completes, since the entry is
      saved before the weights exist on disk; factor the estimate into a shared
      _estimate_used_vram_gb helper. A freshly-downloaded CLIP/SigLIP entry now
      always carries a reasonable estimate so pre-load eviction sizes correctly.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01EPLnsRpNBzWCHLgkXATqRz
    dd848607
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...