Skip to content
Commit 18ab5ad1 authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files

embeddings: GGUF backend via llama.cpp embedding mode



A .gguf path in the embedding registry now loads through llama_cpp
(embedding=True, n_ctx/n_gpu_layers from the model config) instead of
failing in SentenceTransformer. Per-token outputs are mean-pooled,
vectors normalized, matryoshka `dimensions` truncation reuses the
shared path. cleanup() closes the llama ctx so eviction frees VRAM.
Text-only: image requests against a GGUF embedder keep returning 400.

Co-Authored-By: default avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S8VtAvG499SsCbeESRK7V
parent 540566b0
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment