• Stefy Lanza (nextime / spora )'s avatar
    embeddings: multi-vector on the image that can serve it, and a 400 where it cannot (v0.3.73) · da2859f0
    Stefy Lanza (nextime / spora ) authored
    bge-m3's sparse and colbert come from linear heads (sparse_linear.pt /
    colbert_linear.pt) applied to the last hidden state -- a transformers forward
    pass, not an embedding server's output. Two consequences that were both silent:
    
    A DENSE-ONLY BACKEND NOW SAYS SO. `embedding_types` was simply ignored outside
    the native path: a caller asking for sparse got a normal-looking response with
    only `embedding` in it, its hybrid retrieval then ran on half the signal, and
    the result was merely a worse score -- which reads as a model problem, not a
    configuration one. Now a 400 that names the cause, where it DOES work
    (coderai-embeddings), and why the vLLM image cannot: that image is LIGHT core
    with no torch in its main venv BY DESIGN, so _load_bge_m3 could not even import.
    Dense-only requests are untouched.
    
    MULTI-VECTOR LONG INPUTS ARE WINDOWED. 8192 tokens is ~6000 words, so this bites
    on whole documents rather than passages -- and when it bit, the tokenizer's
    `truncation=True` kept the opening and said nothing. Each type is merged the way
    the encoder already merges WITHIN a window, which is what makes these choices
    defensible rather than invented:
    
      dense    mean, RE-NORMALIZED. It is an L2-normalized CLS vector and a mean of
               unit vectors is not one; cosine similarity on a non-unit vector is
               quietly wrong.
      sparse   MAX per token id -- the encoder max-pools per token id already
               (scatter_reduce amax), and a term's weight should be its strongest
               occurrence, not an average diluted by the pages that never mention
               it. A test guards against someone later tidying max into mean.
      colbert  CONCATENATED: max-sim scores each query token against ALL document
               token vectors, so every content token of every window belongs.
    
    `embed_long_input=truncate` keeps the old behaviour explicitly, and `first`
    keeps only the opening window, for an operator who would rather have that than a
    merge.
    
    Also removes three customer references I had put in shipped profile comments,
    two of which went out in 0.3.69. A capability must be scoped and justified by
    what it does, never by one install's needs, and a customer name does not belong
    in a released image. The measurements keep their force without the attribution:
    "hybrid retrieval wants dense and sparse from one pass", "a production fleet",
    "a GPU-less orchestrator".
    
    39 tests. Suite at the 25 pre-existing failures.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    da2859f0
Name
Last commit
Last update
..
common Loading commit data...
dinov2cpp Loading commit data...
linux Loading commit data...
runpod Loading commit data...
windows Loading commit data...
build-rpc-server.sh Loading commit data...
cosign.pub Loading commit data...
patch-colibri.py Loading commit data...
patch-k3.py Loading commit data...
setup-salad-venv.sh Loading commit data...
sign-images.sh Loading commit data...
versions.env Loading commit data...