embeddings: multi-vector on the image that can serve it, and a 400 where it cannot (v0.3.73)

bge-m3's sparse and colbert come from linear heads (sparse_linear.pt /
colbert_linear.pt) applied to the last hidden state -- a transformers forward
pass, not an embedding server's output. Two consequences that were both silent:

A DENSE-ONLY BACKEND NOW SAYS SO. `embedding_types` was simply ignored outside
the native path: a caller asking for sparse got a normal-looking response with
only `embedding` in it, its hybrid retrieval then ran on half the signal, and
the result was merely a worse score -- which reads as a model problem, not a
configuration one. Now a 400 that names the cause, where it DOES work
(coderai-embeddings), and why the vLLM image cannot: that image is LIGHT core
with no torch in its main venv BY DESIGN, so _load_bge_m3 could not even import.
Dense-only requests are untouched.

MULTI-VECTOR LONG INPUTS ARE WINDOWED. 8192 tokens is ~6000 words, so this bites
on whole documents rather than passages -- and when it bit, the tokenizer's
`truncation=True` kept the opening and said nothing. Each type is merged the way
the encoder already merges WITHIN a window, which is what makes these choices
defensible rather than invented:

  dense    mean, RE-NORMALIZED. It is an L2-normalized CLS vector and a mean of
           unit vectors is not one; cosine similarity on a non-unit vector is
           quietly wrong.
  sparse   MAX per token id -- the encoder max-pools per token id already
           (scatter_reduce amax), and a term's weight should be its strongest
           occurrence, not an average diluted by the pages that never mention
           it. A test guards against someone later tidying max into mean.
  colbert  CONCATENATED: max-sim scores each query token against ALL document
           token vectors, so every content token of every window belongs.

`embed_long_input=truncate` keeps the old behaviour explicitly, and `first`
keeps only the opening window, for an operator who would rather have that than a
merge.

Also removes three customer references I had put in shipped profile comments,
two of which went out in 0.3.69. A capability must be scoped and justified by
what it does, never by one install's needs, and a customer name does not belong
in a released image. The measurements keep their force without the attribution:
"hybrid retrieval wants dense and sparse from one pass", "a production fleet",
"a GPU-less orchestrator".

39 tests. Suite at the 25 pre-existing failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.72"
+__version__ = "0.3.73"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/embeddings.py

@@ -1323,6 +1323,59 @@ def _bge_m3_encode(model_obj, texts: List[str], types, max_length: int = 8192):
     return results
 
 
+def _bge_m3_merge(parts: List[dict], types, pool: str = "mean") -> dict:
+    """Merge one text's per-window bge-m3 results into one.
+
+    None of these three is a free choice: each is what the encoder ALREADY does
+    within a single window, extended across windows.
+
+      dense    mean of the window vectors, RE-NORMALIZED. Dense is an
+               L2-normalized CLS vector, and a mean of unit vectors is not one;
+               a caller doing cosine similarity on it would get quietly wrong
+               numbers.
+      sparse   MAX per token id. The encoder max-pools per token id inside a
+               window already (`scatter_reduce_(..., reduce='amax')`), and max
+               is also what retrieval wants: a term's weight is its strongest
+               occurrence in the document, not an average diluted by the pages
+               that do not mention it.
+      colbert  CONCATENATED. Colbert scores max-sim of each query token against
+               ALL document token vectors, so every content token of every
+               window belongs in the list. Note this grows with the document --
+               a 40-page input yields 40 windows' worth of token vectors.
+
+    `pool="first"` keeps only the opening window, for an operator who would
+    rather have the old truncating behaviour explicitly.
+    """
+    if not parts:
+        return {}
+    if len(parts) == 1:
+        return parts[0]
+    if pool == "first":
+        return parts[0]
+    out: dict = {}
+    if 'dense' in types:
+        import math
+        vecs = [p['dense'] for p in parts if p.get('dense')]
+        if vecs:
+            n = len(vecs)
+            mean = [sum(col) / n for col in zip(*vecs)]
+            norm = math.sqrt(sum(x * x for x in mean)) or 1.0
+            out['dense'] = [x / norm for x in mean]
+    if 'sparse' in types:
+        merged: dict = {}
+        for p in parts:
+            for tid, w in (p.get('sparse') or {}).items():
+                if w > merged.get(tid, 0.0):
+                    merged[tid] = w
+        out['sparse'] = merged
+    if 'colbert' in types:
+        toks = []
+        for p in parts:
+            toks.extend(p.get('colbert') or [])
+        out['colbert'] = toks
+    return out
+
+
 async def _bge_m3_response(request, model_obj, texts, _emb_cfg):
     """Build the extended /v1/embeddings response for a bge-m3 model: dense in the usual
     `embedding` field, plus `sparse_embedding` / `colbert_embedding` per requested type.
@@ -1347,9 +1400,41 @@ async def _bge_m3_response(request, model_obj, texts, _emb_cfg):
             raise HTTPException(status_code=400,
                                 detail=f"This model has no {t} head.")
     max_len = int(_emb_cfg.get('n_ctx') or _raw.get('n_ctx') or 8192)
+    # Long inputs: window them rather than letting the tokenizer's
+    # `truncation=True` keep the opening and say nothing. bge-m3's window is
+    # 8192 tokens -- roughly 6000 words, 15-20 pages -- so this bites on whole
+    # documents rather than passages, and when it bites it was invisible.
+    _mode, _pool = _embed_chunk_cfg(getattr(model_obj, 'name', None)
+                                    or getattr(request, 'model', None))
+    _tokzr = model_obj.model[0]
+    _chunks: List[int] = []
 
     def _run():
-        return _bge_m3_encode(model_obj, texts, types, max_len)
+        if _mode == "truncate":
+            return _bge_m3_encode(model_obj, texts, types, max_len)
+
+        def _enc(t):
+            return _tokzr.encode(t, add_special_tokens=True)
+
+        def _dec(ids):
+            return _tokzr.decode(ids, skip_special_tokens=True)
+
+        windows, owner = [], []
+        for idx, t in enumerate(texts):
+            ws = _token_windows(t, max(64, max_len - 16), _enc, _dec)
+            if len(ws) > 1 and _mode == "error":
+                raise ValueError(
+                    f"input {idx} needs {len(ws)} windows of {max_len} tokens; "
+                    f"embed_long_input=error refuses to merge it.")
+            _chunks.append(len(ws))
+            for w in ws:
+                windows.append(w)
+                owner.append(idx)
+        flat = _bge_m3_encode(model_obj, windows, types, max_len)
+        grouped = [[] for _ in texts]
+        for part, idx in zip(flat, owner):
+            grouped[idx].append(part)
+        return [_bge_m3_merge(g, types, _pool) for g in grouped]
 
     try:
         results = await asyncio.get_event_loop().run_in_executor(None, _run)
@@ -1997,6 +2082,40 @@ async def _run_embeddings(request: EmbeddingsRequest, http_request: Request = No
     if getattr(model_obj, 'backend', None) == 'bge-m3':
         return await _bge_m3_response(request, model_obj, texts, _emb_cfg)
 
+    # SPARSE/COLBERT ASKED OF A DENSE-ONLY BACKEND. Until 0.3.73 the
+    # `embedding_types` field was simply ignored here: a caller asking for
+    # sparse got a normal-looking response with only `embedding` in it and no
+    # indication that half of what it asked for does not exist on this path.
+    # Silence is the worst answer -- the caller's hybrid retrieval then runs on
+    # dense alone and merely scores worse, which reads as a model problem.
+    #
+    # vLLM is the case that matters in practice: it serves embeddings DENSE
+    # ONLY. bge-m3's sparse and colbert come from extra linear heads
+    # (sparse_linear.pt / colbert_linear.pt) applied to the last hidden state,
+    # which is a transformers forward pass this backend does not do. And a
+    # `coderai-vllm` pod cannot fall back to the native path either: it is a
+    # LIGHT-core image with no torch in its main venv by design, so
+    # _load_bge_m3 could not even import. The image that serves multi-vector is
+    # `coderai-embeddings` (full core, torch + transformers), where
+    # _is_bge_m3() detects the head and picks the path automatically.
+    _want_types = [str(t).strip().lower()
+                   for t in (getattr(request, 'embedding_types', None) or [])
+                   if str(t).strip()]
+    _extra = [t for t in _want_types if t and t != 'dense']
+    if _extra:
+        _b = getattr(model_obj, 'backend', None) or 'this backend'
+        raise HTTPException(
+            status_code=400,
+            detail=(f"Model '{model_name}' is being served by {_b}, which produces "
+                    f"DENSE vectors only — {', '.join(sorted(set(_extra)))} is not "
+                    f"available on this path. bge-m3's sparse/colbert come from "
+                    f"extra linear heads over the last hidden state, which needs "
+                    f"the native transformers backend. Serve the model on the "
+                    f"coderai-embeddings image (full core: torch + transformers), "
+                    f"where the multi-vector backend is selected automatically "
+                    f"from the repo's sparse_linear.pt. A coderai-vllm pod cannot "
+                    f"do it: no torch in its main venv, by design."))
+
     # Text vectors first, then image vectors — indices follow that order.
     vectors: List[List[float]] = []
     # How many windows each input needed. Reported when any was more than one:

packaging/runpod/profiles/embeddings.txt

@@ -11,3 +11,27 @@ geographiclib>=2.0
 # recognition works without one.
 pytorch_lightning>=2.0.0
 pytorch_metric_learning>=2.0.0
+
+# ---------------------------------------------------------------------------
+# bge-m3 MULTI-VECTOR (dense + sparse + colbert) runs HERE, and only here.
+#
+# Hybrid retrieval wants dense AND sparse from one pass. Those extra vectors
+# come from linear heads (sparse_linear.pt / colbert_linear.pt) applied to the
+# last hidden state -- a transformers forward pass, not an embedding server's
+# output.
+# So:
+#
+#   * THIS image serves them. It is FULL core, so torch and transformers are in
+#     the main venv, and codai/api/embeddings._is_bge_m3() picks the native
+#     multi-vector backend automatically from the presence of sparse_linear.pt
+#     in the repo. Nothing to configure.
+#
+#   * `coderai-vllm` CANNOT, and not for want of a setting: it is a LIGHT-core
+#     image with no torch in its main venv by design, so _load_bge_m3 cannot
+#     even import. vLLM serves embeddings dense only. Asking that pod for
+#     sparse now returns a 400 naming this, rather than a normal-looking
+#     dense-only response that quietly halves a hybrid search.
+#
+# transformers is already pinned above for exactly this (and the heads are plain
+# Linear layers, so FlagEmbedding -- which pins older transformers/peft -- is
+# deliberately NOT a dependency).

packaging/runpod/profiles/ocr.venv-surya.txt

@@ -14,8 +14,8 @@
 # ~250 MB for a CPU build. The `ocr` profile's own card runs docTR and (when
 # baked) surya's local line detector; a CUDA resolve here is ~5 GB of image for
 # a venv that, on a GPU-less orchestrator, can never execute any of it --
-# `torch.cuda.is_available()` is False on the Digesta orchestrator today while
-# its surya venv carries cu130.
+# `torch.cuda.is_available()` is False on a GPU-less orchestrator while its
+# surya venv carries a cu130 build -- measured, and not a hypothetical.
 #
 # Set an explicit CUDA pin here if you WANT surya's line detection on the card.
 --extra-index-url https://download.pytorch.org/whl/cpu

packaging/runpod/profiles/vllm-ocr.venv-vllm.txt

@@ -6,8 +6,8 @@
 # U+FFFD per accented letter. The pin was the wrong conclusion from a real
 # measurement: surya-ocr-2's tokenizer is BYTE-level, so a U+FFFD per accent
 # does not show the model chose it — it shows bytes being split and decoded
-# separately somewhere downstream. The Digesta fleet has since been serving
-# surya-ocr-2 on 0.27.1 (`coderai-vllm`) and reading accented Italian.
+# separately somewhere downstream. A production fleet has since been serving
+# surya-ocr-2 on 0.27.1 (`coderai-vllm`) and reading accented text correctly.
 #
 # So the version difference is gone and the reason for this image changed: it is
 # no longer "a different vLLM" but "the OCR PIPELINE on the pod" — the surya
@@ -15,7 +15,7 @@
 # rasterised and read on the card instead of crossing the network as a bitmap.
 # Keep this list identical to profiles/vllm.venv-vllm.txt: two images serving
 # the same weights on two vLLMs is exactly the invisible difference that cost
-# the Digesta dev three rounds of blaming the pod image.
+# three rounds of blaming a pod image for an orchestrator bug.
 --extra-index-url https://wheels.vllm.ai/0.27.1/cu129
 --extra-index-url https://download.pytorch.org/whl/cu129
 vllm==0.27.1+cu129

tests/test_embed_long_input.py

@@ -251,3 +251,109 @@ def test_the_clip_family_is_left_truncating_on_purpose():
     blk = src.split("def _embed_texts(")[1].split("def _embed_images(")[0]
     clip = blk.split("elif backend == 'clip':")[1].split("elif backend ==")[0]
     assert "_embed_long(" not in clip
+
+
+# ------------------------------------------- multi-vector, merged per type #
+#
+# A long input on the bge-m3 multi-vector path needs all three outputs merged,
+# and none of the three is a free choice: each is what the encoder ALREADY does
+# within one window, extended across windows. Getting these wrong degrades
+# retrieval in a way that reads as a model problem, so they are pinned here.
+def test_one_window_passes_through_untouched():
+    part = {"dense": [1.0, 0.0], "sparse": {5: 0.5}, "colbert": [[1.0, 0.0]]}
+    assert em._bge_m3_merge([part], ("dense", "sparse", "colbert")) is part
+
+
+def test_dense_is_mean_pooled_and_renormalized():
+    """Dense is an L2-normalized CLS vector; a mean of unit vectors is not one,
+    and cosine similarity on a non-unit vector is quietly wrong."""
+    out = em._bge_m3_merge([{"dense": [1.0, 0.0]}, {"dense": [0.0, 1.0]}],
+                           ("dense",))
+    v = out["dense"]
+    assert math.isclose(math.sqrt(sum(x * x for x in v)), 1.0, rel_tol=1e-9)
+    assert math.isclose(v[0], v[1], rel_tol=1e-9)
+
+
+def test_sparse_takes_the_MAX_per_token_id():
+    """The encoder max-pools per token id inside a window already
+    (scatter_reduce amax), and max is what retrieval wants: a term's weight is
+    its strongest occurrence, not an average diluted by the pages that never
+    mention it."""
+    out = em._bge_m3_merge([{"sparse": {5: 0.2, 9: 0.9}},
+                            {"sparse": {5: 0.7, 11: 0.1}}], ("sparse",))
+    assert out["sparse"] == {5: 0.7, 9: 0.9, 11: 0.1}
+
+
+def test_sparse_averaging_would_be_wrong():
+    """Guard against someone 'tidying' max into mean: a term appearing strongly
+    on one page of forty must not be diluted to a fortieth of its weight."""
+    parts = [{"sparse": {7: 1.0}}] + [{"sparse": {}} for _ in range(39)]
+    out = em._bge_m3_merge(parts, ("sparse",))
+    assert out["sparse"][7] == 1.0
+
+
+def test_colbert_token_vectors_are_concatenated():
+    """Colbert scores max-sim of each query token against ALL document token
+    vectors, so every content token of every window belongs in the list."""
+    out = em._bge_m3_merge([{"colbert": [[1.0], [2.0]]},
+                            {"colbert": [[3.0]]}], ("colbert",))
+    assert out["colbert"] == [[1.0], [2.0], [3.0]]
+
+
+def test_only_the_requested_types_are_merged():
+    out = em._bge_m3_merge([{"dense": [1.0, 0.0], "sparse": {1: 1.0}},
+                            {"dense": [0.0, 1.0], "sparse": {2: 1.0}}],
+                           ("dense",))
+    assert "dense" in out and "sparse" not in out
+
+
+def test_first_pooling_keeps_the_opening_window_only():
+    """For an operator who would rather have the old truncating behaviour, said
+    explicitly rather than by accident."""
+    a = {"dense": [1.0, 0.0], "sparse": {1: 1.0}}
+    b = {"dense": [0.0, 1.0], "sparse": {2: 1.0}}
+    assert em._bge_m3_merge([a, b], ("dense", "sparse"), "first") is a
+
+
+def test_merging_nothing_is_empty_not_a_crash():
+    assert em._bge_m3_merge([], ("dense",)) == {}
+
+
+def test_a_missing_type_in_a_window_does_not_break_the_merge():
+    out = em._bge_m3_merge([{"dense": [1.0, 0.0]}, {}], ("dense", "sparse"))
+    assert out["dense"]
+    assert out["sparse"] == {}
+
+
+def test_the_multi_vector_path_windows_long_inputs():
+    src = (ROOT / "codai/api/embeddings.py").read_text()
+    # Split on the next TOP-LEVEL def (column 0): the body contains a nested
+    # `def _run()`, and splitting on "def " cut the slice before the windowing.
+    fn = src.split("async def _bge_m3_response")[1].split("\ndef ")[0]
+    assert "_token_windows(" in fn
+    assert "_bge_m3_merge(" in fn
+
+
+# ------------------------------------- dense-only backends say so out loud #
+def test_sparse_on_a_dense_only_backend_is_refused_with_the_reason():
+    """It used to be ignored: a caller asking for sparse got a normal-looking
+    dense response, and its hybrid retrieval then ran on half the signal and
+    merely scored worse -- which reads as a model problem."""
+    src = (ROOT / "codai/api/embeddings.py").read_text()
+    assert "DENSE vectors only" in src
+    assert "coderai-embeddings" in src, "the message must name where it DOES work"
+    assert "no torch in its main venv" in src, (
+        "the message must say why the vllm image cannot, not just that it cannot")
+
+
+def test_the_refusal_is_a_400_not_a_500():
+    src = (ROOT / "codai/api/embeddings.py").read_text()
+    blk = src.split("_extra = [t for t in _want_types")[1].split("# Text vectors first")[0]
+    assert "status_code=400" in blk
+
+
+def test_dense_only_requests_are_untouched():
+    """Asking for nothing, or for dense, must not start failing."""
+    src = (ROOT / "codai/api/embeddings.py").read_text()
+    blk = src.split("_want_types = [")[1].split("raise HTTPException")[0]
+    assert "t != 'dense'" in blk