an image-only embedder must refuse text, not hang on it

SALAD answered four sentences with nothing at all: no response in 180s, recorded
by the sweep as a 500 with no error logged anywhere, while dinov2 given the
identical request returned a clean 400 immediately.

The refusal for the other image-only models keys on the backend name
(`vision`, `dinov2cpp`). SALAD is `vpr-server`, and its contract — like
EigenPlaces' `vpr` — puts the image in `input` rather than `image`, so the
refusal cannot key on the field that was used, only on what is in it. `vpr`
decodes locally and so fails cleanly; `vpr-server` handed the strings to an
isolated subprocess that could not answer and did not reply. A hang is worse
than a refusal: it holds a request slot and tells the caller nothing.

Both backends now check their inputs look like image references — the same four
shapes _decode_image accepts, tested without decoding so a real image is not
decoded twice. A short base64-valid token is treated as text, because no picture
is eight bytes and the alternative is sending "hello" to the subprocess again.

The first version of this guard sat inside the embed path, after the load: it
stopped the hang but answered 400 in 42 seconds, because SALAD had to spawn its
subprocess first. The call site of the mirror check records that exact lesson
eleven lines away —

    # Refuse an impossible request BEFORE loading anything. The same refusal
    # used to come after the load: images sent to a text-only 8 GB embedder
    # spent forty seconds loading it and then answered 400.

— so repeating it in the other direction was avoidable. _takes_image_in_input
answers from the model NAME, reading the existing _VPR_IDS/_SALAD_IDS sets rather
than a second list that would drift from the ones the loaders dispatch on. The
post-load check stays as a backstop for a model reached by an alias those sets do
not know.

Measured after: text 400 in 0.13s, a real image still returns its 8448-dim
embedding.

Also repairs a pre-existing test whose `[:4000]` source window this change
pushed `_load_embedding_locked` past, so it failed with "substring not found" on
correct code. It now bounds `_run_embeddings` by structure rather than a byte
count — the third time a fixed source window has broken in this repo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.54"
+__version__ = "0.5.55"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/embeddings.py

@@ -1213,6 +1213,59 @@ def _supports_images(model_obj) -> bool:
             or 'image' in type(proc).__name__.lower())
 
 
+def _takes_image_in_input(model_name) -> bool:
+    """Whether this model is image-only AND expects the image in ``input``.
+
+    Name-only, so it can be answered before anything is loaded -- the mirror of
+    :func:`_may_embed_images`, and for the reason its call site records: a
+    refusal that waits for the model to load costs the caller the whole load.
+    Measured after the first version of this guard shipped: text to `salad`
+    stopped hanging but still took 42s to answer 400, because the check sat
+    after the load rather than before it.
+    """
+    n = str(model_name or '').strip().lower().replace('_', '-')
+    return n in _VPR_IDS or n in _SALAD_IDS
+
+
+def _looks_like_image_ref(src) -> bool:
+    """Whether this string could be an image reference at all.
+
+    The same four shapes :func:`_decode_image` accepts -- http(s) URL, data URI,
+    existing file path, or a bare base64 blob -- tested without decoding, so a
+    real image is not decoded twice just to be checked.
+
+    Exists because the image-only models whose contract puts the image in
+    `input` rather than `image` had no way to refuse text. `vpr` decodes locally
+    and so fails cleanly, but `vpr-server` (SALAD) hands the strings to an
+    isolated subprocess, which received four sentences, could not answer, and
+    never replied: measured on zeiss 2026-10-30, the request hung past 180s and
+    the sweep recorded it as a 500 with no error logged anywhere. A hang is worse
+    than a refusal -- it holds a slot and tells the caller nothing -- and dinov2,
+    whose refusal is keyed on its backend name, answered the identical request
+    with a clean 400.
+    """
+    import os as _os
+    import re as _re
+    if not isinstance(src, str) or not src.strip():
+        return False
+    t = src.strip()
+    if t.startswith(('http://', 'https://', 'data:')):
+        return True
+    try:
+        if _os.path.isfile(t):
+            return True
+    except (OSError, ValueError):
+        pass        # absurdly long strings raise rather than returning False
+    # A bare base64 blob. Short strings are text, not an encoded picture.
+    if len(t) < 64:
+        return False
+    try:
+        base64.b64decode(_re.sub(r'\s+', '', t), validate=True)
+        return True
+    except Exception:
+        return False
+
+
 def _decode_image(src: str):
     """Accept a data URI, http(s) URL, local file path or bare base64 blob."""
     import io
@@ -1972,12 +2025,24 @@ def _embed_texts(model_obj, texts: List[str], dimensions=None,
         # image model: the HomeHunter contract sends the image data URI in 'input'
         return _geoclip_embed_images(
             model, [_decode_image(t) for t in texts], dimensions)
-    elif backend == 'vpr':
-        # VPR is image-only; the contract sends the image data URI in 'input'.
+    elif backend in ('vpr', 'vpr-server'):
+        # Image-only, and their contract puts the image in `input` rather than
+        # `image` -- so the refusal cannot be keyed on the field that was used,
+        # only on what is in it. Checked BEFORE dispatch: vpr-server hands these
+        # strings to a subprocess that cannot answer and does not reply, so text
+        # arriving here used to hang the request instead of failing it.
+        _bad = [i for i, t in enumerate(texts) if not _looks_like_image_ref(t)]
+        if _bad:
+            raise ValueError(
+                f"this embedding model is image-only (no text tower) and takes "
+                f"the image in 'input' as a data URI, http(s) URL, file path or "
+                f"base64 blob -- input[{_bad[0]}] is none of those. Send an "
+                f"image, or use a text/multimodal embedding model."
+                + (f" ({len(_bad)} of {len(texts)} inputs are not images.)"
+                   if len(_bad) > 1 else ""))
+        if backend == 'vpr':
             return _vpr_embed_images(
                 model, [_decode_image(t) for t in texts], dimensions)
-    elif backend == 'vpr-server':
-        # SALAD subprocess is image-only; 'input' carries image data URIs.
         return _embed_images(model_obj, texts, dimensions)
     elif backend == 'llama':
         # llama.cpp embedding mode. With a pooling type baked into the GGUF the
@@ -2355,6 +2420,20 @@ async def _run_embeddings(request: EmbeddingsRequest, http_request: Request = No
     # used to come after the load: images sent to a text-only 8 GB embedder
     # spent forty seconds loading it and then answered 400. A caller cannot
     # tell that from a hang, and the work was wasted either way.
+    # The mirror case, refused here for the same reason: these models are
+    # image-only and take the image in `input`, so text in `input` can never be
+    # served -- and loading SALAD to find that out costs a subprocess spawn.
+    if texts and not images and _takes_image_in_input(model_name):
+        _bad = [i for i, t in enumerate(texts) if not _looks_like_image_ref(t)]
+        if _bad:
+            raise HTTPException(
+                status_code=400,
+                detail=f"Model '{model_name}' is image-only (no text tower) and "
+                       f"takes the image in 'input' as a data URI, http(s) URL, "
+                       f"file path or base64 blob — input[{_bad[0]}] is none of "
+                       f"those. Send an image, or use a text/multimodal "
+                       f"embedding model.")
+
     if images and model_obj is None and not _may_embed_images(model_name, _emb_cfg):
         raise HTTPException(
             status_code=400,

tests/test_embedding_routing_and_image_refusal.py

@@ -43,6 +43,7 @@ hang.
 
 import io
 import os
+import re
 import sys
 
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
@@ -59,8 +60,22 @@ _SRC = io.open(os.path.join(_ROOT, "codai/api/embeddings.py"), encoding="utf-8")
 # What the entry names is what gets loaded
 # --------------------------------------------------------------------------
 
+def _run_embeddings_body():
+    """The whole of `_run_embeddings`, not a fixed slice of it.
+
+    A `[:4000]` window broke when a pre-load refusal was added ahead of the
+    load: `_load_embedding_locked` moved past the cutoff and the test failed
+    with "substring not found" on code that was correct. The window was never
+    the thing being asserted — ordering within the function is.
+    """
+    after = _SRC.split("async def _run_embeddings", 1)[1]
+    # Up to the next top-level def, so the body is bounded by structure.
+    nxt = re.search(r"\n(?:async def |def )", after)
+    return after[:nxt.start()] if nxt else after
+
+
 def test_the_entry_path_decides_what_is_loaded():
-    body = _SRC.split("async def _run_embeddings", 1)[1][:4000]
+    body = _run_embeddings_body()
     assert "_cfg_path" in body, "the entry's path is never consulted"
     assert body.index("_cfg_path") < body.index("_load_embedding_locked"), (
         "the name is resolved after the load, which is too late to change it")

tests/test_image_only_embedders_refuse_text.py

+"""An image-only embedder must refuse text, not hang on it.
+
+`/v1/embeddings` has two image-only families whose contract puts the image in
+`input` rather than `image`:
+
+    vpr          EigenPlaces, decoded in-process
+    vpr-server   DINOv2-SALAD, handed to an isolated subprocess
+
+The refusal for the OTHER image-only models is keyed on the backend name:
+
+    elif backend in ('vision', 'dinov2cpp'):
+        raise ValueError("this embedding model is image-only (no text tower) …")
+
+so `dinov2` answers text with a clean 400. `vpr-server` had no such guard: it
+passed the strings straight to a subprocess that could not answer and did not
+reply. Measured on zeiss 2026-10-30, mid-sweep: four sentences sent to `salad`
+produced no response in 180s and the sweep recorded a 500 with no error logged
+anywhere, while `dinov2` given the identical request returned 400 immediately.
+
+A hang is worse than a refusal: it holds a request slot and tells the caller
+nothing.
+"""
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from codai.api.embeddings import _looks_like_image_ref
+
+SRC = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                   'codai', 'api', 'embeddings.py')
+
+
+# ── what counts as an image reference ────────────────────────────────────────
+
+@pytest.mark.parametrize('ref', [
+    'data:image/png;base64,iVBORw0KGgoAAAANSUhEUg==',
+    'data:image/jpeg;base64,/9j/4AAQ',
+    'http://example.com/a.png',
+    'https://example.com/a.jpg?x=1',
+])
+def test_the_four_accepted_shapes_pass(ref):
+    assert _looks_like_image_ref(ref) is True
+
+
+def test_a_real_file_path_passes(tmp_path):
+    p = tmp_path / 'x.png'
+    p.write_bytes(b'\x89PNG\r\n\x1a\n')
+    assert _looks_like_image_ref(str(p)) is True
+
+
+def test_a_long_base64_blob_passes():
+    assert _looks_like_image_ref('QUJD' * 40) is True
+
+
+@pytest.mark.parametrize('text', [
+    'a test sentence',
+    'what is in this image?',
+    'Qwen3-Embedding-4B',
+    '', '   ', None, 123, [],
+])
+def test_text_and_nonsense_are_refused(text):
+    assert _looks_like_image_ref(text) is False
+
+
+def test_a_short_base64_string_is_treated_as_text():
+    """'aGVsbG8=' decodes as base64 but is the word "hello". No picture is 8
+    bytes, and treating every short token as an image would send text to the
+    subprocess again."""
+    assert _looks_like_image_ref('aGVsbG8=') is False
+
+
+def test_a_path_that_does_not_exist_is_refused():
+    assert _looks_like_image_ref('/no/such/file.png') is False
+
+
+def test_an_absurdly_long_string_does_not_raise():
+    """os.path.isfile raises rather than returning False past the OS name limit,
+    and this runs on every input of every request."""
+    assert _looks_like_image_ref('x' * 100000) in (True, False)
+
+
+def test_it_never_decodes_to_check():
+    """A real image must not be decoded twice — once to validate and once to
+    embed. The check reads the prefix and the base64 alphabet only."""
+    src = open(SRC).read()
+    i = src.index('def _looks_like_image_ref')
+    body = src[i:src.index('def _decode_image', i)]
+    assert 'Image.open' not in body
+    assert 'convert(' not in body
+
+
+# ── the guard is wired, and placed before dispatch ───────────────────────────
+
+def test_both_image_only_input_backends_are_guarded():
+    src = open(SRC).read()
+    assert "elif backend in ('vpr', 'vpr-server'):" in src, (
+        'vpr-server is no longer guarded alongside vpr')
+
+
+def test_the_check_happens_before_the_subprocess_is_handed_anything():
+    """The whole point: vpr-server cannot refuse after dispatch, because the
+    subprocess simply never replies."""
+    src = open(SRC).read()
+    i = src.index("elif backend in ('vpr', 'vpr-server'):")
+    block = src[i:i + 1600]
+    assert block.index('_looks_like_image_ref') < block.index('_embed_images('), \
+        'text reaches the subprocess before being validated'
+    assert block.index('_looks_like_image_ref') < block.index('_vpr_embed_images('), \
+        'text reaches the in-process decoder before being validated'
+
+
+def test_the_refusal_says_where_the_image_goes():
+    """`input`, not `image` — the opposite of what the dinov2 message says, and
+    a caller told the wrong field just fails differently."""
+    src = open(SRC).read()
+    i = src.index("elif backend in ('vpr', 'vpr-server'):")
+    block = src[i:i + 1600]
+    assert "image-only (no text tower)" in block
+    assert "in 'input'" in block
+    assert 'data URI' in block and 'base64' in block
+
+
+def test_the_refusal_names_the_offending_input():
+    """A batch of 4 with one bad entry should say which one."""
+    src = open(SRC).read()
+    i = src.index("elif backend in ('vpr', 'vpr-server'):")
+    block = src[i:i + 1600]
+    assert 'input[{_bad[0]}]' in block or '_bad[0]' in block
+
+
+def test_the_other_image_only_backends_keep_their_refusal():
+    """dinov2's guard was already correct; this change must not disturb it."""
+    src = open(SRC).read()
+    assert "elif backend in ('vision', 'dinov2cpp'):" in src
+    assert "this embedding model is image-only (no text tower) — send 'image' " in src
+
+
+def test_vpr_backends_are_still_listed_as_image_capable():
+    """They must keep accepting images — the fix refuses text, not pictures."""
+    src = open(SRC).read()
+    i = src.index("if backend in ('clip', 'vision', 'qwenvl'")
+    line = src[i:i + 200]
+    assert "'vpr'" in line and "'vpr-server'" in line
+
+
+# ── refused BEFORE the model is loaded ───────────────────────────────────────
+#
+# The first version of this guard sat inside the embed path, after the load. It
+# stopped the hang but answered 400 in 42 seconds, because SALAD had to spawn
+# its subprocess first. The call site of the mirror check records that exact
+# lesson for images-to-a-text-model:
+#
+#   # Refuse an impossible request BEFORE loading anything. The same refusal
+#   # used to come after the load: images sent to a text-only 8 GB embedder
+#   # spent forty seconds loading it and then answered 400.
+#
+# so repeating it in the other direction was avoidable.
+
+def test_the_model_is_identified_by_name_alone():
+    """A name-only predicate is what makes a pre-load refusal possible."""
+    from codai.api.embeddings import _takes_image_in_input
+    assert _takes_image_in_input('salad') is True
+    assert _takes_image_in_input('dinov2-salad') is True
+    assert _takes_image_in_input('vpr') is True
+    assert _takes_image_in_input('eigenplaces') is True
+    assert _takes_image_in_input('Qwen3-Embedding-4B') is False
+    assert _takes_image_in_input('dinov2-large') is False
+    assert _takes_image_in_input(None) is False
+
+
+def test_the_name_check_is_case_and_separator_insensitive():
+    from codai.api.embeddings import _takes_image_in_input
+    for n in ('SALAD', 'Salad', 'dinov2_salad', '  salad  '):
+        assert _takes_image_in_input(n) is True, n
+
+
+def test_it_reads_the_existing_id_sets_rather_than_a_new_list():
+    """A second hand-kept list of salad/vpr aliases would drift from the ones
+    the loaders actually dispatch on."""
+    src = open(SRC).read()
+    i = src.index('def _takes_image_in_input')
+    body = src[i:src.index('def _looks_like_image_ref', i)]
+    assert '_VPR_IDS' in body and '_SALAD_IDS' in body
+
+
+#: The guard itself, not the function name. Searching for the bare name finds
+#: the `def` line first, 1200 lines from the call and with no condition near it
+#: -- the first version of these two tests asserted against the definition and
+#: failed on correct code.
+GUARD = 'if texts and not images and _takes_image_in_input(model_name):'
+
+
+def test_the_refusal_happens_before_any_load():
+    src = open(SRC).read()
+    pre = src.index(GUARD)
+    load = src.index('_load_embedding_locked(', pre)
+    assert pre < load, 'the refusal still waits for the model to load'
+
+
+def test_it_only_fires_when_there_is_no_image_at_all():
+    """A request carrying a real image must not be refused, and a mixed request
+    must reach the normal path."""
+    assert GUARD in open(SRC).read()
+
+
+def test_the_post_load_guard_is_still_there_as_a_backstop():
+    """The pre-load check is keyed on the model NAME; a model reached by an
+    alias the id sets do not know still has to be refused rather than hang."""
+    src = open(SRC).read()
+    assert "elif backend in ('vpr', 'vpr-server'):" in src
+    i = src.index("elif backend in ('vpr', 'vpr-server'):")
+    assert '_looks_like_image_ref' in src[i:i + 1600]