runpod: an embedding model on a capability pod had no capability (v0.3.77)

Switching a model's `runpod.engine` from `coderai-vllm` to `coderai` -- the
correct setting for a capability pod -- silently removed its capability.
`_CODERAI_SERVER_ENGINES["coderai-vllm"]` had been supplying the classification;
plain `coderai` supplies none, and `model_capability()` returned "".

Both halves of the failure follow from that:
  * the front could not classify the request, so the model was loaded LOCALLY
    on a GPU-less orchestrator;
  * default_capability_image("") is "", so a blank image field raises
    "set `image` on the runpod block" at provisioning time.

It presented as "tries to load on the wrong machine", which is a long way from
"this entry has no capability".

The cause: model_capability() receives a BARE DICT, and _model_entry_for()
iterates md.values() -- it matches inside `embedding_models` and discards the
section key. Nothing had ever told it that an entry in `embedding_models` is an
embeddings model. Now it asks: by object identity first (the dict the loader
holds), falling back to the id so a copied or rebuilt entry still resolves, and
returning False on any failure because a capability GUESS is never worth an
exception in a placement path. An explicit `capability` field still wins, which
is what that field is for.

Only `embedding_models` is inferred, deliberately. The other sections work
today, and remapping them all to fix one would change placement for every model
kind -- a blast radius this does not need. A test asserts the inference does not
reach into text_models, image_models, video_models or tts_models.

Found by a user report ("broken -- front loads bge-m3 locally on the GPU-less
host; not in /v1/models") after I made exactly that config change. I had already
noted this gap when the capability came back empty during the switch, said it
was worth raising rather than working around, and then worked around it in the
config anyway -- which is how it reached production.

11 tests. Suite at the 25 pre-existing failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.76"
+__version__ = "0.3.77"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/runpod_worker.py

@@ -412,6 +412,34 @@ MODEL_TYPE_CAPABILITY = {
 _LLM_MODEL_TYPES = ("text_models", "gguf_models", "vision_models")
 
 
+def _in_embedding_section(entry: dict) -> bool:
+    """Whether this entry lives in models.json's `embedding_models`.
+
+    By IDENTITY first (the same dict object the loader holds), falling back to
+    the id -- so an entry copied or rebuilt still resolves. Returns False on any
+    failure: a capability guess is never worth an exception in a placement path.
+    """
+    try:
+        from codai.admin.routes import config_manager as _cm
+        md = getattr(_cm, "models_data", None) if _cm else None
+        lst = (md or {}).get("embedding_models")
+        if not isinstance(lst, list):
+            return False
+        for m in lst:
+            if m is entry:
+                return True
+        want = str(entry.get("id") or entry.get("path") or "").strip().lower()
+        if not want:
+            return False
+        for m in lst:
+            if isinstance(m, dict) and str(
+                    m.get("id") or m.get("path") or "").strip().lower() == want:
+                return True
+    except Exception:
+        pass
+    return False
+
+
 def model_capability(entry: dict, include_text: bool = False) -> str:
     """The capability a model entry belongs to.
 
@@ -429,6 +457,22 @@ def model_capability(entry: dict, include_text: bool = False) -> str:
     explicit = str(entry.get("capability") or "").strip().lower()
     if explicit:
         return explicit
+    # An entry in `embedding_models` IS an embeddings model. Nothing told
+    # model_capability that: it receives a bare dict, and _model_entry_for()
+    # iterates md.values() and discards the section key it matched in.
+    #
+    # Before this, the capability came from `runpod.engine` being one of the
+    # coderai-* server engines -- so an embedding model with `engine: coderai`
+    # (the correct setting for a capability pod) resolved to NO capability, the
+    # front could not classify it, and it was loaded LOCALLY on a GPU-less
+    # orchestrator. It failed as "tries to load on the wrong machine" rather
+    # than as "this needs a capability", which is a long way from the cause.
+    #
+    # Deliberately only this one section. The others work today, and remapping
+    # them all would change placement for every model kind to fix one -- a
+    # blast radius this does not need.
+    if _in_embedding_section(entry):
+        return "embeddings"
     # A LongCat-Video model is a video model, but the 'video' image cannot serve it: its
     # pipeline is not in diffusers and it needs its own Python 3.10 venv. Without this
     # it would be placed on an image that boots, passes its health check, accepts the

tests/test_embedding_capability.py

+"""An embedding model on a capability pod resolved to NO capability.
+
+Switching a model's `runpod.engine` from `coderai-vllm` to `coderai` -- the
+correct setting for a capability pod -- removed an IMPLICIT capability:
+`_CODERAI_SERVER_ENGINES["coderai-vllm"]` had been supplying the classification.
+With plain `coderai` and no `capability` field, `model_capability()` returned ""
+and:
+
+  * the front could not classify the request, so bge-m3 was loaded LOCALLY on a
+    GPU-less orchestrator;
+  * `default_capability_image("")` is "", so a blank image field would raise
+    "set `image` on the runpod block" at provisioning time.
+
+It failed as "tries to load on the wrong machine", which is a long way from
+"this entry has no capability". An entry in `embedding_models` IS an embeddings
+model; nothing had told model_capability that, because it receives a bare dict
+and `_model_entry_for()` discards the section key it matched in.
+"""
+
+import pathlib
+import sys
+
+import pytest
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from codai.api import runpod_worker as rw                      # noqa: E402
+
+
+class _CM:
+    def __init__(self, data):
+        self.models_data = data
+
+
+def _wire(monkeypatch, data):
+    """Point the capability lookup at a models_data we control."""
+    import types
+    mod = types.ModuleType("codai.admin.routes")
+    mod.config_manager = _CM(data)
+    monkeypatch.setitem(sys.modules, "codai.admin.routes", mod)
+
+
+def test_an_embedding_entry_gets_the_embeddings_capability(monkeypatch):
+    entry = {"id": "BAAI/bge-m3", "runpod": {"engine": "coderai"}}
+    _wire(monkeypatch, {"embedding_models": [entry]})
+    assert rw.model_capability(entry, include_text=True) == "embeddings"
+
+
+def test_it_resolves_by_identity(monkeypatch):
+    """The same dict object the loader holds, even with no id at all."""
+    entry = {"path": "/models/whatever", "runpod": {}}
+    _wire(monkeypatch, {"embedding_models": [entry]})
+    assert rw._in_embedding_section(entry) is True
+
+
+def test_it_resolves_a_copy_by_id(monkeypatch):
+    """An entry rebuilt or copied must still resolve, or the inference would
+    depend on object identity surviving a round trip."""
+    entry = {"id": "BAAI/bge-m3", "runpod": {}}
+    _wire(monkeypatch, {"embedding_models": [dict(entry)]})
+    assert rw._in_embedding_section(entry) is True
+
+
+def test_a_text_model_is_not_claimed(monkeypatch):
+    entry = {"id": "qwen38-awq", "runpod": {"engine": "coderai-vllm"}}
+    _wire(monkeypatch, {"text_models": [entry], "embedding_models": []})
+    assert rw._in_embedding_section(entry) is False
+
+
+def test_an_explicit_capability_still_wins(monkeypatch):
+    """A model that serves something its section does not imply must keep its
+    override -- that is what the field is for."""
+    entry = {"id": "x", "capability": "rerank"}
+    _wire(monkeypatch, {"embedding_models": [entry]})
+    assert rw.model_capability(entry, include_text=True) == "rerank"
+
+
+def test_the_capability_now_resolves_an_image(monkeypatch):
+    """The second half of the bug: with no capability, a blank image field
+    raises at provisioning time instead of defaulting."""
+    entry = {"id": "BAAI/bge-m3", "runpod": {"engine": "coderai"}}
+    _wire(monkeypatch, {"embedding_models": [entry]})
+    cap = rw.model_capability(entry, include_text=True)
+    assert rw.default_capability_image(cap).endswith("coderai-embeddings:latest")
+
+
+@pytest.mark.parametrize("data", [
+    {},
+    {"embedding_models": None},
+    {"embedding_models": [None, "junk", 3]},
+])
+def test_a_broken_models_data_is_false_not_an_exception(monkeypatch, data):
+    """A capability GUESS is never worth an exception in a placement path."""
+    _wire(monkeypatch, data)
+    assert rw._in_embedding_section({"id": "x"}) is False
+
+
+def test_no_config_manager_at_all_is_false(monkeypatch):
+    import types
+    mod = types.ModuleType("codai.admin.routes")
+    mod.config_manager = None
+    monkeypatch.setitem(sys.modules, "codai.admin.routes", mod)
+    assert rw._in_embedding_section({"id": "x"}) is False
+
+
+def test_only_the_embedding_section_is_inferred():
+    """Remapping every section to fix one would change placement for every
+    model kind. The others work today; this stays narrow on purpose."""
+    src = (ROOT / "codai/api/runpod_worker.py").read_text()
+    fn = src.split("def _in_embedding_section")[1].split("\ndef ")[0]
+    for other in ("text_models", "image_models", "video_models", "tts_models"):
+        assert other not in fn, f"the inference reaches into {other}"