ship the libraries a compiled helper needs, and look for it where it lives (v0.5.5)

The DINOv2 GGUF embedder answered 500 for about twelve hours -- 6,176 image
embedding requests -- for two independent reasons, and the capability can be
dropped again by an ordinary rebuild, so both the lookup and the packaging are
fixed here.

The resolver looked only under /opt/coderai/bin, a directory that does not
exist in the image, and reported its absence as a generic load failure. It now
tries DINOV2_EMBED_BIN, then PATH, then the places the binary is actually
installed, and when it finds nothing it names every one of them plus the build
script -- because there is no fallback: llama.cpp cannot load a ViT-only arch,
so this binary is the only runtime this model has.

The binary was also never listed in the image build, having been compiled by
hand once; the next rebuild silently shipped without it. It is listed now,
along with the private shared libraries it cannot start without.

Those libraries are the larger half of the problem. The image has no system
OpenCV -- only the venv's cv2 wheel, a different version with mangled sonames
-- so the closure travels beside the binary in dinov2-embed-libs, reached
through its own RPATH rather than LD_LIBRARY_PATH: OpenCV placed in
local-libs would be offered to every process in the image, cv2 included, while
a private directory on one binary's RPATH cannot shadow anything.

Four things make that work, each learned by watching it fail:

  - the whole dependency tail, not just libopencv_*. Debian's imgcodecs links
    GDAL, GDCM and OpenEXR, which a JPEG decoder never calls and the loader
    demands anyway. Copying three libraries left six still "not found".
  - cp -L. Most of these paths are symlinks, and copying the link gives a
    20-byte file -- which also made a first size estimate off by 7x.
  - RPATH on every library, not only on the binary. patchelf writes
    DT_RUNPATH, which unlike DT_RPATH is not inherited by a dependency's
    dependencies, so the binary found imgcodecs and imgcodecs found nothing.
  - everything above the base ABI, not only what the image lacks. libtiff.so.6
    exists there, so it was skipped, and the newer GDAL then wanted
    LIBTIFF_4.6.1 from it. Mixing distributions works only if the whole tier
    comes from one of them.

What stays behind is what any Linux has: libc, libm, libstdc++, libgcc,
libgomp, pthread, dl, rt, the loader, and libvulkan, which must match the GPU
driver.

The tarball build had the same two gaps as the image build and is fixed with
it: it copied a binary without its -libs sibling, shipping a program that
cannot start, and a --local-binary-dir sweep would have flattened those .so
files into bin/ as programs, away from the only place the RPATH looks.

Verified against the HF vision backend on the same images: cosine 0.990-0.995
per image, with image-to-image geometry agreeing to 0.002-0.017 -- the
quantized binary and the reference implementation describe the same space, so
an index built by one stays valid under the other.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66

.gitignore

@@ -25,6 +25,10 @@ debug.log
 # A HuggingFace cache a voice-conversion model writes beside the source when it
 # is given no cache dir (1.4 GB of weights, with its CACHEDIR.TAG).
 /checkpoints/
+# Locally compiled helper binaries and the private shared libraries that
+# travel with them (packaging/dinov2cpp/build.sh puts dinov2-embed here, which
+# live-code mode bind-mounts into the container). Build products, not source.
+/third_party/bin/
 
 # Third-party source clone of the GPTQ quantizer — installed into the venv from
 # source; the working tree is not part of this repo (it has its own .git).

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.4"
+__version__ = "0.5.5"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/embeddings.py

@@ -727,6 +727,56 @@ def _vpr_embed_images(model, pil_images, dimensions=None):
 _SALAD_IDS = {'salad', 'dinov2-salad'}
 
 
+#: Where a locally compiled `dinov2-embed` may live, in order. The image
+#: installs helper binaries it collected (whisper-server, ds4-server, …) into
+#: /usr/local/bin, so that is the first place to look -- NOT
+#: /opt/coderai/bin, which was the only place this used to look and which does
+#: not exist in the image at all. Consequence, measured on 2026-10-26: 6,176
+#: image-embedding requests answered 500 over about twelve hours, with the
+#: binary absent from the image and the error naming a directory nobody
+#: installs to.
+_DINOV2_EMBED_CANDIDATES = (
+    "/usr/local/bin/dinov2-embed",
+    "/opt/coderai/bin/dinov2-embed",
+    # The working tree, for a developer who built it where the source is. This
+    # path is bind-mounted into the container in live-code mode, so a build
+    # landing here is reachable without rebuilding the image.
+    "/opt/coderai/app/third_party/bin/dinov2-embed",
+)
+
+
+def _dinov2_embed_bin() -> str:
+    """The `dinov2-embed` binary, or a RuntimeError naming every place looked.
+
+    DINOV2_EMBED_BIN still wins, so an operator can point at a build anywhere;
+    without it, PATH and the install locations are searched. The error has to
+    list what was tried, because the one-path version sent whoever read it to
+    build a binary into a directory that would not have been used anyway.
+    """
+    import os as _os
+    import shutil as _shutil
+    explicit = (_os.environ.get("DINOV2_EMBED_BIN") or "").strip()
+    if explicit:
+        if _os.path.isfile(explicit):
+            return explicit
+        raise RuntimeError(
+            f"dinov2-embed binary not found at {explicit!r} (from "
+            f"DINOV2_EMBED_BIN) — build it with packaging/dinov2cpp/build.sh")
+    on_path = _shutil.which("dinov2-embed")
+    if on_path:
+        return on_path
+    for cand in _DINOV2_EMBED_CANDIDATES:
+        if _os.path.isfile(cand):
+            return cand
+    raise RuntimeError(
+        "dinov2-embed binary not found — looked on PATH and at "
+        + ", ".join(_DINOV2_EMBED_CANDIDATES)
+        + ". Build it with packaging/dinov2cpp/build.sh and install it to "
+          "/usr/local/bin (where the image collects helper binaries), or point "
+          "DINOV2_EMBED_BIN at it. This model is image-only and has no "
+          "fallback runtime: without the binary every request for it fails.")
+
+
 def _is_salad(model_name) -> bool:
     return str(model_name).strip().lower().replace('_', '-') in _SALAD_IDS
 
@@ -827,11 +877,7 @@ def _load_embedding_model(model_name: str, device: str, model_config: dict = Non
             and 'dinov2' in _os_basename(model_name).lower()):
         import os as _os
         import subprocess
-        _bin = _os.environ.get('DINOV2_EMBED_BIN', '/opt/coderai/bin/dinov2-embed')
-        if not _os.path.isfile(_bin):
-            raise RuntimeError(
-                f"dinov2-embed binary not found at {_bin} — build it with "
-                "packaging/dinov2cpp/build.sh")
+        _bin = _dinov2_embed_bin()
         cfg = model_config or {}
         raw = cfg.get('_raw_cfg') if isinstance(cfg.get('_raw_cfg'), dict) else {}
         env = dict(_os.environ)

packaging/dinov2cpp/build.sh

@@ -72,4 +72,71 @@ cmake -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON \
       -DBUILD_SHARED_LIBS=OFF \
       -DBUILD_REALTIME=OFF -DBUILD_QUANTIZE=ON ..
 make -j"$(nproc)" embed quantize
+
+# Also leave it under the name the server looks for. The binary cmake produces
+# is `embed`, but the image build stages a helper binary under its BASENAME and
+# the embeddings backend resolves `dinov2-embed` -- so a copy built as `embed`
+# installed as /usr/local/bin/embed and was never found. Both names, so neither
+# convention breaks.
+cp -f bin/embed bin/dinov2-embed
+
+# ── private shared libraries ────────────────────────────────────────────────
+# OpenCV is a build dep AND a runtime one, and the coderai image has no system
+# OpenCV: only the venv's cv2 wheel, at a different version and with mangled
+# sonames. So the libraries this binary needs and the image lacks are shipped
+# BESIDE it, in `dinov2-embed-libs`, reached through the binary's own RPATH.
+#
+# Not /opt/coderai/local-libs, which is on the global LD_LIBRARY_PATH: OpenCV
+# 4.10 placed there would be offered to every process in the image, cv2
+# included. A private directory on one binary's RPATH cannot shadow anything.
+#
+# RPATH lists the sibling directory AND $ORIGIN, because the image stages
+# helper binaries flat into /usr/local/bin while a developer keeps the
+# directory next to the build -- both layouts then resolve.
+if command -v patchelf >/dev/null 2>&1; then
+    LIBDIR="$(pwd)/bin/dinov2-embed-libs"
+    mkdir -p "$LIBDIR"
+    # OpenCV is not the end of it: Debian's libopencv_imgcodecs links GDAL,
+    # GDCM and OpenEXR, none of which this binary uses -- it decodes JPEG and
+    # PNG -- but all of which the loader demands at startup. Copying only
+    # libopencv_* therefore moved the error instead of removing it (measured:
+    # three libraries copied, six more still "not found").
+    #
+    # So the whole dependency tail goes in, MINUS what any Linux has anyway:
+    # libc, libm, libstdc++, libgcc, libgomp, pthread, dl and the Vulkan
+    # loader, which must come from the host/image or the binary would be
+    # running two C libraries at once.
+    #
+    # cp -L, because most of these paths are symlinks; copying the link gives a
+    # 20-byte file and a mystery at load time.
+    BASE_SYSTEM='^(linux-vdso|ld-linux|libc|libm|libdl|libpthread|librt|libstdc\+\+|libgcc_s|libgomp|libvulkan)\.'
+    ldd bin/dinov2-embed \
+      | sed -n 's/^[[:space:]]*\([^ ]*\) => \([^ ]*\).*/\1 \2/p' \
+      | grep -Ev "$BASE_SYSTEM" \
+      | while read -r soname path; do
+            [ -e "$path" ] || continue
+            cp -Lf "$path" "$LIBDIR/$soname"
+        done
+    patchelf --set-rpath '$ORIGIN/dinov2-embed-libs:$ORIGIN' bin/dinov2-embed
+    # And every copied library needs its OWN RUNPATH. patchelf writes
+    # DT_RUNPATH, and unlike the old DT_RPATH that is NOT inherited by a
+    # dependency's dependencies: with it set only on the binary, the binary
+    # found libopencv_imgcodecs and libopencv_imgcodecs then failed to find
+    # libjpeg, GDAL and GDCM. $ORIGIN on each .so is what manylinux wheels do
+    # for exactly this.
+    for so in "$LIBDIR"/*.so*; do
+        [ -e "$so" ] || continue
+        patchelf --set-rpath '$ORIGIN' "$so" 2>/dev/null \
+            || echo "  WARNING: could not set RUNPATH on $(basename "$so")"
+    done
+    echo "PRIVATE LIBS: $LIBDIR ($(find "$LIBDIR" -type f | wc -l) file(s), $(du -sh "$LIBDIR" | cut -f1))"
+else
+    echo "WARNING: patchelf not installed -- no private library directory was"
+    echo "         made. The binary will need its OpenCV libraries resolvable"
+    echo "         wherever it runs."
+fi
+
 echo "BUILT: $(pwd)/bin/embed"
+echo "BUILT: $(pwd)/bin/dinov2-embed   (install this one as "
+echo "       /usr/local/bin/dinov2-embed, where the image collects helpers;"
+echo "       keep dinov2-embed-libs/ beside it -- the image build copies it)"

packaging/linux/build_oci_image.sh

@@ -324,6 +324,14 @@ discover_local_binaries() {
     "${CODERAI_K3_DIR:-$HOME/.coderai/kimi-k3-in-c}/bin/k3"
     "/usr/local/bin/rife-ncnn-vulkan"
     "$HOME/.local/bin/rife-ncnn-vulkan"
+    # dinov2.cpp's embed server (packaging/dinov2cpp/build.sh). The DINOv2 GGUF
+    # embedder has no other runtime -- llama.cpp cannot load a ViT-only arch --
+    # so an image built without this binary serves that model as a 500 and
+    # nothing else notices. It was missing for exactly that reason: built by
+    # hand, never listed here, dropped by the next rebuild.
+    "/usr/local/bin/dinov2-embed"
+    "$HOME/.local/bin/dinov2-embed"
+    "${CODERAI_DINOV2_DIR:-/tmp/dinov2cpp-build}/dinov2.cpp/build/bin/dinov2-embed"
   )
   local path
   for path in "${candidates[@]}"; do
@@ -369,6 +377,22 @@ prepare_venv_bundle() {
     cp -aL --remove-destination "$bin_path" "$dest_path"
   done
 
+  # A helper binary may carry PRIVATE shared libraries in a sibling
+  # "<name>-libs" directory, reached through its own RPATH ($ORIGIN/<name>-libs)
+  # rather than through LD_LIBRARY_PATH. dinov2-embed needs OpenCV, which this
+  # image does not have as system libraries -- only the venv's cv2 wheel, at a
+  # different version and with mangled sonames -- and putting OpenCV in
+  # local-libs would offer it to every process in the image, cv2 included.
+  # A private directory on one binary's RPATH cannot shadow anything.
+  for bin_path in "${LOCAL_BINARIES[@]}"; do
+    local priv_libs="${bin_path}-libs"
+    [[ -d "$priv_libs" ]] || continue
+    mkdir -p "$bundle/local-bin/$(basename "$priv_libs")"
+    rsync -a "$priv_libs/" "$bundle/local-bin/$(basename "$priv_libs")/"
+    printf 'Included private libraries for %s: %s file(s)\n' \
+      "$(basename "$bin_path")" "$(find "$priv_libs" -type f | wc -l)"
+  done
+
   if [[ ${#LOCAL_BINARIES[@]} -gt 0 ]]; then
     printf 'Included local compiled binaries:\n'
     printf '  %s\n' "${LOCAL_BINARIES[@]}"

packaging/linux/make_tarball_from_venv.sh

@@ -101,6 +101,12 @@ add_local_binary() {
   local path="$1"
   local abs_path
   [[ -x "$path" && -f "$path" ]] || return 0
+  # A "<name>-libs" directory beside a binary holds that binary's PRIVATE
+  # shared libraries, reached through its own RPATH. Those .so files are
+  # executable, so a --local-binary-dir sweep would otherwise flatten them
+  # into bin/ as if they were programs -- and away from the one place the
+  # RPATH looks for them.
+  [[ "$(dirname "$path")" == *-libs ]] && return 0
   abs_path="$(cd "$(dirname "$path")" && pwd)/$(basename "$path")"
   for existing in "${LOCAL_BINARIES[@]}"; do
     [[ "$existing" == "$abs_path" ]] && return 0
@@ -120,6 +126,16 @@ done
 for b in "${LOCAL_BINARIES[@]}"; do
   [[ -x "$b" ]] || { echo "Error: local binary is not executable: $b" >&2; exit 2; }
   cp -a "$b" "$STAGE/bin/"
+  # ...and its private library directory, if it has one. dinov2-embed needs
+  # OpenCV, which no coderai artefact carries as system libraries (only the
+  # venv's cv2 wheel, at a different version and with mangled sonames), so the
+  # libraries travel beside the binary on its RPATH. Shipping the binary
+  # without them produces a program that cannot start -- which is exactly how
+  # the DINOv2 embedder came to answer 500 for twelve hours.
+  if [[ -d "${b}-libs" ]]; then
+    rsync -a "${b}-libs/" "$STAGE/bin/$(basename "${b}")-libs/"
+    echo "Included private libraries for $(basename "$b"): $(find "${b}-libs" -type f | wc -l) file(s)"
+  fi
 done
 
 if [[ "$INCLUDE_LOCAL_LIBS" == "1" ]]; then

tests/test_dinov2_embed_binary_lookup.py

+"""The DINOv2 GGUF embedder's binary has to be FOUND where it is installed.
+
+Measured on a running install, 2026-10-26: 6,176 image-embedding requests
+answered 500 over about twelve hours, every one of them
+``dinov2-embed binary not found at /opt/coderai/bin/dinov2-embed``. Two
+mistakes compounded:
+
+* the image collects locally compiled helper binaries into ``/usr/local/bin``
+  (whisper-server, ds4-server, colibri, rife), and ``/opt/coderai/bin`` does
+  not exist in it at all -- so the one path it looked at could never hold the
+  binary, and the error told whoever read it to install it there;
+* ``dinov2-embed`` was not in the image build's discovery list, so it was
+  dropped by a rebuild and nothing failed until a client asked.
+
+This model is image-only and llama.cpp cannot load a ViT-only arch, so there
+is no fallback runtime: a missing binary is the whole capability gone.
+"""
+
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+import codai.api.embeddings as emb
+
+
+@pytest.fixture(autouse=True)
+def _no_inherited_env(monkeypatch):
+    monkeypatch.delenv("DINOV2_EMBED_BIN", raising=False)
+
+
+def test_the_install_location_is_looked_at_first(monkeypatch, tmp_path):
+    """/usr/local/bin is where the image puts it; it must be tried."""
+    assert emb._DINOV2_EMBED_CANDIDATES[0] == "/usr/local/bin/dinov2-embed"
+    import shutil
+    monkeypatch.setattr(shutil, "which", lambda name: None)
+    found = {"/usr/local/bin/dinov2-embed"}
+    monkeypatch.setattr(emb.os.path, "isfile", lambda p: p in found)
+    assert emb._dinov2_embed_bin() == "/usr/local/bin/dinov2-embed"
+
+
+def test_path_wins_over_the_hardcoded_list(monkeypatch):
+    import shutil
+    monkeypatch.setattr(shutil, "which",
+                        lambda name: "/somewhere/else/dinov2-embed"
+                        if name == "dinov2-embed" else None)
+    assert emb._dinov2_embed_bin() == "/somewhere/else/dinov2-embed"
+
+
+def test_an_explicit_override_is_obeyed(monkeypatch, tmp_path):
+    binary = tmp_path / "dinov2-embed"
+    binary.write_text("#!/bin/sh\n")
+    monkeypatch.setenv("DINOV2_EMBED_BIN", str(binary))
+    assert emb._dinov2_embed_bin() == str(binary)
+
+
+def test_an_explicit_override_that_is_missing_says_so(monkeypatch):
+    monkeypatch.setenv("DINOV2_EMBED_BIN", "/nope/dinov2-embed")
+    with pytest.raises(RuntimeError) as exc:
+        emb._dinov2_embed_bin()
+    assert "DINOV2_EMBED_BIN" in str(exc.value)
+
+
+def test_the_error_names_every_place_it_looked(monkeypatch):
+    """The old message named one directory, and the wrong one."""
+    import shutil
+    monkeypatch.setattr(shutil, "which", lambda name: None)
+    monkeypatch.setattr(emb.os.path, "isfile", lambda p: False)
+    with pytest.raises(RuntimeError) as exc:
+        emb._dinov2_embed_bin()
+    msg = str(exc.value)
+    for cand in emb._DINOV2_EMBED_CANDIDATES:
+        assert cand in msg
+    assert "PATH" in msg
+    assert "packaging/dinov2cpp/build.sh" in msg
+    # And that it is not a degraded mode: there is no other runtime.
+    assert "no fallback" in msg
+
+
+def test_the_image_build_collects_the_binary():
+    """Otherwise the next rebuild drops the capability again, silently."""
+    script = (Path(__file__).resolve().parents[1]
+              / "packaging/linux/build_oci_image.sh").read_text()
+    assert "/usr/local/bin/dinov2-embed" in script, (
+        "dinov2-embed is not in discover_local_binaries, so an image built "
+        "from this script serves the DINOv2 GGUF model as a 500")
+
+
+def test_the_native_build_produces_that_name():
+    build = (Path(__file__).resolve().parents[1]
+             / "packaging/dinov2cpp/build.sh").read_text()
+    assert "bin/dinov2-embed" in build
+
+
+# ===========================================================================
+#  the private library directory
+# ===========================================================================
+#  The binary links OpenCV, and the image has NO system OpenCV -- only the
+#  venv's cv2 wheel, a different version with mangled sonames. So the
+#  libraries it needs travel beside it, on its own RPATH. Not in
+#  /opt/coderai/local-libs: that is on the global LD_LIBRARY_PATH, where
+#  OpenCV 4.10 would be offered to every process in the image, cv2 included.
+
+def _build_script() -> str:
+    return (Path(__file__).resolve().parents[1]
+            / "packaging/dinov2cpp/build.sh").read_text()
+
+
+def _image_script() -> str:
+    return (Path(__file__).resolve().parents[1]
+            / "packaging/linux/build_oci_image.sh").read_text()
+
+
+def test_the_build_makes_a_private_library_directory():
+    src = _build_script()
+    assert "dinov2-embed-libs" in src
+    assert "patchelf --set-rpath" in src
+
+
+def test_the_rpath_covers_both_layouts():
+    """A sibling directory for a developer, flat for the image's /usr/local/bin."""
+    src = _build_script()
+    assert "'$ORIGIN/dinov2-embed-libs:$ORIGIN'" in src
+
+
+def test_the_whole_dependency_tail_is_copied_not_just_opencv():
+    """Copying only libopencv_* moved the error instead of removing it.
+
+    Debian's libopencv_imgcodecs links GDAL, GDCM and OpenEXR -- unused by a
+    JPEG/PNG decoder, demanded by the loader all the same. Measured: three
+    libraries copied, six more still "not found".
+    """
+    src = _build_script()
+    assert "BASE_SYSTEM=" in src, "the tail must be copied minus the base system"
+    # Not an opencv-only filter any more.
+    assert "s/^[[:space:]]*\\(libopencv[^ ]*\\) =>" not in src
+
+
+def test_the_base_system_is_left_to_the_image():
+    """libc/libstdc++/libgomp/Vulkan must come from the image: shipping a
+    second C library next to the first is how a binary stops starting."""
+    src = _build_script()
+    base = src.split("BASE_SYSTEM=")[1].split("\n")[0]
+    for lib in ("libc", "libstdc", "libgcc_s", "libgomp", "libpthread",
+                "libvulkan", "ld-linux"):
+        assert lib in base, f"{lib} should be excluded from the private copy"
+
+
+def test_symlinks_are_dereferenced():
+    """Most of these paths are symlinks; copying the link gives a 20-byte file
+    and a mystery at load time -- which is what made the first measurement of
+    this set read 20 MB when it was not."""
+    src = _build_script()
+    assert 'cp -Lf "$path"' in src
+
+
+def test_the_image_build_carries_the_private_directory():
+    src = _image_script()
+    assert '${bin_path}-libs' in src
+    assert "local-bin/$(basename \"$priv_libs\")" in src
+
+
+def test_the_private_libraries_do_not_go_into_local_libs():
+    """local-libs is global; this must stay private to the one binary."""
+    src = _build_script()
+    assert "local-libs" in src, "the reason should be written down"
+    assert "cp" not in src.split("local-libs")[1].split("\n")[0]
+
+
+def test_every_private_library_gets_its_own_runpath():
+    """DT_RUNPATH is not inherited by a dependency's dependencies.
+
+    With it set only on the binary, the binary found libopencv_imgcodecs and
+    libopencv_imgcodecs then failed to find libjpeg, GDAL and GDCM -- six
+    libraries still "not found" with all 98 of them sitting in the directory
+    next to it.
+    """
+    src = _build_script()
+    body = src.split("$LIBDIR")[0]  # ensure the loop is in the lib section
+    assert 'patchelf --set-rpath \'$ORIGIN\' "$so"' in src
+    assert "DT_RUNPATH" in src, "the reason belongs next to the loop"