align_corners was not the cause either; fix the rebuild path regardless

I wrote into the docs that our llama.cpp predating b4aa7dd47 ("mtmd : use
align_corners for qwen3vl vision position embedding interpolation") was "the
actual cause" of Qwen3.8-27B describing every image identically. The argument was
a plausible commit plus date arithmetic — 0.3.30 vendors llama.cpp from
2026-06-16, the fix landed 2026-07-21, 0.3.36 vendors 2026-10-01 — and every
observation fitted it. I had not run it.

Tested now: 0.3.36 built from source with the full
-DGGML_CUDA=ON -DGGML_VULKAN=ON -DGGML_RPC=ON set and put into service. The same
two images still return byte-identical answers. The fourth theory joins the other
three, and the doc says so.

What the test buys is that the last candidate is eliminated by experiment rather
than argument. The remaining reading is that the qwen3vl graph does not fit this
model at all — the base is qwen3_5 while the projector declares qwen3vl_merger —
and no version bump addresses that. mmproj stays off the qwen38 cards; vision
goes to gemma-4-31B-heretic, which reads the same images correctly.

The rebuild was worth doing for everything else, and it needed three fixes to
complete at all:

  * build-rpc-server.sh asked for the `rpc-server` CMake target, renamed
    upstream to `ggml-rpc-server`. It now asks the generator which exists, so it
    spans both.
  * my first version of that detection used `grep -q` under `set -euo pipefail`:
    grep exits on first match, closing the pipe, cmake dies on SIGPIPE, and
    pipefail reports the pipeline as failed — a SUCCESSFUL match tested as a
    failure. Matched in pure bash now. It passes in a toy test because printf
    finishes before the pipe closes, so the race only appears with a slow
    producer.
  * Dockerfile.update-llama EXECUTED `rpc-server --help` at build time, which
    needs libcuda.so.1 — the driver, absent from any build by definition. The
    file's own comment eleven lines above states the rule ("the check is ldd,
    not import"); the wheel check obeyed it and this one did not. It had never
    been reachable before, so the first build to get that far could not pass.
    Now checks linkage and tolerates only libcuda being unresolved.

And Dockerfile.update now carries vosk and g++, because the incremental overlay
never touches the venv: an updated image shipped without vosk while /v1/models
advertised vosk-en and vosk-it, and without any C++ compiler at all, which is
what XTTS's Japanese phonemiser needs to build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.52"
+__version__ = "0.5.53"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

docs/serving-a-model-this-box-cannot-hold.md

@@ -172,37 +172,51 @@ is correct at two different model sizes, and Qwen3.5's OWN vision tower is
 correct when transformers runs it instead of llama.cpp — so it is not the model,
 not the family, and not our pipeline.
 
-## The actual cause: our llama.cpp predates the fix
-
-```
-our build (0.3.30)  vendors llama.cpp e3a74b299   2026-06-16
-the fix             b4aa7dd47  "mtmd : use align_corners for qwen3vl vision
-                    position embedding interpolation" (#25781)   2026-07-21
-0.3.36              vendors llama.cpp 0c1e57098   2026-10-01   <- contains it
-```
-
-Five weeks. Wrong position-embedding interpolation makes the vision features
-spatially meaningless, which is why the output is constant rather than merely
-wrong, and why `gemma4v` -- a different graph, untouched by that bug -- reads the
-same images correctly through the same pipeline.
-
-Everything observed fits: the projector pairs correctly, loads, and injects
-tokens the model notices (it says so when there is no image); two publishers'
-conversions fail identically because the defect is in the consumer, not the
-artifact; and the image-token floor is irrelevant, confirmed by testing with it
-at 1024 and at 0 and getting byte-identical answers.
-
-`packaging/linux/Dockerfile.update-llama` is bumped to 0.3.36 for this reason.
-It takes effect only on a rebuild, so until one runs, `mmproj` stays off the
-qwen38 cards and vision goes to gemma-4-31B-heretic.
-
-One method note, since it cost most of the investigation: three conclusions were
-drawn from reading headers and code, and all three were wrong. The floor was
-dismissed with "the symptom predates it" -- never checked, and it could not have
-been, because the floor is applied automatically whenever the projector type
-matches. Two projectors were compared while the one variable introduced by this
-repo was held constant in both. What settled it was varying one thing at a time
-against a known-good control.
+## A fourth wrong theory: the llama.cpp version
+
+I claimed the cause was found, and it was not. Recorded in full because the
+reasoning looked strong and was still worthless.
+
+The argument was: our 0.3.30 vendors llama.cpp e3a74b299 (2026-06-16); the
+commit b4aa7dd47 "mtmd : use align_corners for qwen3vl vision position embedding
+interpolation" (#25781) landed 2026-07-21; 0.3.36 vendors 0c1e57098 (2026-10-01)
+and so contains it. Wrong position-embedding interpolation would make vision
+features spatially meaningless, which fits a constant answer exactly, and
+`gemma4v` runs a different graph so it would be untouched. Every observation
+fitted.
+
+It was still only a plausible commit plus date arithmetic, and I wrote it into
+these docs as "the actual cause" without having run it.
+
+**Tested: llama-cpp-python 0.3.36 was built from source with the full
+-DGGML_CUDA=ON -DGGML_VULKAN=ON -DGGML_RPC=ON set and put into service. The same
+two images still return byte-identical answers.** So align_corners was not it,
+and the fourth theory joins the other three.
+
+What that does buy: the last standing candidate is eliminated BY TEST. The
+remaining explanation is that llama.cpp's qwen3vl graph does not fit this model
+at all -- the base is `qwen3_5` / `Qwen3_5ForConditionalGeneration` while the
+projector declares `qwen3vl_merger`, so a faithful implementation of the wrong
+graph on correctly-shaped tensors is still the best reading, and one that no
+version bump fixes.
+
+`mmproj` stays off the qwen38 cards; vision goes to gemma-4-31B-heretic.
+
+## The method, since it cost more than the finding
+
+Four theories, each argued from reading rather than running: bad pairing, missing
+deepstack, a bad artifact, an outdated llama.cpp. All four wrong.
+
+The floor was dismissed with "the symptom predates it" -- never checked, and it
+could not have been, because the floor is applied automatically whenever the
+projector type matches, so every observation had it on. Two publishers'
+projectors were then compared while the one variable this repo introduced was
+held constant in both.
+
+What actually produced knowledge: gemma as a known-good control through the same
+pipeline, varying one thing at a time, and accepting a negative result. Three of
+the four theories could have been killed in the time it took to write the
+paragraph asserting them.
 
 ## If you try a newer llama.cpp, do NOT pip install it
 

packaging/build-rpc-server.sh

@@ -57,9 +57,41 @@ CM=(-DGGML_RPC=ON -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_
 [ -n "$CUDA" ] && CM+=(-DGGML_CUDA=ON ${CMAKE_CUDA_ARCHITECTURES:+-DCMAKE_CUDA_ARCHITECTURES="$CMAKE_CUDA_ARCHITECTURES"})
 [ -n "$VULKAN" ] && CM+=(-DGGML_VULKAN=ON)
 cmake -S "$SRC" -B build "${CM[@]}" ${EXTRA_CMAKE_ARGS:-} >/dev/null
-cmake --build build --target rpc-server -j"$(nproc)"
-BIN=$(find build -type f -name rpc-server | head -1)
-[ -x "$BIN" ] || { echo "rpc-server did not build" >&2; exit 1; }
+
+# The CMake target was renamed: tools/rpc/CMakeLists.txt declares
+# `ggml-rpc-server` in current llama.cpp and `rpc-server` in older ones. Asking
+# for the wrong one fails the whole image build with
+#     gmake: *** No rule to make target 'rpc-server'.  Stop.
+# which names the target but not the rename -- measured on the 0.3.30 -> 0.3.36
+# bump, where the wheel itself built fine and only this step died. Ask the
+# generator which targets exist rather than guessing, so the script spans both.
+# Matched in pure bash, with the target list captured first. NOT
+# `cmake --target help | grep -q`: this script runs under `set -euo pipefail`,
+# and `grep -q` exits the moment it matches, which closes the pipe, kills cmake
+# with SIGPIPE, and pipefail then reports the whole pipeline as failed -- so a
+# SUCCESSFUL match tested as a failure and both candidates were skipped. The
+# diagnostic below used `grep -i rpc` without -q, read to EOF, and printed the
+# target it had just "not found", which is what gave it away.
+TARGETS="$(cmake --build build --target help 2>/dev/null || true)"
+RPC_TARGET=""
+for cand in ggml-rpc-server rpc-server; do
+    # make lists targets as "... name"; some generators omit the prefix.
+    if [[ $'\n'"$TARGETS"$'\n' == *$'\n'"... $cand"$'\n'* ]] \
+       || [[ $'\n'"$TARGETS"$'\n' == *$'\n'"$cand"$'\n'* ]]; then
+        RPC_TARGET="$cand"; break
+    fi
+done
+if [ -z "$RPC_TARGET" ]; then
+    echo "neither ggml-rpc-server nor rpc-server is a target in this llama.cpp;" >&2
+    echo "available targets containing 'rpc':" >&2
+    cmake --build build --target help 2>/dev/null | grep -i rpc >&2 || true
+    exit 1
+fi
+echo "rpc target: $RPC_TARGET"
+cmake --build build --target "$RPC_TARGET" -j"$(nproc)"
+BIN=$(find build -type f \( -name ggml-rpc-server -o -name rpc-server \) | head -1)
+[ -x "$BIN" ] || { echo "$RPC_TARGET did not build" >&2; exit 1; }
+# Installed under the name coderai invokes regardless of the upstream target.
 install -m 755 "$BIN" "$DEST/rpc-server"
-echo "installed $DEST/rpc-server"
+echo "installed $DEST/rpc-server (from $RPC_TARGET)"
 "$DEST/rpc-server" --help 2>&1 | head -3 || true

packaging/linux/Dockerfile.update

@@ -54,6 +54,24 @@ RUN /opt/coderai/python/bin/python3 -m pip install --no-input --no-cache-dir "nv
 # without it (discovery just stays off), hence `|| true`.
 RUN /opt/coderai/python/bin/python3 -m pip install --no-input --no-cache-dir "zeroconf>=0.130" || true
 
+# Vosk, the CPU STT backend. It is declared in requirements.txt, which the FULL
+# build picks up through the pre-built venv bundle -- but this overlay never
+# touches the venv, so an image updated incrementally shipped without it and
+# every vosk transcription answered
+#     Transcription error: No module named 'vosk'
+# while /v1/models advertised vosk-en and vosk-it as available. Installed here
+# for the same reason nvidia-ml-py and zeroconf are: the overlay has to carry
+# what the bundle would have.
+#
+# Install into the venv's own site-packages explicitly. Installing it the
+# ordinary way inside the running container put it in ~/.local instead, because
+# the venv is not writable by the container user, and the engine -- which does
+# not load the user site -- went on reporting the module as missing while pip
+# reported success.
+RUN /opt/coderai/python/bin/python3 -m pip install --no-input --no-cache-dir \
+        --target /opt/coderai/python/lib/python3.13/site-packages \
+        "vosk>=0.3.45" || true
+
 # openssh-client so `coderai-docker --upgrade --ssh-key …` can fetch over SSH
 # (git is present but the CUDA runtime base ships no ssh). Tiny; keep the overlay
 # still fast. `|| true` keeps offline overlay builds working (HTTPS upgrade still
@@ -62,7 +80,15 @@ RUN /opt/coderai/python/bin/python3 -m pip install --no-input --no-cache-dir "ze
 # model with real audio instead of a reachability check. Without it the probe
 # degrades silently to "configured, not verified" — caught by inspecting a built
 # image, not by any test, because the fallback is deliberately not an error.
-RUN apt-get update && apt-get install -y --no-install-recommends openssh-client espeak \
+# g++: a RUNTIME requirement, like the gcc the base carries for Triton. XTTS's
+# Japanese phonemiser (cutlet) pulls mojimoji, a C++ extension with no cp313
+# wheel, so it compiles when the isolated TTS venv is built on demand. The
+# bundled Python records clang++ as its C++ compiler and no clang is present, so
+# the build asked for clang++ and found nothing -- and the image had no C++
+# compiler at all: g++, c++ and clang were all absent and /usr/include/c++ did
+# not exist, which made gcc's presence look like it covered this. Without it
+# XTTS loses exactly one of its seventeen languages.
+RUN apt-get update && apt-get install -y --no-install-recommends openssh-client espeak g++ \
     && rm -rf /var/lib/apt/lists/* || true
 
 # Fonts for burned-in captions and overlays (codai/compose): Montserrat and

packaging/linux/Dockerfile.update-llama

@@ -88,6 +88,27 @@ assert "libggml-rpc" in names or "ggml_backend_rpc" in syms, \
     "RPC backend missing from the rebuilt llama-cpp-python"
 print("llama-cpp-python links cleanly and carries the RPC backend: OK")
 PY
-# Run it the way the launcher does (local-libs on the library path); fail the
-# build if it cannot start, not just print an error and carry on.
-RUN LD_LIBRARY_PATH=/opt/coderai/python/lib:/opt/coderai/local-libs /opt/coderai/python/bin/rpc-server --help >/dev/null
+# Check what it LINKS, not that it runs — the same rule the wheel check above
+# states and follows. Executing it here fails on a correctly built binary:
+#
+#   rpc-server: error while loading shared libraries: libcuda.so.1
+#
+# libcuda.so.1 is the NVIDIA driver, mounted by the container runtime and so
+# absent at build time by definition. The old `--help` invocation only became
+# reachable once the build stopped failing earlier (the rpc-server CMake target
+# was renamed to ggml-rpc-server upstream), and a CUDA-enabled rpc-server links
+# the driver directly — so the first build that got this far could not pass.
+#
+# Every other dependency must still resolve: a missing libggml or libgomp here
+# means a binary that will not start on a real host either.
+RUN set -eux; \
+    LD_LIBRARY_PATH=/opt/coderai/python/lib:/opt/coderai/local-libs \
+      ldd /opt/coderai/python/bin/rpc-server > /tmp/rpc-ldd.txt || true; \
+    cat /tmp/rpc-ldd.txt; \
+    if grep -v 'libcuda\.so' /tmp/rpc-ldd.txt | grep -q 'not found'; then \
+        echo "rpc-server has unresolved libraries other than the driver:" >&2; \
+        grep 'not found' /tmp/rpc-ldd.txt >&2; \
+        exit 1; \
+    fi; \
+    rm -f /tmp/rpc-ldd.txt; \
+    echo "rpc-server links cleanly (libcuda.so.1 resolves at run time): OK"