The engines run with LD_LIBRARY_PATH=/opt/coderai/local-libs first, and that
beats the wheel's DT_RUNPATH. So llama-cpp-python dlopens its own libllama.so
by absolute path, and that library then resolves libggml-base.so.0 against
local-libs -- which still held the PREVIOUS build, because replacing the wheel
never touched that directory. A 0.3.36 wheel (libggml 0.25.3, libllama 0.5.0)
against local-libs' 0.9.7/0.0.8351 produced, in all five engines at once:

    Failed to load shared library '.../llama_cpp/lib/libllama.so':
    undefined symbol: ggml_rope_set_offset

That is no GGUF serving at all. It hid because `docker restart` keeps the
container's original image, so the image built this morning was never executed
until the container was recreated this evening -- the rebuild was reported as
deployed and the running process was still 0.3.30.

The libs this build produces are now copied over the stale ones, so both search
paths resolve to the same version in whatever order they are found. Orphaned
older sonames are left alone: harmless once the soname every binary asks for
points at the new file, and removing them would break anything else in the
bundle still linked against them.

The check that passed on the broken image is replaced. `ldd` answers "can every
needed library be found" and is blind to a library that IS found and is the
wrong version; `ldd -r` relocates too and names the symbol exactly as the
runtime does. Two details decide whether it proves anything, and the first
version of it got both wrong:

  - it must run with LD_LIBRARY_PATH set the way the engines set it, because
    the search ORDER is the entire bug;
  - it must check the WHEEL's libllama.so, the file actually dlopened, not
    local-libs' own copy. Old libllama against old libggml is a self-consistent
    pair that reports nothing: checked that way, this image passed with zero
    findings. Checked correctly it reported ten.

Verified in a throwaway container against the broken image: 10 undefined
symbols before, including ggml_rope_set_offset, and 0 after.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC