Moving it out of the server process was the only way to serve it at all, and
the move turned up what its remote code really needs. Every step here is a
measured constraint, in the order it surfaced on zeiss 2026-10-31:

* peft, then datasets. Its own modeling file imports peft; the retriever it
  loads with trust_remote_code (nvidia/NV-Embed-v1, named in its config.json)
  imports datasets. transformers checks both by SCANNING the files' imports, so
  each absence surfaced only as the next load failure.
* The same retriever is why transformers 5 was never the problem to shim
  around: it imports MISTRAL_INPUTS_DOCSTRING, deleted in 5, and runs its OWN
  decoder loop, calling each Mistral layer without `position_embeddings` --
  which transformers computed inside the attention layer until the rotary
  refactor of 4.43. On 4.49 it dies in transformers' own code with "cannot
  unpack non-iterable NoneType object"; its card says transformers==4.42.4.
  So the venv is per PIN, not one shared "transformers 4" venv.
* 4.42.4 pins tokenizers<0.20, whose newest wheel is cp312, so that venv needs
  python 3.12 -- and the image's /usr/bin/python3.12 has no `-venv` package,
  prints "After installing the python3-venv package…" and STILL EXITS 0,
  leaving a tree with no pip. The check is now `pip --version`, not the exit
  code, and the ladder falls through to a fetched standalone interpreter.
* 4.42.4 also pins numpy<2, which the current peft and datasets cannot live
  with: peft 0.21 imports EncoderDecoderCache (>= 4.43) and datasets 5 pulls
  pyarrow 26 ("requires NumPy 2.0 or newer"). Pinning datasets alone left
  pyarrow 26 in place, because datasets 2.20 accepts pyarrow>=15. Hence
  PIN_EXTRAS: peft 0.11.1, datasets 2.20.0, pyarrow 16.1.0, beside the
  transformers version they belong to and in no other venv.

Verified end to end through coderai's own worker, child and protocol:

    [tf4-embed] nvidia/MM-Embed ready on cpu (4096-d, transformers 4.42.4)
    MM-Embed: text 4096-d image 4096-d finite=True

and jina-clip-v2, unmoved on its own 4.49.0 venv, still answers 1024-d with
cos(it, image)=+0.2588 and cos(ar, image)=+0.2688.

Three defects the bring-up found in the machinery itself, each fixed with tests:

* The probe decided whether an EXISTING venv was usable from five imports, so a
  dependency added to the requirements never reached one: ensure_installed
  returned early and skipped pip entirely. It now lists everything a pin
  installs, including its extras.
* The venv path was not the same in every process. CODERAI_CACHE_DIR is set per
  engine (/AI/guffcache there, /cache elsewhere), so the engine built one venv
  and another process built a second -- 11 GB of the same thing, and the models
  page would have reported a venv the serving engine was not using. The chosen
  path is now recorded in the config root, which the whole fleet shares.
* A child's load failure arrived without a traceback: one line of "TypeError:
  cannot unpack non-iterable NoneType object" from another process, in another
  venv, inside a model's remote code. It now carries one.

The interpreter-provisioning ladder the lip-sync engines needed first moved to
codai.api.venv_env, because this needed exactly the same thing. The register
shim that let MM-Embed overwrite LlavaNextConfig in-process is deleted: out of
the process that cannot happen, and a test keeps its remote code from ever
being imported here again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC