give an isolated worker the environment it needs, and a resident speech model room to run

Two production failures from the same render, both fixed at the level they
belong to.

**Speech engines "listed but not installed".** XTTS answered 501 "no venv
containing it was found" against a venv that was there and populated; parler
answered 500 "python -m venv returned non-zero exit status 1"; a worker's
torch died with CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED. Three different
failures, one cause: the engine processes export
PYTHONHOME=/opt/coderai/python, every worker was launched with
env=dict(os.environ), and a venv interpreter started under a foreign
PYTHONHOME takes its stdlib and site-packages from THAT installation — so the
venv's own packages are not on sys.path at all. The import probe was the first
casualty, which is why it reported the venv as missing.

codai/api/venv_env.py — which already existed for exactly this, and which the
TTS/STT workers were not using — is now the one place a worker's environment
and its venv are decided. Every call INTO a venv goes through it: the probe,
`python -m venv`, each pip install and the service itself (a clean service with
a poisoned probe still reports the venv as broken; a poisoned pip installs into
the wrong tree). LD_LIBRARY_PATH is decided per venv, because a foreign python
must not see coderai's library directories while a venv built from the
server's own interpreter needs its libpython — and sys.prefix no longer counts
as "ours" when it IS /usr, which on a pip install would strip the driver path
from every worker.

venv_usable() replaces four private copies of "can this venv be used". It
catches what none of them did: a venv whose python runs perfectly and imports
nothing, because its 875 MB of packages are installed for a different minor
version (an xtts_venv built by the host's 3.13 pointing at the container's
3.12). `pyvenv.cfg` spells the version two ways; both are read. An intentional
overlay (--system-site-packages, for the H3 venv that inherits the server's
torch) is respected in both directions.

And `GET /v1/audio/speech/capabilities` now reports needs_venv / env_ready /
env_detail, from the filesystem alone — never importing into the venv, never
loading a model — so a client knows before it offers a voice whether the first
request will spend minutes building the environment.

**"CrisperWhisper out of GPU memory during compose", every scene.**

    word timestamps failed (500: … CUDA out of memory.
    Tried to allocate 86.00 MiB …)

The speech model was loaded and idle between scenes while the TTS and visual
models of the same job loaded around it and filled the card. 86 MiB is what a
resident model needs to run one window, and nothing in the system was keeping
it: every reservation is taken at LOAD time, and this model was already
loaded. Every caption in that render lost its word timings.

Now three numbers per speech model, all measured: the weights, what one pass
needs on top of what was already allocated, and their sum. The per-pass
number is ensured free before each pass (manager.ensure_free_vram) and is a
DELTA, not the process peak — the peak would charge a speech model for a 16 GB
image model resident beside it, and every later load of it would evict the
world. An OOM that still happens frees the card and retries once, on both the
in-process and the isolated-worker path.

The in-process reserve was also a third of the truth: 3 GB hard-coded for a
whisper-family model, where the isolated worker serving the same checkpoint
used 8 GB and said in its own comment that 3 GB was about a third of the real
peak. Both paths now call one function, so eviction frees the same amount
whichever way the model is running.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.17"
+__version__ = "0.5.18"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/audiocraft_worker.py

@@ -30,6 +30,8 @@ import subprocess
 import threading
 from pathlib import Path
 
+from codai.api import venv_env
+
 _REPO_ROOT = Path(__file__).resolve().parents[2]
 _SERVICE = _REPO_ROOT / "tools" / "audiocraft_service.py"
 
@@ -55,8 +57,8 @@ def available() -> bool:
     if not py.is_file() or not _SERVICE.is_file():
         return False
     try:
-        return subprocess.run([str(py), "-c", "import audiocraft"],
-                              capture_output=True, timeout=120).returncode == 0
+        return venv_env.run([str(py), "-c", "import audiocraft"],
+                            capture_output=True, timeout=120).returncode == 0
     except Exception:
         return False
 
@@ -69,7 +71,7 @@ class AudiocraftModel:
             [str(venv_python()), str(_SERVICE)],
             stdin=subprocess.PIPE, stdout=subprocess.PIPE,
             text=True, bufsize=1,
-            env={**os.environ, "PYTHONUNBUFFERED": "1"})
+            env=venv_env.env_for_python(venv_python(), {"PYTHONUNBUFFERED": "1"}))
         self._params: dict = {"duration": 30.0}
         reply = self._call({"op": "load", "model": model_name})
         if not reply.get("ok"):

codai/api/canary_worker.py

@@ -21,11 +21,12 @@ eviction tears the process down.
 import os
 import socket
 import subprocess
-import sys
 import threading
 import time
 from pathlib import Path
 
+from codai.api import venv_env
+
 _REPO_ROOT = Path(__file__).resolve().parents[2]
 _SERVICE_SCRIPT = _REPO_ROOT / "tools" / "canary_service.py"
 _REQUIREMENTS = _REPO_ROOT / "requirements-nemo.txt"
@@ -55,7 +56,7 @@ def _venv_python(venv: Path) -> Path:
 
 def _pip_ok(py: Path) -> bool:
     try:
-        return subprocess.run(
+        return venv_env.run(
             [str(py), "-c", "import nemo.collections.asr"],
             capture_output=True).returncode == 0
     except Exception:
@@ -73,19 +74,20 @@ def _bootstrap_venv() -> Path:
     if py.exists() and _pip_ok(py):
         _bootstrapped = True
         return py
-    if not py.exists():
-        print(f"[canary] creating isolated NeMo venv at {venv} …", flush=True)
-        venv.parent.mkdir(parents=True, exist_ok=True)
-        subprocess.run([sys.executable, "-m", "venv", str(venv)], check=True)
-    py = _venv_python(venv)
+    # create_venv also REBUILDS what cannot be used: a dangling
+    # bin/python from a venv built by another interpreter, or one whose
+    # packages live under a different minor version's site-packages. Both
+    # run `python -m venv` over the top in the old code, which dies on the
+    # very thing that broke them.
+    py = venv_env.create_venv(venv)
     if not _pip_ok(py):
         if not _REQUIREMENTS.exists():
             raise RuntimeError(f"NeMo requirements file missing: {_REQUIREMENTS}")
         print("[canary] installing NeMo (nemo_toolkit[asr]) into the isolated venv "
               "(first run downloads several GB, this can take a while) …", flush=True)
-        subprocess.run([str(py), "-m", "pip", "install", "-U", "pip"], check=True)
-        subprocess.run([str(py), "-m", "pip", "install", "-r", str(_REQUIREMENTS)],
-                       check=True)
+        venv_env.run([str(py), "-m", "pip", "install", "-U", "pip"], check=True)
+        venv_env.run([str(py), "-m", "pip", "install", "-r", str(_REQUIREMENTS)],
+                     check=True)
         if not _pip_ok(py):
             raise RuntimeError("NeMo install did not yield an importable nemo.collections.asr")
     _bootstrapped = True
@@ -147,19 +149,14 @@ def ensure_service(model_name: str, config: dict = None,
         py = _bootstrap_venv()
         port = _free_port()
         url = f"http://127.0.0.1:{port}"
-        env = dict(os.environ)
-        # Share coderai's HF cache. NeMo pulls .nemo checkpoints from HF hub; we
-        # allow the fetch (coderai's downloader doesn't handle .nemo) but keep it
-        # in the shared cache so it's pulled once.
-        try:
-            # All four HF variables, not just the hub pair: this worker runs in
-            # its own venv and never imports codai, so HF_HOME and
-            # TRANSFORMERS_CACHE would stay as the image set them and the model
-            # would be fetched a second time into that directory.
-            from codai.hf_cache import child_env as _hf_child_env
-            env = _hf_child_env(env)
-        except Exception:
-            pass
+        # One call for all three halves: coderai's HF cache variables (this
+        # worker never imports codai, so without them it downloads a second
+        # copy of every model); a clean interpreter environment -- the engine
+        # exports PYTHONHOME, under which this venv's python would take its
+        # stdlib and site-packages from the SERVER's installation; and the
+        # library path this particular venv needs, which depends on whose
+        # interpreter it holds (see venv_env.env_for_venv).
+        env = venv_env.env_for_python(py)
         # Pin the worker to its configured GPU (the 3090 by default).
         gpu = config.get("gpu_device")
         if gpu is not None:

codai/api/crisperwhisper_worker.py

@@ -17,11 +17,12 @@ as a remote STT backend. Mirrors codai.api.canary_worker / pyannote_worker.
 import os
 import socket
 import subprocess
-import sys
 import threading
 import time
 from pathlib import Path
 
+from codai.api import venv_env
+
 _REPO_ROOT = Path(__file__).resolve().parents[2]
 _SERVICE_SCRIPT = _REPO_ROOT / "tools" / "crisperwhisper_service.py"
 _REQUIREMENTS = _REPO_ROOT / "requirements-crisperwhisper.txt"
@@ -65,7 +66,7 @@ def _venv_python(venv: Path) -> Path:
 
 def _pip_ok(py: Path) -> bool:
     try:
-        return subprocess.run(
+        return venv_env.run(
             [str(py), "-c", "import transformers, torch, soundfile"],
             capture_output=True).returncode == 0
     except Exception:
@@ -81,19 +82,20 @@ def _bootstrap_venv() -> Path:
     if py.exists() and _pip_ok(py):
         _bootstrapped = True
         return py
-    if not py.exists():
-        print(f"[crisperwhisper] creating isolated venv at {venv} …", flush=True)
-        venv.parent.mkdir(parents=True, exist_ok=True)
-        subprocess.run([sys.executable, "-m", "venv", str(venv)], check=True)
-    py = _venv_python(venv)
+    # create_venv also REBUILDS what cannot be used: a dangling
+    # bin/python from a venv built by another interpreter, or one whose
+    # packages live under a different minor version's site-packages. Both
+    # run `python -m venv` over the top in the old code, which dies on the
+    # very thing that broke them.
+    py = venv_env.create_venv(venv)
     if not _pip_ok(py):
         if not _REQUIREMENTS.exists():
             raise RuntimeError(f"CrisperWhisper requirements missing: {_REQUIREMENTS}")
         print("[crisperwhisper] installing pinned transformers+torch into the isolated "
               "venv (first run downloads several GB) …", flush=True)
-        subprocess.run([str(py), "-m", "pip", "install", "-U", "pip"], check=True)
-        subprocess.run([str(py), "-m", "pip", "install", "-r", str(_REQUIREMENTS)],
-                       check=True)
+        venv_env.run([str(py), "-m", "pip", "install", "-U", "pip"], check=True)
+        venv_env.run([str(py), "-m", "pip", "install", "-r", str(_REQUIREMENTS)],
+                     check=True)
         if not _pip_ok(py):
             raise RuntimeError("CrisperWhisper venv install did not yield importable deps")
     _bootstrapped = True
@@ -164,16 +166,14 @@ def ensure_service(model_name: str, config: dict = None,
         py = _bootstrap_venv()
         port = _free_port()
         url = f"http://127.0.0.1:{port}"
-        env = dict(os.environ)
-        try:
-            # All four HF variables, not just the hub pair: this worker runs in
-            # its own venv and never imports codai, so HF_HOME and
-            # TRANSFORMERS_CACHE would stay as the image set them and the model
-            # would be fetched a second time into that directory.
-            from codai.hf_cache import child_env as _hf_child_env
-            env = _hf_child_env(env)
-        except Exception:
-            pass
+        # One call for all three halves: coderai's HF cache variables (this
+        # worker never imports codai, so without them it downloads a second
+        # copy of every model); a clean interpreter environment -- the engine
+        # exports PYTHONHOME, under which this venv's python would take its
+        # stdlib and site-packages from the SERVER's installation; and the
+        # library path this particular venv needs, which depends on whose
+        # interpreter it holds (see venv_env.env_for_venv).
+        env = venv_env.env_for_python(py)
         gpu = config.get("gpu_device")
         if gpu is not None:
             env["CUDA_VISIBLE_DEVICES"] = str(gpu)
@@ -265,8 +265,19 @@ def reserved_vram_gb(model_name: str, config: dict = None) -> float:
     here (3 GB) is about a third of what a verbatim pass actually peaks at, so
     eviction freed a third of the room needed and the pass OOMed with the card
     full of models nobody had asked it to keep."""
+    # One source of truth, shared with the in-process path: the same
+    # checkpoint is served both ways and the two must not disagree about how
+    # big it is. DEFAULT_VRAM_GB stays the family default over there.
+    try:
+        from codai.api import stt_backends
+        return stt_backends.reserve_gb(model_name, config, "crisperwhisper")
+    except Exception:
+        pass
     cfg = config or {}
-    explicit = float(cfg.get("used_vram_gb") or 0.0)
+    try:
+        explicit = float(cfg.get("used_vram_gb") or 0.0)
+    except (TypeError, ValueError):
+        explicit = 0.0
     if explicit > 0:
         return explicit
     try:

codai/api/h3_worker.py

@@ -41,12 +41,13 @@ import collections
 import os
 import socket
 import subprocess
-import sys
 import threading
 import time
 from pathlib import Path
 from typing import Optional
 
+from codai.api import venv_env
+
 _REPO_ROOT = Path(__file__).resolve().parents[2]
 _SERVICE_SCRIPT = _REPO_ROOT / "tools" / "h3_service.py"
 _REQUIREMENTS = _REPO_ROOT / "requirements-h3.txt"
@@ -168,7 +169,7 @@ def _venv_python(venv: Path) -> Path:
 def _diffusers_ok(py: Path) -> bool:
     """The venv is usable only if its diffusers actually carries the H3 blocks."""
     try:
-        return subprocess.run(
+        return venv_env.run(
             [str(py), "-c", "from diffusers import MiniMaxH3ModularPipeline"],
             capture_output=True).returncode == 0
     except Exception:
@@ -199,15 +200,13 @@ def _bootstrap_venv(config: dict = None) -> Path:
         _bootstrapped = True
         return py
     standalone = os.environ.get("CODERAI_H3_STANDALONE_VENV", "") in ("1", "true", "yes")
+    kind = "standalone" if standalone else "overlay"
     if not py.exists():
-        kind = "standalone" if standalone else "overlay"
         print(f"[h3] creating {kind} venv at {venv} …", flush=True)
-        venv.parent.mkdir(parents=True, exist_ok=True)
-        cmd = [sys.executable, "-m", "venv"]
-        if not standalone:
-            cmd.append("--system-site-packages")
-        subprocess.run(cmd + [str(venv)], check=True)
-        py = _venv_python(venv)
+    # The overlay inherits the server's torch on purpose, so system_site is
+    # part of what "usable" means here -- an isolated venv found where an
+    # overlay is wanted is rebuilt, and the other way round.
+    py = venv_env.create_venv(venv, system_site=not standalone)
     if not _diffusers_ok(py):
         # NOTE: always `python -m pip`, never the venv's pip script — a venv that
         # was copied carries a stale shebang and would install into the ORIGINAL
@@ -217,13 +216,13 @@ def _bootstrap_venv(config: dict = None) -> Path:
                 raise RuntimeError(f"H3 requirements file missing: {_REQUIREMENTS}")
             print("[h3] installing diffusers>=0.40 + torch into a standalone venv "
                   "(first run downloads several GB, this takes a while) …", flush=True)
-            subprocess.run([str(py), "-m", "pip", "install", "-U", "pip"], check=True)
-            subprocess.run([str(py), "-m", "pip", "install", "-r", str(_REQUIREMENTS)],
-                           check=True)
+            venv_env.run([str(py), "-m", "pip", "install", "-U", "pip"], check=True)
+            venv_env.run([str(py), "-m", "pip", "install", "-r", str(_REQUIREMENTS)],
+                         check=True)
         else:
             print("[h3] adding diffusers>=0.40 to the overlay (torch and the rest "
                   "are inherited from the server's venv) …", flush=True)
-            subprocess.run([str(py), "-m", "pip", "install", "-q",
+            venv_env.run([str(py), "-m", "pip", "install", "-q",
                             "diffusers>=0.40.0", "huggingface_hub>=1.23"], check=True)
         if not _diffusers_ok(py):
             raise RuntimeError(
@@ -297,16 +296,14 @@ def ensure_service(model_path: str, config: dict = None,
         py = _bootstrap_venv(config)
         port = _free_port()
         url = f"http://127.0.0.1:{port}"
-        env = dict(os.environ)
-        try:
-            # All four HF variables, not just the hub pair: this worker runs in
-            # its own venv and never imports codai, so HF_HOME and
-            # TRANSFORMERS_CACHE would stay as the image set them and the model
-            # would be fetched a second time into that directory.
-            from codai.hf_cache import child_env as _hf_child_env
-            env = _hf_child_env(env)
-        except Exception:
-            pass
+        # One call for all three halves: coderai's HF cache variables (this
+        # worker never imports codai, so without them it downloads a second
+        # copy of every model); a clean interpreter environment -- the engine
+        # exports PYTHONHOME, under which this venv's python would take its
+        # stdlib and site-packages from the SERVER's installation; and the
+        # library path this particular venv needs, which depends on whose
+        # interpreter it holds (see venv_env.env_for_venv).
+        env = venv_env.env_for_python(py)
         gpu = config.get("gpu_device")
         if gpu is not None:
             env["CUDA_VISIBLE_DEVICES"] = str(gpu)

codai/api/melotts_worker.py

@@ -24,11 +24,12 @@ the process; the synthesis lives in tools/melotts_service.py.
 import os
 import socket
 import subprocess
-import sys
 import threading
 import time
 from pathlib import Path
 
+from codai.api import venv_env
+
 _REPO_ROOT = Path(__file__).resolve().parents[2]
 _SERVICE_SCRIPT = _REPO_ROOT / "tools" / "melotts_service.py"
 
@@ -50,17 +51,8 @@ def _venv_python() -> Path:
 
 def _pip_ok(py: Path) -> bool:
     try:
-        return subprocess.run([str(py), "-c", "import melo, soundfile"],
-                              capture_output=True).returncode == 0
-    except Exception:
-        return False
-
-
-def _venv_is_system_site() -> bool:
-    """True if the venv was built with --system-site-packages (can't isolate)."""
-    try:
-        return "include-system-site-packages = true" in \
-            (_VENV_DIR / "pyvenv.cfg").read_text().lower()
+        return venv_env.run([str(py), "-c", "import melo, soundfile"],
+                            capture_output=True).returncode == 0
     except Exception:
         return False
 
@@ -75,21 +67,15 @@ def _bootstrap_venv() -> Path:
     py = _venv_python()
     if _bootstrapped and py.exists():
         return py
-    # A previously-created shared-site venv leaks the server's transformers in;
-    # rebuild it isolated.
-    if py.exists() and _venv_is_system_site():
-        import shutil
-        print("[melotts] rebuilding venv as fully isolated …", flush=True)
-        shutil.rmtree(_VENV_DIR, ignore_errors=True)
-    if not _venv_python().exists():
-        print(f"[melotts] creating isolated venv at {_VENV_DIR} …", flush=True)
-        _VENV_DIR.parent.mkdir(parents=True, exist_ok=True)
-        subprocess.run([sys.executable, "-m", "venv", str(_VENV_DIR)], check=True)
-    py = _venv_python()
+    # One rebuild decision for every way a venv can be unusable: a shared-site
+    # build that leaks the server's transformers back in, a dangling
+    # bin/python from another interpreter, or packages installed for a
+    # different minor version. See venv_env.venv_usable.
+    py = venv_env.create_venv(_VENV_DIR)
     if not _pip_ok(py):
         print("[melotts] installing melotts + torch into the isolated venv "
               "(first run, downloads several GB, this can take a while) …", flush=True)
-        subprocess.run([str(py), "-m", "pip", "install",
+        venv_env.run([str(py), "-m", "pip", "install",
                         "git+https://github.com/huggingface/melotts.git",
                         "soundfile"], check=True)
         if not _pip_ok(py):
@@ -155,20 +141,14 @@ def ensure_service(model_name: str, ready_timeout: float = 1800.0,
         py = _bootstrap_venv()
         port = _free_port()
         url = f"http://127.0.0.1:{port}"
-        env = dict(os.environ)
-        # The worker must use the model already pulled via coderai's HF download
-        # interface — it never downloads anything itself. Point it at coderai's
-        # cache and force offline mode, so a missing model fails fast instead of
-        # silently fetching.
-        try:
-            # All four HF variables, not just the hub pair: this worker runs in
-            # its own venv and never imports codai, so HF_HOME and
-            # TRANSFORMERS_CACHE would stay as the image set them and the model
-            # would be fetched a second time into that directory.
-            from codai.hf_cache import child_env as _hf_child_env
-            env = _hf_child_env(env)
-        except Exception:
-            pass
+        # One call for all three halves: coderai's HF cache variables (this
+        # worker never imports codai, so without them it downloads a second
+        # copy of every model); a clean interpreter environment -- the engine
+        # exports PYTHONHOME, under which this venv's python would take its
+        # stdlib and site-packages from the SERVER's installation; and the
+        # library path this particular venv needs, which depends on whose
+        # interpreter it holds (see venv_env.env_for_venv).
+        env = venv_env.env_for_python(py)
         env["HF_HUB_OFFLINE"] = "1"
         env["TRANSFORMERS_OFFLINE"] = "1"
         proc = subprocess.Popen(

codai/api/parler_worker.py

@@ -20,11 +20,12 @@ manager's normal eviction tears the process down — no manual setup or config.
 import os
 import socket
 import subprocess
-import sys
 import threading
 import time
 from pathlib import Path
 
+from codai.api import venv_env
+
 _REPO_ROOT = Path(__file__).resolve().parents[2]
 _SERVICE_SCRIPT = _REPO_ROOT / "tools" / "parler_tts_service.py"
 
@@ -44,43 +45,10 @@ def _venv_python() -> Path:
         "python.exe" if os.name == "nt" else "python")
 
 
-def _venv_runs(py: Path) -> bool:
-    """Whether this venv's interpreter actually RUNS.
-
-    ``Path.exists()`` is not enough and its failure mode is nasty. A venv's
-    ``bin/python`` is a symlink to the interpreter that built it, so a venv
-    created by a DIFFERENT python -- on the host rather than in the container,
-    or before the image moved its interpreter -- leaves a dangling link.
-    ``exists()`` follows symlinks and answers False, so the code concludes the
-    venv is missing and tries to create one over the top; ``python -m venv``
-    then trips over the same dangling ``bin/python3`` and dies:
-
-        Error: [Errno 2] No such file or directory: '.../parler_venv/bin/python3'
-
-    which surfaced as "Command '[... -m venv ...]' returned non-zero exit
-    status 1" on every request, for ever, with a fully populated venv sitting
-    right there. Measured 2026-10-28.
-    """
-    try:
-        return subprocess.run([str(py), "-c", "import sys"],
-                              capture_output=True, timeout=120).returncode == 0
-    except Exception:
-        return False
-
-
 def _pip_ok(py: Path) -> bool:
     try:
-        return subprocess.run([str(py), "-c", "import parler_tts, soundfile"],
-                              capture_output=True).returncode == 0
-    except Exception:
-        return False
-
-
-def _venv_is_system_site() -> bool:
-    """True if the venv was built with --system-site-packages (can't isolate)."""
-    try:
-        return "include-system-site-packages = true" in \
-            (_VENV_DIR / "pyvenv.cfg").read_text().lower()
+        return venv_env.run([str(py), "-c", "import parler_tts, soundfile"],
+                            capture_output=True).returncode == 0
     except Exception:
         return False
 
@@ -95,28 +63,16 @@ def _bootstrap_venv() -> Path:
     py = _venv_python()
     if _bootstrapped and py.exists():
         return py
-    # A previously-created shared-site venv leaks the server's transformers in;
-    # rebuild it isolated.
-    if py.exists() and _venv_is_system_site():
-        import shutil
-        print("[parler] rebuilding venv as fully isolated …", flush=True)
-        shutil.rmtree(_VENV_DIR, ignore_errors=True)
-    # A venv built by another interpreter cannot be repaired in place -- see
-    # _venv_runs. Clear it out so the creation below starts from nothing.
-    if _VENV_DIR.exists() and not _venv_runs(_venv_python()):
-        import shutil
-        print(f"[parler] {_VENV_DIR} was built by a different interpreter "
-              f"(its python does not run here); rebuilding …", flush=True)
-        shutil.rmtree(_VENV_DIR, ignore_errors=True)
-    if not _venv_python().exists():
-        print(f"[parler] creating isolated venv at {_VENV_DIR} …", flush=True)
-        _VENV_DIR.parent.mkdir(parents=True, exist_ok=True)
-        subprocess.run([sys.executable, "-m", "venv", str(_VENV_DIR)], check=True)
-    py = _venv_python()
+    # Everything that makes a venv unusable -- a shared-site build that leaks
+    # the server's transformers in, an interpreter that no longer runs, a
+    # minor-version mismatch against its own site-packages -- is one rebuild
+    # decision, taken in venv_env.create_venv. None of them can be repaired
+    # in place: `python -m venv` over the top dies on the thing that broke it.
+    py = venv_env.create_venv(_VENV_DIR)
     if not _pip_ok(py):
         print("[parler] installing parler-tts + torch into the isolated venv "
               "(first run, downloads several GB, this can take a while) …", flush=True)
-        subprocess.run([str(py), "-m", "pip", "install",
+        venv_env.run([str(py), "-m", "pip", "install",
                         "git+https://github.com/huggingface/parler-tts.git",
                         "soundfile"], check=True)
         if not _pip_ok(py):
@@ -178,22 +134,18 @@ def ensure_service(model_name: str, ready_timeout: float = 1800.0) -> str:
         py = _bootstrap_venv()
         port = _free_port()
         url = f"http://127.0.0.1:{port}"
-        env = dict(os.environ)
         # The worker must use the model already pulled via coderai's HF download
         # interface — it never downloads anything itself. Point it at coderai's
-        # cache and force offline mode, so a missing model fails fast instead of
-        # silently fetching.
-        try:
-            # All four HF variables, not just the hub pair: this worker runs in
-            # its own venv and never imports codai, so HF_HOME and
-            # TRANSFORMERS_CACHE would stay as the image set them and the model
-            # would be fetched a second time into that directory.
-            from codai.hf_cache import child_env as _hf_child_env
-            env = _hf_child_env(env)
-        except Exception:
-            pass
-        env["HF_HUB_OFFLINE"] = "1"
-        env["TRANSFORMERS_OFFLINE"] = "1"
+        # cache (all four HF variables, since it never imports codai) and force
+        # offline mode, so a missing model fails fast instead of silently
+        # fetching. env_for_python also drops PYTHONHOME -- under it this
+        # venv's python took its stdlib and site-packages from the SERVER's
+        # installation, which is how a populated venv reported itself broken --
+        # and decides the library path from whose interpreter the venv holds:
+        # a foreign python loading this one's extension modules is the
+        # CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED shape.
+        env = venv_env.env_for_python(py, {"HF_HUB_OFFLINE": "1",
+                                           "TRANSFORMERS_OFFLINE": "1"})
         proc = subprocess.Popen(
             [str(py), str(_SERVICE_SCRIPT), "--model", model_name,
              "--host", "127.0.0.1", "--port", str(port)],

codai/api/pyannote_worker.py

@@ -25,11 +25,12 @@ token is needed.
 import os
 import socket
 import subprocess
-import sys
 import threading
 import time
 from pathlib import Path
 
+from codai.api import venv_env
+
 _REPO_ROOT = Path(__file__).resolve().parents[2]
 _SERVICE_SCRIPT = _REPO_ROOT / "tools" / "pyannote_service.py"
 _REQUIREMENTS = _REPO_ROOT / "requirements-pyannote.txt"
@@ -57,7 +58,7 @@ def _venv_python(venv: Path) -> Path:
 
 def _pip_ok(py: Path) -> bool:
     try:
-        return subprocess.run(
+        return venv_env.run(
             [str(py), "-c", "import pyannote.audio"],
             capture_output=True).returncode == 0
     except Exception:
@@ -73,19 +74,20 @@ def _bootstrap_venv() -> Path:
     if py.exists() and _pip_ok(py):
         _bootstrapped = True
         return py
-    if not py.exists():
-        print(f"[pyannote] creating isolated venv at {venv} …", flush=True)
-        venv.parent.mkdir(parents=True, exist_ok=True)
-        subprocess.run([sys.executable, "-m", "venv", str(venv)], check=True)
-    py = _venv_python(venv)
+    # create_venv also REBUILDS what cannot be used: a dangling
+    # bin/python from a venv built by another interpreter, or one whose
+    # packages live under a different minor version's site-packages. Both
+    # run `python -m venv` over the top in the old code, which dies on the
+    # very thing that broke them.
+    py = venv_env.create_venv(venv)
     if not _pip_ok(py):
         if not _REQUIREMENTS.exists():
             raise RuntimeError(f"pyannote requirements file missing: {_REQUIREMENTS}")
         print("[pyannote] installing pyannote.audio into the isolated venv "
               "(first run downloads several GB, this can take a while) …", flush=True)
-        subprocess.run([str(py), "-m", "pip", "install", "-U", "pip"], check=True)
-        subprocess.run([str(py), "-m", "pip", "install", "-r", str(_REQUIREMENTS)],
-                       check=True)
+        venv_env.run([str(py), "-m", "pip", "install", "-U", "pip"], check=True)
+        venv_env.run([str(py), "-m", "pip", "install", "-r", str(_REQUIREMENTS)],
+                     check=True)
         if not _pip_ok(py):
             raise RuntimeError("pyannote install did not yield an importable pyannote.audio")
     _bootstrapped = True
@@ -172,16 +174,14 @@ def ensure_service(model_name: str = None, config: dict = None,
         py = _bootstrap_venv()
         port = _free_port()
         url = f"http://127.0.0.1:{port}"
-        env = dict(os.environ)
-        try:
-            # All four HF variables, not just the hub pair: this worker runs in
-            # its own venv and never imports codai, so HF_HOME and
-            # TRANSFORMERS_CACHE would stay as the image set them and the model
-            # would be fetched a second time into that directory.
-            from codai.hf_cache import child_env as _hf_child_env
-            env = _hf_child_env(env)
-        except Exception:
-            pass
+        # One call for all three halves: coderai's HF cache variables (this
+        # worker never imports codai, so without them it downloads a second
+        # copy of every model); a clean interpreter environment -- the engine
+        # exports PYTHONHOME, under which this venv's python would take its
+        # stdlib and site-packages from the SERVER's installation; and the
+        # library path this particular venv needs, which depends on whose
+        # interpreter it holds (see venv_env.env_for_venv).
+        env = venv_env.env_for_python(py)
         # Pass an HF token through for the gated model (harmless for ungated).
         tok = _resolve_hf_token(config)
         if tok:

codai/api/transcriptions.py

@@ -640,10 +640,16 @@ async def _run_transcription_inner(
                     uses_gpu = bool(torch.cuda.is_available())
                 except Exception:
                     uses_gpu = False
-                # whisper HF (e.g. CrisperWhisper large) is a bit bigger than
-                # wav2vec2; reserve a touch more by default.
-                _dflt = 3.0 if family == 'whisper_hf' else 2.0
-                needed_gb = float(stt_cfg.get('used_vram_gb') or _dflt) if uses_gpu else 0.0
+                # One source of truth for how big a speech model is, shared
+                # with the isolated worker that serves the same checkpoint the
+                # other way: the operator's number, else what this model
+                # measured on an earlier pass, else the family default. The 3.0
+                # that used to be hard-coded here was about a third of what a
+                # verbatim pass peaks at, so eviction freed a third of the room
+                # and the pass OOMed on a card full of models nobody had asked
+                # to keep.
+                needed_gb = (stt_backends.reserve_gb(model_name, stt_cfg, family)
+                             if uses_gpu else 0.0)
 
             def _run_backend():
                 def _loader():

codai/api/tts.py

@@ -110,12 +110,23 @@ async def speech_capabilities(model: str = ""):
         voices = tts_backends.family_voices(name, backend)
     except Exception as exc:
         print(f"  [tts] could not list voices for {name!r}: {exc}", flush=True)
+    # Whether the engine can answer now, or whether the first request will
+    # spend minutes building its isolated venv. Checked from the filesystem,
+    # never by importing into that venv -- a catalogue call must not cost
+    # seconds, and must not load a model.
+    env = {"needs_venv": False, "env_ready": True, "env_detail": ""}
+    try:
+        env = tts_backends.environment_report(name)
+    except Exception as exc:
+        print(f"  [tts] could not check the environment for {name!r}: {exc}",
+              flush=True)
     return {
         "model": name,
         "family": tts_backends._family(name),
         "emotions": tts_backends.family_emotions(name),
         "styles": tts_backends.family_styles(name),
         **voices,
+        **env,
     }
 
 

codai/api/tts_backends.py

@@ -658,3 +658,79 @@ def encode_audio(samples: np.ndarray, sample_rate: int, fmt: str) -> Tuple[bytes
         except Exception as exc:
             print(f"  ffmpeg transcode to {fmt} failed ({exc}); returning WAV")
     return wav_bytes, "wav"
+
+
+# --------------------------------------------------------------------------- #
+# Whether an engine can run at all, before anything is loaded
+# --------------------------------------------------------------------------- #
+#
+# coqui-tts and parler-tts each need a transformers the server cannot have, so
+# each runs in a venv of its own, built on first use. That build downloads
+# several GB, which is a surprising thing for a client to discover inside its
+# first synthesis request -- reported from production as "speech engines listed
+# but not installed", with a 501 and a 500 as the only explanation.
+#
+# This answers the question before a user picks a voice, and it answers it
+# WITHOUT importing anything from the venv: an import probe into a foreign
+# interpreter costs seconds, and a catalogue call that costs seconds is how a
+# listing route becomes a loader.
+
+#: family -> (module that proves the venv works, venv locator)
+_VENV_FAMILIES = {
+    "coqui": "TTS",
+    "parler": "parler_tts",
+    "melo": "melo",
+}
+
+
+def _family_venv_dir(fam: str):
+    """Where this family's isolated venv lives, or None if it needs none."""
+    try:
+        if fam == "coqui":
+            from codai.api.xtts_worker import venv_dir
+            return venv_dir()
+        if fam == "parler":
+            from codai.api.parler_worker import _VENV_DIR
+            return _VENV_DIR
+        if fam == "melo":
+            from codai.api.melotts_worker import _VENV_DIR
+            return _VENV_DIR
+    except Exception:
+        return None
+    return None
+
+
+def environment_report(model_name: str) -> dict:
+    """``{needs_venv, env_ready, env_detail}`` for this model's engine.
+
+    ``env_ready`` false does not mean the engine is unusable: the venv is built
+    on first use. It means the first request will spend minutes building it,
+    which is exactly what a client wants to know before offering the voice.
+    """
+    from pathlib import Path
+
+    fam = _family(model_name)
+    module = _VENV_FAMILIES.get(fam)
+    if not module:
+        return {"needs_venv": False, "env_ready": True, "env_detail": ""}
+    d = _family_venv_dir(fam)
+    if d is None:
+        return {"needs_venv": True, "env_ready": False,
+                "env_detail": "this engine's venv location is not resolvable here"}
+    try:
+        from codai.api import venv_env
+        # probe=False: filesystem only. This route is a catalogue call.
+        ok, why = venv_env.venv_usable(d, probe=False)
+    except Exception as exc:
+        return {"needs_venv": True, "env_ready": False,
+                "env_detail": f"the venv could not be inspected ({exc})"}
+    if not ok:
+        return {"needs_venv": True, "env_ready": False,
+                "env_detail": f"{d}: {why or 'it is not there yet'}"}
+    # The package directory, not an import: a populated venv is the evidence,
+    # and importing into it would cost seconds on a catalogue call.
+    found = any(Path(d).glob(f"lib/python*/site-packages/{module}"))
+    if not found:
+        return {"needs_venv": True, "env_ready": False,
+                "env_detail": f"{d} exists but {module} is not installed in it"}
+    return {"needs_venv": True, "env_ready": True, "env_detail": str(d)}

codai/api/xtts_worker.py

@@ -27,10 +27,11 @@ import io
 import json
 import os
 import subprocess
-import sys
 import threading
 from pathlib import Path
 
+from codai.api import venv_env
+
 _REPO_ROOT = Path(__file__).resolve().parents[2]
 _SERVICE = _REPO_ROOT / "tools" / "xtts_service.py"
 
@@ -56,8 +57,11 @@ def available() -> bool:
     if not py.is_file() or not _SERVICE.is_file():
         return False
     try:
-        return subprocess.run([str(py), "-c", "import TTS"],
-                              capture_output=True, timeout=120).returncode == 0
+        # Clean environment: the server exports PYTHONHOME, and under it this
+        # probe imports the SERVER's site-packages, so a populated venv
+        # answered "coqui-tts is not installed" for ever (see venv_env).
+        return venv_env.run([str(py), "-c", "import TTS"],
+                            capture_output=True, timeout=120).returncode == 0
     except Exception:
         return False
 
@@ -66,13 +70,15 @@ _bootstrap_lock = threading.Lock()
 _bootstrapped = False
 
 
-def _venv_runs(py: Path) -> bool:
-    """Whether this venv's interpreter actually RUNS (see parler_worker)."""
-    try:
-        return subprocess.run([str(py), "-c", "import sys"],
-                              capture_output=True, timeout=120).returncode == 0
-    except Exception:
-        return False
+def _venv_ok() -> bool:
+    """Whether the venv on disk is one this process can use at all.
+
+    Not just "its python runs": a venv built by another 3.13 whose packages
+    live in a different minor version's directory runs and imports nothing.
+    See :func:`codai.api.venv_env.venv_usable`.
+    """
+    ok, _why = venv_env.venv_usable(venv_dir())
+    return ok
 
 
 def bootstrap(timeout: int = 3600) -> bool:
@@ -101,28 +107,20 @@ def bootstrap(timeout: int = 3600) -> bool:
             return True
         d = venv_dir()
         try:
-            py = venv_python()
-            # A venv built by a different interpreter leaves a dangling
-            # bin/python: is_file() says no, and `python -m venv` over the top
-            # then dies on that same dangling link, so the engine is stuck for
-            # good with a populated venv sitting right there. Clear it first.
-            if d.exists() and not _venv_runs(py):
-                import shutil
-                print(f"[xtts] {d} was built by a different interpreter "
-                      f"(its python does not run here); rebuilding …",
-                      flush=True)
-                shutil.rmtree(d, ignore_errors=True)
-            if not py.is_file():
-                print(f"[xtts] creating isolated venv at {d} …", flush=True)
-                d.parent.mkdir(parents=True, exist_ok=True)
-                subprocess.run([sys.executable, "-m", "venv", str(d)],
-                               check=True, timeout=600)
-            py = venv_python()
+            # create_venv rebuilds what cannot be used -- a dangling
+            # bin/python, a minor-version mismatch against its own
+            # site-packages, or a --system-site-packages build -- because
+            # `python -m venv` over any of those dies on the thing that broke
+            # it, leaving the engine stuck for good with a populated venv
+            # sitting right there.
+            if not venv_python().is_file() or not _venv_ok():
+                print(f"[xtts] preparing the isolated venv at {d} …", flush=True)
+            py = venv_env.create_venv(d)
             print("[xtts] installing coqui-tts into the isolated venv "
                   "(first run, downloads several GB) …", flush=True)
-            subprocess.run([str(py), "-m", "pip", "install", "--no-input",
+            venv_env.run([str(py), "-m", "pip", "install", "--no-input",
                             "coqui-tts", "soundfile", "requests"],
-                           check=True, timeout=timeout)
+                         check=True, timeout=timeout)
         except Exception as exc:
             # A partially built venv is worse than none: `available()` stays
             # False, and the next attempt resumes rather than reporting a venv
@@ -153,10 +151,11 @@ class XttsSubprocess:
         self._proc = subprocess.Popen(
             [str(venv_python()), str(_SERVICE)],
             stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True, bufsize=1,
-            env={**os.environ, "PYTHONUNBUFFERED": "1",
-                 # CPML acceptance: configuring an XTTS model IS the decision,
-                 # and the alternative is an interactive prompt on a pipe.
-                 "COQUI_TOS_AGREED": os.environ.get("COQUI_TOS_AGREED", "1")})
+            env=venv_env.env_for_python(venv_python(), {
+                "PYTHONUNBUFFERED": "1",
+                # CPML acceptance: configuring an XTTS model IS the decision,
+                # and the alternative is an interactive prompt on a pipe.
+                "COQUI_TOS_AGREED": os.environ.get("COQUI_TOS_AGREED", "1")}))
         reply = self._call({"op": "load", "model": model_name})
         if not reply.get("ok"):
             self.close()

codai/models/manager.py

@@ -5763,6 +5763,31 @@ class MultiModelManager:
                 print(f"  Idle unload of '{key}' failed: {e}")
         return freed
 
+    def ensure_free_vram(self, gb: float, reason: str = "") -> float:
+        """Make sure ``gb`` is free right now, evicting others if it is not.
+
+        Different question from :meth:`ensure_vram_for`, which asks "can this
+        model be LOADED": a model that is already resident still needs room for
+        the pass it is about to run. Measured on a production render,
+        2026-10-29: an in-process CrisperWhisper sat resident while the TTS and
+        visual models for the same job loaded around it, filling the card to
+        the edge; its next window then failed on an 86 MiB allocation, once per
+        scene, and every caption lost its word timings.
+
+        Returns the free VRAM afterwards, so a caller can log what it got
+        rather than what it asked for.
+        """
+        if not gb or gb <= 0:
+            return self._get_free_vram_gb()
+        free = self._get_free_vram_gb()
+        if free >= gb:
+            return free
+        print(f"Need {gb:.2f} GB free VRAM"
+              + (f" for {reason}" if reason else "")
+              + f", have {free:.2f} GB — evicting to make room")
+        self._evict_models_for_vram(gb)
+        return self._get_free_vram_gb()
+
     def _evict_models_for_vram(self, needed_gb: float):
         """Unload loaded models (LRU first) until we have at least needed_gb free VRAM.
 

docs/README.md

@@ -29,6 +29,7 @@ reference; these are the deep dives.
 | [chunking.md](chunking.md) | `POST /v1/chunk`: splitting a document for a retrieval index, with the structure decided per request by a template |
 | [moderation.md](moderation.md) | `POST /v1/moderation`: screening generated images and video before release (explicitness, faces, age, performer match) — and why it fails closed |
 | [expressive-tts.md](expressive-tts.md) | Expressive text-to-speech |
+| [isolated-worker-environments.md](isolated-worker-environments.md) | Engines that need an interpreter of their own: the inherited `PYTHONHOME` that made three different failures, when a venv that is there cannot be used, and `env_ready` before the first request |
 | [image-prompts.md](image-prompts.md) | Image prompts: the `(tag:1.3)` / `[tag]` / `BREAK` grammar, prompts past CLIP's 77 tokens, `clip_skip`, negative prompts, textual-inversion embeddings, size validation |
 | [zimage-lora-training.md](zimage-lora-training.md) | LoRA training for image models |
 | [text-to-music.md](text-to-music.md) | `POST /v1/audio/generate` with ACE-Step: style prompt + sung lyrics → a 48 kHz stereo track, the python-3.12 worker it needs, and the two image variants (`--with-acestep`) |

docs/isolated-worker-environments.md

+# Capabilities that need an interpreter of their own
+
+Some engines cannot share the server's interpreter. coqui-tts imports a
+symbol transformers 5 removed; parler-tts, NeMo and pyannote pin an old
+transformers/tokenizers; MeloTTS the same. Pinning any of them in the server
+would break every other capability, so each gets a venv of its own, built on
+first use, and coderai drives it as a subprocess.
+
+That part worked. What did not was the environment those subprocesses were
+started in.
+
+## The one variable that broke all of them
+
+Measured on the engine process, 2026-10-29:
+
+```
+HOME=/home/ubuntu
+PYTHONHOME=/opt/coderai/python
+```
+
+Every worker was launched with `env=dict(os.environ)`, so every one of them
+inherited that `PYTHONHOME`. A venv interpreter started under a `PYTHONHOME`
+that points at a *different* installation takes its stdlib and its
+`site-packages` from there — the venv's own packages are not on `sys.path` at
+all.
+
+What the operator saw instead:
+
+| reported as | actually |
+|---|---|
+| `501 … no venv containing it was found` (XTTS) | the venv was there and populated; the `import TTS` probe ran against the server's site-packages |
+| `Command '[… -m venv …]' returned non-zero exit status 1` (parler) | `python -m venv` run under a foreign `PYTHONHOME` |
+| `CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED` | the same shape one level down: the worker's torch resolving shared libraries through another installation's directories |
+
+Three different-looking failures, one cause — which is why each was chased
+separately and none was fixed.
+
+`codai/api/venv_env.py` is now the only place a worker's environment is
+built. It drops what redirects an interpreter (`PYTHONHOME`, `PYTHONPATH`,
+`PYTHONSTARTUP`, `PYTHONEXECUTABLE`, `VIRTUAL_ENV`, …).
+
+`LD_LIBRARY_PATH` is decided per venv, because the two kinds of worker need
+opposite things:
+
+* a **foreign** interpreter (ACE-Step's 3.12, LatentSync's 3.11, the lipsync
+  3.10) must not see coderai's library directories, or it loads this python's
+  extension modules — `ImportError: libtorch_python.so: undefined symbol:
+  PyObject_GetOptionalAttrString`, a CPython 3.13 API, from a 3.10 process;
+* a venv built from **the server's own** interpreter (parler, CrisperWhisper,
+  NeMo, pyannote) runs that very python and must find its `libpython`, so
+  stripping those directories would stop it starting at all.
+
+`env_for_venv` reads the venv's `pyvenv.cfg` and answers for that venv.
+Everything outside the server's installation is kept either way: a worker that
+cannot see `/usr/local/nvidia/lib` has no GPU, which is a worse failure than
+the one being fixed. (And `sys.prefix` is ignored when it IS the system
+prefix — a pip install puts it at `/usr`, where treating it as "ours" would
+strip the driver from every worker.)
+
+It applies to **every** call into a venv: the import probe, `python -m venv`,
+each `pip install` and the service process itself. A clean service process with
+a probe run under the poisoned environment still reports the venv as broken,
+and a clean probe with a poisoned `pip` installs into the wrong tree. A test
+enforces that no isolated worker builds its own environment dict.
+
+It does **not** apply to coderai's own children — the engine supervisor,
+cluster RPC — which are the same interpreter, and for which `PYTHONHOME` is how
+they find it.
+
+## When a venv that is there cannot be used
+
+`python -m venv` over a broken venv dies on the very thing that broke it, so a
+venv is never repaired in place. `venv_env.create_venv` rebuilds it, and
+`venv_usable` is the single decision about when:
+
+* **a dangling interpreter.** `bin/python` is a symlink to the python that
+  built it. Build it on the host, run it in the container — or build it before
+  the image moved its interpreter — and the link dangles. `Path.exists()`
+  follows symlinks and answers False, so the old code concluded "missing" and
+  ran `venv` over the top: `Errno 2 … '/…/parler_venv/bin/python3'`, on every
+  request, for ever, with 7 GB of packages sitting in it.
+* **a minor-version mismatch.** Measured the same day: an `xtts_venv` whose
+  `bin/python3` pointed at `/usr/bin/python3`, built by the host's 3.13 while
+  the container's is 3.12. The interpreter runs perfectly and imports none of
+  the 875 MB under `lib/python3.13/site-packages`. "Does it run" cannot see
+  this.
+* **`--system-site-packages` where isolation was wanted** — the server's
+  incompatible stack leaks straight back in. The reverse counts too: the
+  MiniMax-H3 overlay inherits the server's torch *on purpose*, so an isolated
+  venv found where an overlay is wanted is also rebuilt.
+
+## Knowing before you ask
+
+A venv is built on first use, and that build downloads several GB. Discovering
+that inside a synthesis request is a surprise, so
+`GET /v1/audio/speech/capabilities?model=…` answers it up front:
+
+```json
+{
+  "model": "coqui/XTTS-v2",
+  "family": "coqui",
+  "voices": [{"id": "Claribel Dervla", "gender": null, "languages": []}],
+  "voice_selection": "builtin|clone",
+  "needs_venv": true,
+  "env_ready": false,
+  "env_detail": "/home/ubuntu/.coderai/xtts_venv: it does not exist"
+}
+```
+
+`env_ready: false` does not mean unusable — it means the first request will
+spend minutes building the environment, which is what a client wants to know
+before offering the voice.
+
+Answered from the filesystem alone: the venv's `pyvenv.cfg`, its recorded
+interpreter and the package directory. Never by importing into the venv, and
+never by loading a model — a catalogue call that costs seconds is how a listing
+route turns into a loader.
+
+## The engines
+
+| engine | needs a venv | package |
+|---|---|---|
+| Kokoro, Bark, transformers TTS | no | — |
+| XTTS (coqui) | yes | `coqui-tts` |
+| Parler | yes | `parler-tts` |
+| MeloTTS | yes | `melo` |
+| CrisperWhisper | yes | pinned `transformers` + `torch` |
+| NeMo / Canary | yes | `nemo_toolkit[asr]` |
+| pyannote (diarization) | yes | `pyannote.audio` |
+| MiniMax-H3 | overlay | `diffusers>=0.40` |
+
+`CODERAI_XTTS_VENV`, `CODERAI_PARLER_VENV` and the equivalents point at a venv
+built elsewhere — a pod image that bakes one, for instance. A `service_url` on
+the model entry points at a worker running on another host entirely, and then
+nothing is built, spawned or downloaded locally.

docs/memory-accounting.md

@@ -112,6 +112,53 @@ cache that the kernel reclaims by itself and that evicting a model cannot free.
 The cap can only be met by releasing memory something actually holds, so the
 metric has to be one that releasing memory moves.
 
+## A resident model still needs room to run
+
+Every reservation above is taken when a model LOADS. A model that is already
+loaded asks for nothing — and a pass still costs more than the weights.
+
+Measured on a production render, 2026-10-29: a speech model sat resident
+between scenes while the TTS and visual models of the same job loaded around
+it and filled the card. Then every scene logged
+
+```
+word timestamps failed (500: … CUDA out of memory. Tried to allocate 86.00 MiB)
+```
+
+and every caption in the render lost its word timings. 86 MiB is what a
+resident model needs to run one window, and nothing in the system was keeping
+it.
+
+Three numbers per speech model, all measured after its first pass:
+
+| | |
+|---|---|
+| `weights` | what the load put on the card |
+| `activation` | what one pass needed **on top of what was already allocated** |
+| reserve | `weights + activation`: what to free before loading it |
+
+`activation` is what is now ensured free before each pass
+(`ensure_free_vram`), and the reserve is what is freed before a load.
+
+It is a delta on purpose. The process's absolute peak would charge a speech
+model for every other model resident beside it — a 16 GB image model sitting
+on the card during a transcription would be recorded as part of the speech
+model's footprint, and every later load of it would evict the world. Both are
+high-water marks, never averages: a one-word clip touches a fraction of what a
+full 30 s window does, and an average under-reserves for the longer audio that
+follows. Until a model has run once, a 1 GB floor stands in for `activation` —
+deliberately modest, because the model is already resident and the failure
+being prevented was 86 MiB.
+
+The same checkpoint can be served in-process or by an isolated worker, and the
+two used to disagree about its size: 3 GB in one path against 8 GB in the
+other, where 8 GB was the measured one. They now share one function, so
+eviction frees the same amount whichever way the model is running.
+
+An OOM that still happens mid-pass frees the card and retries once. A pass is
+seconds of work on audio that already exists; the expensive outcome is a
+render whose captions have no timings.
+
 ## Idle unload
 
 Eviction frees a model when something else needs the memory.

tests/test_hf_cache_is_configured_not_inherited.py

@@ -181,10 +181,21 @@ def test_current_dir_is_derived_from_the_hub_path(monkeypatch):
 ])
 def test_no_worker_sets_only_the_hub_pair(worker):
     """The regression this guards: two of four variables set, model downloaded
-    twice. Any worker that spawns a venv must go through child_env."""
+    twice. Any worker that spawns a venv must build its environment with a
+    helper that applies all four -- never by hand.
+
+    The helper moved: the workers now call ``venv_env.env_for_python``, which
+    applies ``hf_cache.child_env`` AND decides the library path from whose
+    interpreter the venv holds. What is asserted is that the environment comes
+    from one of those helpers, not which one."""
     src = open(f"codai/api/{worker}_worker.py").read()
     assert 'env["HF_HUB_CACHE"]' not in src, f"{worker}: sets the hub pair by hand"
-    assert "child_env" in src
+    assert any(h in src for h in ("child_env", "env_for_python", "env_for_venv")), (
+        f"{worker}: builds its environment without a helper that applies all "
+        f"four HF variables")
+    assert "dict(os.environ)" not in src and "{**os.environ" not in src, (
+        f"{worker}: builds its own environment dict, so it inherits both the "
+        f"image's HF variables and the engine's PYTHONHOME")
 
 
 def test_the_ocr_supervisor_uses_the_configured_dir(image_env):

tests/test_stt_vram_headroom.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+"""A resident speech model still needs room for the pass it is about to run.
+
+Measured on a production render, 2026-10-29. Every scene logged:
+
+    word timestamps failed (500: … CrisperWhisper worker error 500:
+    CUDA out of memory. Tried to allocate 86.00 MiB …)
+
+The speech model was loaded and idle between scenes while the TTS and visual
+models of the same job loaded around it and filled the card. 86 MiB is what a
+resident model needs to run; nothing was holding it, because the only
+reservation in the system is taken at LOAD time and this model was already
+loaded. Every caption in the render lost its word timings.
+
+Two defects, fixed together:
+
+* the reserve for a whisper-family model was hard-coded at 3 GB in the
+  in-process path, while the isolated worker serving the same checkpoint used
+  8 GB and said in its own comment that 3 GB "is about a third of what a
+  verbatim pass actually peaks at". Eviction freed a third of the room needed;
+* nothing re-checked once a model was resident.
+"""
+
+import io
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import pytest
+
+from codai.api import crisperwhisper_worker as W
+from codai.api import stt_backends as S
+
+_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _src(rel):
+    return io.open(os.path.join(_ROOT, rel), encoding="utf-8").read()
+
+
+@pytest.fixture(autouse=True)
+def _clean_measurements(monkeypatch):
+    monkeypatch.setattr(S, "_activation_gb", {})
+    from codai.models.manager import multi_model_manager as m
+    monkeypatch.setattr(m, "_measured_vram_gb", dict(m._measured_vram_gb))
+    yield
+
+
+def test_a_verbatim_model_is_not_sized_at_a_third_of_its_peak():
+    assert S.reserve_gb("nyrahealth/CrisperWhisper", {}, "crisperwhisper") == 8.0
+    assert S.reserve_gb("nyrahealth/CrisperWhisper", {}, "whisper_hf") == 8.0
+    assert S.reserve_gb("facebook/wav2vec2-base-960h", {}, "wav2vec2") == 3.0
+
+
+def test_the_two_ways_of_serving_one_checkpoint_agree_about_its_size():
+    """The same model runs in-process or in an isolated worker. Two different
+    answers means one of the two paths always evicts the wrong amount."""
+    for cfg in ({}, {"used_vram_gb": 5.5}):
+        assert S.reserve_gb("nyrahealth/CrisperWhisper", cfg, "crisperwhisper") == \
+            W.reserved_vram_gb("nyrahealth/CrisperWhisper", cfg)
+
+
+def test_the_operator_outranks_a_measurement_and_a_measurement_the_default():
+    S.record_pass("some/model", 3.0, 4.0)          # weights + per-pass
+    assert S.reserve_gb("some/model", {}, "whisper_hf") == 7.0
+    assert S.reserve_gb("some/model", {"used_vram_gb": 4.0}, "whisper_hf") == 4.0
+
+
+def test_a_measurement_is_a_high_water_mark_not_an_average():
+    """A one-word clip touches a fraction of what a 30 s window does, so an
+    average under-reserves for the longer audio that follows."""
+    S.record_pass("some/model", 3.0, 4.0)
+    S.record_pass("some/model", 3.0, 0.2)
+    assert S.reserve_gb("some/model", {}, "whisper_hf") == 7.0
+    assert S.activation_gb("some/model") == pytest.approx(4.0)
+
+
+def test_the_per_pass_headroom_is_measured_after_the_first_pass():
+    assert S.activation_gb("fresh/model") == S._FIRST_PASS_ACTIVATION_GB
+    assert S._FIRST_PASS_ACTIVATION_GB > 0, (
+        "a zero floor reserves nothing, which is the bug being fixed")
+    S.record_pass("fresh/model", 3.2, 4.2)
+    assert S.activation_gb("fresh/model") == pytest.approx(4.2)
+
+
+def test_a_negative_delta_reports_no_headroom():
+    """Another model freeing memory during the pass can make the delta
+    negative. Reserved as a negative number it would silently disable the
+    check."""
+    S.record_pass("odd/model", 9.0, -4.0)
+    assert S.activation_gb("odd/model") == S._FIRST_PASS_ACTIVATION_GB
+    assert S.reserve_gb("odd/model", {}, "whisper_hf") == 9.0
+
+
+def test_the_pass_is_charged_its_own_delta_not_the_whole_card():
+    """The absolute process peak would charge a speech model for every other
+    model resident beside it -- a 16 GB image model on the same card would be
+    recorded as part of its footprint, and every later load would evict the
+    world."""
+    src = _src("codai/api/stt_backends.py")
+    body = src.split("class _Wav2Vec2Backend", 1)[1].split("class _VoskBackend", 1)[0]
+    assert "memory_allocated() / 1e9" in body, "nothing records what was held"
+    assert "max_memory_allocated() / 1e9 - _held" in body, (
+        "the pass is measured as the process peak, not as its own delta")
+
+
+# --------------------------------------------------------------------------
+# The manager's side
+# --------------------------------------------------------------------------
+
+def test_room_is_only_made_when_there_is_not_enough(monkeypatch):
+    from codai.models.manager import multi_model_manager as m
+    calls = []
+    monkeypatch.setattr(m, "_evict_models_for_vram", lambda gb: calls.append(gb))
+
+    monkeypatch.setattr(m, "_get_free_vram_gb", lambda: 10.0)
+    m.ensure_free_vram(4.0, reason="a pass")
+    assert calls == [], "it evicted with room to spare"
+
+    monkeypatch.setattr(m, "_get_free_vram_gb", lambda: 0.3)
+    m.ensure_free_vram(4.0, reason="a pass")
+    assert calls == [4.0]
+
+
+def test_asking_for_nothing_evicts_nothing(monkeypatch):
+    from codai.models.manager import multi_model_manager as m
+    calls = []
+    monkeypatch.setattr(m, "_evict_models_for_vram", lambda gb: calls.append(gb))
+    monkeypatch.setattr(m, "_get_free_vram_gb", lambda: 0.0)
+    m.ensure_free_vram(0.0)
+    m.ensure_free_vram(-1.0)
+    assert calls == []
+
+
+# --------------------------------------------------------------------------
+# The wiring
+# --------------------------------------------------------------------------
+
+def test_both_transcription_paths_reserve_before_the_pass():
+    src = _src("codai/api/stt_backends.py")
+    in_process = src.split("class _Wav2Vec2Backend", 1)[1].split("class _VoskBackend", 1)[0]
+    remote = src.split("class _RemoteCrisperWhisperBackend", 1)[1]
+    for name, body in (("in-process", in_process), ("worker", remote)):
+        assert "ensure_free_vram" in body, (
+            f"the {name} path runs a pass without asking for room")
+        assert "activation_gb(" in body, (
+            f"the {name} path does not use the measured per-pass headroom")
+
+
+def test_an_oom_mid_pass_is_retried_once_after_freeing():
+    """A pass is seconds of work on audio that already exists. Losing the
+    render's captions to save that retry is the worse trade."""
+    src = _src("codai/api/stt_backends.py")
+    for marker in ("out of memory", "_evict_models_for_vram"):
+        assert src.count(marker) >= 2, (
+            f"{marker!r} appears {src.count(marker)}x — both the in-process and "
+            f"the worker path need the retry")
+
+
+def test_the_entry_point_no_longer_carries_its_own_number():
+    src = _src("codai/api/transcriptions.py")
+    assert "stt_backends.reserve_gb(" in src
+    assert "3.0 if family == 'whisper_hf'" not in src, (
+        "the hard-coded third-of-the-peak reserve is back")
+
+
+def test_the_measurement_is_fed_back_from_both_paths():
+    src = _src("codai/api/stt_backends.py")
+    assert "record_pass(" in src, "an in-process pass measures nothing"
+    assert "crisperwhisper_worker.record_measurement(" in src, (
+        "the worker's own peak is not recorded")

tests/test_tts_voice_listing.py

@@ -136,3 +136,63 @@ def test_the_capabilities_route_does_not_collide_with_voiceprints():
         f"the TTS voice list is being served from the speaker-recognition "
         f"path; routes declared here: {routes}")
     assert "/v1/audio/speech/capabilities" in routes
+
+
+# --------------------------------------------------------------------------
+# Whether the engine can answer at all
+# --------------------------------------------------------------------------
+#
+# "Speech engines listed but not installed": XTTS answered 501 and parler 500,
+# and a client had no way to know before it asked. Both build their venv on
+# first use now, so the honest answer is not "unavailable" but "the first
+# request will spend minutes building it" -- which a client can act on.
+
+
+def test_an_engine_that_needs_no_venv_is_always_ready():
+    for name in ("hexgrad/Kokoro-82M", "suno/bark"):
+        got = T.environment_report(name)
+        assert got == {"needs_venv": False, "env_ready": True, "env_detail": ""}, name
+
+
+def test_a_missing_venv_is_reported_as_not_ready(tmp_path, monkeypatch):
+    monkeypatch.setattr(T, "_family_venv_dir", lambda fam: tmp_path / "nothing_here")
+    got = T.environment_report("coqui/XTTS-v2")
+    assert got["needs_venv"] is True
+    assert got["env_ready"] is False
+    assert "does not exist" in got["env_detail"]
+
+
+def test_a_venv_without_the_package_is_not_ready(tmp_path, monkeypatch):
+    """The venv builds fine and the install is what failed -- the state a
+    half-finished first request leaves behind."""
+    from codai.api import venv_env
+    venv = tmp_path / "xtts_venv"
+    venv_env.create_venv(venv)
+    monkeypatch.setattr(T, "_family_venv_dir", lambda fam: venv)
+    got = T.environment_report("coqui/XTTS-v2")
+    assert got["env_ready"] is False
+    assert "TTS is not installed" in got["env_detail"]
+
+
+def test_a_populated_venv_is_ready_without_importing_anything(tmp_path, monkeypatch):
+    from codai.api import venv_env
+    venv = tmp_path / "xtts_venv"
+    venv_env.create_venv(venv)
+    site = next(iter(venv.glob("lib/python*/site-packages")))
+    (site / "TTS").mkdir()
+    monkeypatch.setattr(T, "_family_venv_dir", lambda fam: venv)
+
+    def _no_imports(*a, **k):
+        raise AssertionError("the readiness check imported into the venv")
+
+    monkeypatch.setattr(venv_env, "run", _no_imports)
+    got = T.environment_report("coqui/XTTS-v2")
+    assert got["env_ready"] is True
+
+
+def test_the_capabilities_route_reports_it():
+    import os
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    src = open(os.path.join(root, "codai/api/tts.py"), encoding="utf-8").read()
+    route = src.split("async def speech_capabilities", 1)[1].split("@router", 1)[0]
+    assert "environment_report" in route