pin parler's own CUDA stack, and check a venv can run what the worker needs

With the environment fixed, Parler got further: the venv built, the service
started, the model loaded — and the first synthesis died in the DAC audio
decoder's conv1d.

    RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed
    ptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED

Every cudnn sublibrary present in the venv, and the stack unable to load them:
a torch 2.15 with the CUDA-13 wheels that `pip install git+…parler-tts` had
resolved on its own, because nothing pinned it. requirements-parler.txt names
the build now, the way every other isolated worker's file does.

The health check was the deeper problem. `import parler_tts, soundfile`
answered True for that venv for ever, so nothing ever rebuilt it. A venv's
check has to exercise what the worker actually does: the decoder is
convolutional, so the probe is one conv1d on the GPU, and a venv that imports
but cannot run it is rebuilt from the pinned file. A CPU-only host passes —
there is no GPU stack there to be broken.

The service also latches cudnn off if it still fails there. torch has native
kernels for those convolutions, the decoder is a fraction of the work, and the
alternative is no speech at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

README.md

@@ -607,6 +607,7 @@ built on demand from the admin UI:
 | `requirements-nemo.txt` | NVIDIA NeMo Canary/Parakeet STT | yes |
 | `requirements-crisperwhisper.txt` | CrisperWhisper verbatim STT | yes |
 | `requirements-pyannote.txt` | Speaker diarization | yes |
+| `requirements-parler.txt` | Parler-TTS expressive speech | yes |
 
 ---
 

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.18"
+__version__ = "0.5.19"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/parler_worker.py

@@ -28,6 +28,7 @@ from codai.api import venv_env
 
 _REPO_ROOT = Path(__file__).resolve().parents[2]
 _SERVICE_SCRIPT = _REPO_ROOT / "tools" / "parler_tts_service.py"
+_REQUIREMENTS = _REPO_ROOT / "requirements-parler.txt"
 
 # Dedicated venv for the (incompatible) parler-tts stack. Created with access to
 # the base interpreter's packages so torch/numpy aren't re-downloaded; parler's
@@ -53,6 +54,43 @@ def _pip_ok(py: Path) -> bool:
         return False
 
 
+#: The op that failed. Parler's audio decoder (DAC) is convolutional, so a
+#: venv whose CUDA stack cannot run one convolution cannot synthesize anything
+#: -- however importable its packages are.
+_GPU_PROBE = (
+    "import torch, torch.nn.functional as F\n"
+    "if not torch.cuda.is_available():\n"
+    "    raise SystemExit(0)\n"
+    "x = torch.zeros(1, 1, 16, device='cuda')\n"
+    "w = torch.zeros(1, 1, 3, device='cuda')\n"
+    "F.conv1d(x, w).sum().item()\n")
+
+
+def _gpu_ok(py: Path) -> tuple:
+    """``(ok, detail)`` — whether this venv can actually run on the GPU.
+
+    "Are the packages importable" is not the same question, and the difference
+    cost a whole bring-up. Measured on zeiss, 2026-10-29: the venv imported
+    parler_tts and soundfile perfectly, the service started, the model loaded,
+    and the first synthesis died in the decoder's conv1d with
+    ``CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED`` — a CUDA-13 torch that pip had
+    resolved on its own. ``_pip_ok`` answered True for that venv for ever, so
+    nothing ever rebuilt it.
+
+    A CPU-only host passes: it exits before the convolution, because there is
+    no GPU stack to be broken.
+    """
+    try:
+        r = venv_env.run([str(py), "-c", _GPU_PROBE],
+                         capture_output=True, text=True, timeout=600)
+    except Exception as exc:
+        return False, f"the probe could not be run ({exc})"
+    if r.returncode == 0:
+        return True, ""
+    tail = ((r.stderr or "") or (r.stdout or "")).strip().splitlines()
+    return False, (tail[-1][:300] if tail else f"exit {r.returncode}")
+
+
 def _bootstrap_venv() -> Path:
     """Create a fully-isolated venv and install parler-tts (idempotent).
 
@@ -69,14 +107,37 @@ def _bootstrap_venv() -> Path:
     # decision, taken in venv_env.create_venv. None of them can be repaired
     # in place: `python -m venv` over the top dies on the thing that broke it.
     py = venv_env.create_venv(_VENV_DIR)
-    if not _pip_ok(py):
-        print("[parler] installing parler-tts + torch into the isolated venv "
-              "(first run, downloads several GB, this can take a while) …", flush=True)
-        venv_env.run([str(py), "-m", "pip", "install",
-                        "git+https://github.com/huggingface/parler-tts.git",
-                        "soundfile"], check=True)
+    _ok = _pip_ok(py)
+    if _ok:
+        # Importable is not enough: a venv whose CUDA stack cannot run a
+        # convolution synthesizes nothing, and the old check called it healthy
+        # for ever. Rebuilt from the pinned requirements, not from whatever
+        # pip resolves today -- that resolution is what broke it.
+        _ok, _why = _gpu_ok(py)
+        if not _ok:
+            import shutil
+            print(f"[parler] the venv imports but cannot run on the GPU "
+                  f"({_why}); rebuilding it from {_REQUIREMENTS.name}", flush=True)
+            shutil.rmtree(_VENV_DIR, ignore_errors=True)
+            py = venv_env.create_venv(_VENV_DIR)
+    if not _ok:
+        if not _REQUIREMENTS.exists():
+            raise RuntimeError(f"Parler requirements file missing: {_REQUIREMENTS}")
+        print("[parler] installing parler-tts + a pinned torch into the isolated "
+              "venv (first run, downloads several GB, this can take a while) …",
+              flush=True)
+        venv_env.run([str(py), "-m", "pip", "install", "-U", "pip"], check=True)
+        venv_env.run([str(py), "-m", "pip", "install", "-r", str(_REQUIREMENTS)],
+                     check=True)
         if not _pip_ok(py):
             raise RuntimeError("parler-tts install did not yield an importable package")
+        _gpu, _why = _gpu_ok(py)
+        if not _gpu:
+            # Loud, and not fatal: a synthesis will fail with the real error
+            # from the real op, which is more useful than a bootstrap that
+            # refuses to finish on a machine whose CUDA this cannot judge.
+            print(f"[parler] warning: the rebuilt venv still cannot run a "
+                  f"convolution on the GPU ({_why})", flush=True)
     _bootstrapped = True
     return py
 

docs/isolated-worker-environments.md

@@ -89,6 +89,41 @@ venv is never repaired in place. `venv_env.create_venv` rebuilds it, and
   MiniMax-H3 overlay inherits the server's torch *on purpose*, so an isolated
   venv found where an overlay is wanted is also rebuilt.
 
+## Importable is not working
+
+With the environment fixed, Parler got further: the venv built, the service
+started, the model loaded — and the first synthesis died in the audio
+decoder's `conv1d`:
+
+```
+RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed
+ptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED
+```
+
+Every cudnn sublibrary was present in the venv. The stack was a torch 2.15
+with the CUDA-13 wheels, which `pip install git+…parler-tts` had resolved on
+its own, because nothing pinned it. And `import parler_tts, soundfile` — the
+only health check — answered True for that venv for ever, so nothing ever
+rebuilt it.
+
+Two changes:
+
+* **the stack is pinned.** `requirements-parler.txt` names the torch build,
+  the way every other isolated worker's file does. Pip resolving a CUDA stack
+  on its own is how a venv ends up newer than the fleet it runs on.
+* **the health check runs the op that fails.** The decoder is convolutional,
+  so the probe is one `conv1d` on the GPU. A venv that imports but cannot run
+  it is rebuilt from the pinned file; a CPU-only host passes, because there is
+  no GPU stack to be broken.
+
+And the service itself latches cudnn off if it still fails there: torch has
+native kernels for those convolutions, the decoder is a fraction of the work,
+and the alternative is no speech at all.
+
+The general rule, which cost two bring-ups here: **a venv's health check must
+exercise what the worker actually does.** An import proves the files are in
+the right directory and nothing else.
+
 ## Knowing before you ask
 
 A venv is built on first use, and that build downloads several GB. Discovering

requirements-parler.txt

+# Isolated venv for Parler-TTS (parler-tts/parler-tts-*).
+#
+# parler-tts pins an old transformers/tokenizers/huggingface-hub that conflict
+# with the coderai server's 5.x stack, so it runs in its own venv and coderai
+# talks to it over HTTP (codai/api/parler_worker.py +
+# tools/parler_tts_service.py). Do NOT merge these into requirements.txt.
+#
+# torch is PINNED here, which is the point of the file. Installing parler-tts
+# with a bare `pip install` resolved whatever was newest — measured on zeiss,
+# 2026-10-29: torch 2.15 with the CUDA-13 wheels (nvidia_cudnn_cu13 9.26). The
+# venv built, the service started, the model loaded, and the first synthesis
+# died inside the DAC audio decoder's conv1d:
+#
+#   RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed
+#   ptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED
+#
+# — every cudnn sublibrary present in the venv, and the stack unable to load
+# them. Pinning to the same CUDA build that is proven on this fleet (the
+# server runs 2.10+cu128, the CrisperWhisper venv 2.5.1+cu124) takes that
+# resolution out of pip's hands. An operator on a newer CUDA can raise it.
+
+--extra-index-url https://download.pytorch.org/whl/cu124
+torch==2.5.1+cu124
+
+# parler-tts has no release on PyPI that carries the multilingual models, so it
+# comes from the repo. It brings its own transformers pin with it; nothing here
+# overrides that, because that pin is the reason this venv exists.
+git+https://github.com/huggingface/parler-tts.git
+
+soundfile>=0.12.0

tests/test_foreign_venv_rebuild.py

@@ -197,3 +197,63 @@ def test_no_isolated_worker_launches_its_venv_with_a_bare_subprocess():
     assert not offenders, (
         "these calls into a venv inherit the engine's environment: "
         + ", ".join(offenders))
+
+
+# --------------------------------------------------------------------------
+# Importable is not the same as working
+# --------------------------------------------------------------------------
+#
+# Measured on zeiss, 2026-10-29, right after the environment fix above: the
+# parler venv built, the service started, the model loaded, and the first
+# synthesis died in the decoder's conv1d with
+# CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED -- a CUDA-13 torch that pip had
+# resolved on its own, with every cudnn sublibrary present and unloadable.
+# `import parler_tts, soundfile` answered True for that venv for ever, so
+# nothing ever rebuilt it.
+
+
+def test_the_parler_venv_is_probed_on_the_gpu_not_just_imported():
+    import os
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    src = open(os.path.join(root, "codai/api/parler_worker.py"),
+               encoding="utf-8").read()
+    assert "_gpu_ok(" in src, "the venv is trusted on an import alone"
+    assert "conv1d" in src, (
+        "the probe does not run the op that fails (the decoder is convolutional)")
+    boot = src.split("def _bootstrap_venv", 1)[1]
+    assert boot.index("_gpu_ok(py)") < boot.index("rmtree"), (
+        "the probe does not drive the rebuild")
+    assert "requirements-parler.txt" in src or "_REQUIREMENTS" in boot, (
+        "the rebuild installs whatever pip resolves today, which is the bug")
+
+
+def test_a_cpu_only_host_passes_the_probe():
+    """There is no GPU stack to be broken, so the probe must not fail the
+    bootstrap on a CPU-only machine."""
+    import os
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    src = open(os.path.join(root, "codai/api/parler_worker.py"),
+               encoding="utf-8").read()
+    probe = src.split("_GPU_PROBE = (", 1)[1].split(")\n", 1)[0]
+    assert "is_available" in probe and "SystemExit(0)" in probe
+
+
+def test_the_parler_requirements_pin_torch():
+    import os
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    req = open(os.path.join(root, "requirements-parler.txt"), encoding="utf-8").read()
+    assert "torch==" in req, "torch is unpinned, so pip picks the stack again"
+    assert "parler-tts" in req
+
+
+def test_the_service_survives_a_cudnn_that_cannot_load():
+    """Turning cudnn off costs a little speed on a decoder that is a fraction
+    of the work. The alternative is no speech at all."""
+    import os
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    src = open(os.path.join(root, "tools/parler_tts_service.py"),
+               encoding="utf-8").read()
+    body = src.split("def speak", 1)[1]
+    assert "cudnn.enabled = False" in body
+    assert "not torch.backends.cudnn.enabled" in body, (
+        "a second failure would retry for ever instead of raising")

tools/parler_tts_service.py

@@ -76,9 +76,29 @@ class _Engine:
         self.sr = int(self._model.config.sampling_rate)
 
     def speak(self, text: str, description: str) -> np.ndarray:
+        import torch
         ids = self._tok(description, return_tensors="pt").input_ids.to(self._device)
         prompt = self._tok(text, return_tensors="pt").input_ids.to(self._device)
-        gen = self._model.generate(input_ids=ids, prompt_input_ids=prompt)
+        try:
+            gen = self._model.generate(input_ids=ids, prompt_input_ids=prompt)
+        except RuntimeError as exc:
+            # The audio decoder (DAC) is convolutional, and on a venv whose
+            # cudnn cannot load its sublibraries every synthesis dies in that
+            # one conv1d:
+            #
+            #   CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed …
+            #   cudnn_status: CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED
+            #
+            # torch has native kernels for these convolutions, so turning cudnn
+            # off costs a little speed on a decoder that is a fraction of the
+            # work -- and the alternative is no speech at all. Latched, so one
+            # failure does not mean a failed attempt on every later request.
+            if "cudnn" not in str(exc).lower() or not torch.backends.cudnn.enabled:
+                raise
+            print(f"[parler] cudnn failed in the decoder ({str(exc)[:160]}); "
+                  f"disabling cudnn for this worker and retrying", flush=True)
+            torch.backends.cudnn.enabled = False
+            gen = self._model.generate(input_ids=ids, prompt_input_ids=prompt)
         return np.asarray(gen.cpu().numpy().squeeze(), dtype=np.float32)