-
Stefy Lanza (nextime / spora ) authored
Two production failures from the same render, both fixed at the level they belong to. **Speech engines "listed but not installed".** XTTS answered 501 "no venv containing it was found" against a venv that was there and populated; parler answered 500 "python -m venv returned non-zero exit status 1"; a worker's torch died with CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED. Three different failures, one cause: the engine processes export PYTHONHOME=/opt/coderai/python, every worker was launched with env=dict(os.environ), and a venv interpreter started under a foreign PYTHONHOME takes its stdlib and site-packages from THAT installation — so the venv's own packages are not on sys.path at all. The import probe was the first casualty, which is why it reported the venv as missing. codai/api/venv_env.py — which already existed for exactly this, and which the TTS/STT workers were not using — is now the one place a worker's environment and its venv are decided. Every call INTO a venv goes through it: the probe, `python -m venv`, each pip install and the service itself (a clean service with a poisoned probe still reports the venv as broken; a poisoned pip installs into the wrong tree). LD_LIBRARY_PATH is decided per venv, because a foreign python must not see coderai's library directories while a venv built from the server's own interpreter needs its libpython — and sys.prefix no longer counts as "ours" when it IS /usr, which on a pip install would strip the driver path from every worker. venv_usable() replaces four private copies of "can this venv be used". It catches what none of them did: a venv whose python runs perfectly and imports nothing, because its 875 MB of packages are installed for a different minor version (an xtts_venv built by the host's 3.13 pointing at the container's 3.12). `pyvenv.cfg` spells the version two ways; both are read. An intentional overlay (--system-site-packages, for the H3 venv that inherits the server's torch) is respected in both directions. And `GET /v1/audio/speech/capabilities` now reports needs_venv / env_ready / env_detail, from the filesystem alone — never importing into the venv, never loading a model — so a client knows before it offers a voice whether the first request will spend minutes building the environment. **"CrisperWhisper out of GPU memory during compose", every scene.** word timestamps failed (500: … CUDA out of memory. Tried to allocate 86.00 MiB …) The speech model was loaded and idle between scenes while the TTS and visual models of the same job loaded around it and filled the card. 86 MiB is what a resident model needs to run one window, and nothing in the system was keeping it: every reservation is taken at LOAD time, and this model was already loaded. Every caption in that render lost its word timings. Now three numbers per speech model, all measured: the weights, what one pass needs on top of what was already allocated, and their sum. The per-pass number is ensured free before each pass (manager.ensure_free_vram) and is a DELTA, not the process peak — the peak would charge a speech model for a 16 GB image model resident beside it, and every later load of it would evict the world. An OOM that still happens frees the card and retries once, on both the in-process and the isolated-worker path. The in-process reserve was also a third of the truth: 3 GB hard-coded for a whisper-family model, where the isolated worker serving the same checkpoint used 8 GB and said in its own comment that 3 GB was about a third of the real peak. Both paths now call one function, so eviction frees the same amount whichever way the model is running. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
3d507226