• Stefy Lanza (nextime / spora )'s avatar
    give an isolated worker the environment it needs, and a resident speech model room to run · 3d507226
    Stefy Lanza (nextime / spora ) authored
    Two production failures from the same render, both fixed at the level they
    belong to.
    
    **Speech engines "listed but not installed".** XTTS answered 501 "no venv
    containing it was found" against a venv that was there and populated; parler
    answered 500 "python -m venv returned non-zero exit status 1"; a worker's
    torch died with CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED. Three different
    failures, one cause: the engine processes export
    PYTHONHOME=/opt/coderai/python, every worker was launched with
    env=dict(os.environ), and a venv interpreter started under a foreign
    PYTHONHOME takes its stdlib and site-packages from THAT installation — so the
    venv's own packages are not on sys.path at all. The import probe was the first
    casualty, which is why it reported the venv as missing.
    
    codai/api/venv_env.py — which already existed for exactly this, and which the
    TTS/STT workers were not using — is now the one place a worker's environment
    and its venv are decided. Every call INTO a venv goes through it: the probe,
    `python -m venv`, each pip install and the service itself (a clean service with
    a poisoned probe still reports the venv as broken; a poisoned pip installs into
    the wrong tree). LD_LIBRARY_PATH is decided per venv, because a foreign python
    must not see coderai's library directories while a venv built from the
    server's own interpreter needs its libpython — and sys.prefix no longer counts
    as "ours" when it IS /usr, which on a pip install would strip the driver path
    from every worker.
    
    venv_usable() replaces four private copies of "can this venv be used". It
    catches what none of them did: a venv whose python runs perfectly and imports
    nothing, because its 875 MB of packages are installed for a different minor
    version (an xtts_venv built by the host's 3.13 pointing at the container's
    3.12). `pyvenv.cfg` spells the version two ways; both are read. An intentional
    overlay (--system-site-packages, for the H3 venv that inherits the server's
    torch) is respected in both directions.
    
    And `GET /v1/audio/speech/capabilities` now reports needs_venv / env_ready /
    env_detail, from the filesystem alone — never importing into the venv, never
    loading a model — so a client knows before it offers a voice whether the first
    request will spend minutes building the environment.
    
    **"CrisperWhisper out of GPU memory during compose", every scene.**
    
        word timestamps failed (500: … CUDA out of memory.
        Tried to allocate 86.00 MiB …)
    
    The speech model was loaded and idle between scenes while the TTS and visual
    models of the same job loaded around it and filled the card. 86 MiB is what a
    resident model needs to run one window, and nothing in the system was keeping
    it: every reservation is taken at LOAD time, and this model was already
    loaded. Every caption in that render lost its word timings.
    
    Now three numbers per speech model, all measured: the weights, what one pass
    needs on top of what was already allocated, and their sum. The per-pass
    number is ensured free before each pass (manager.ensure_free_vram) and is a
    DELTA, not the process peak — the peak would charge a speech model for a 16 GB
    image model resident beside it, and every later load of it would evict the
    world. An OOM that still happens frees the card and retries once, on both the
    in-process and the isolated-worker path.
    
    The in-process reserve was also a third of the truth: 3 GB hard-coded for a
    whisper-family model, where the isolated worker serving the same checkpoint
    used 8 GB and said in its own comment that 3 GB was about a third of the real
    peak. Both paths now call one function, so eviction frees the same amount
    whichever way the model is running.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    3d507226
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-acestep.txt Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-latentsync.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-musetalk.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...