packaging: build a profile's extra venvs generically, and make audiocraft's installable

The audio build failed with 'No matching distribution found for torch==2.1.0':
audiocraft's pin is unsatisfiable on Python 3.12 at all, because torch support
for 3.12 starts at 2.2. So the venv uses the oldest torch this Python can run
and installs audiocraft --no-deps, with the deps it actually uses listed
explicitly. --no-deps is reckless in a shared venv and routine in an isolated
one: nothing in there can reach the server's stack, and the build's import check
fails the image if the version judgement turns out to be wrong.

The step is now generic — any <profile>.venv-<name>.txt builds
/opt/coderai/venvs/<name>, with an optional .nodeps beside it — rather than
hardcoding audiocraft into the Dockerfile. The next library that pins an
impossible torch will not need a new build step.
Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
parent 1cf69b3c
...@@ -57,12 +57,18 @@ RUN --mount=type=cache,target=/root/.cache/pip set -eux; \ ...@@ -57,12 +57,18 @@ RUN --mount=type=cache,target=/root/.cache/pip set -eux; \
ARG PROFILE ARG PROFILE
COPY packaging/runpod/profiles/ /tmp/venv-profiles/ COPY packaging/runpod/profiles/ /tmp/venv-profiles/
RUN set -eux; \ RUN set -eux; \
extra="/tmp/venv-profiles/${PROFILE}.venv-audiocraft.txt"; \ for extra in /tmp/venv-profiles/${PROFILE}.venv-*.txt; do \
if [ -f "$extra" ]; then \ [ -f "$extra" ] || continue; \
python -m venv /opt/coderai/venvs/audiocraft; \ name="$(basename "$extra" .txt)"; name="${name#${PROFILE}.venv-}"; \
/opt/coderai/venvs/audiocraft/bin/pip install --no-cache-dir -r "$extra"; \ vdir="/opt/coderai/venvs/${name}"; \
/opt/coderai/venvs/audiocraft/bin/python -c "import audiocraft; print('audiocraft', audiocraft.__version__)"; \ python -m venv "$vdir"; \
fi; \ "$vdir/bin/pip" install --no-cache-dir -r "$extra"; \
nodeps="${extra%.txt}.nodeps"; \
if [ -f "$nodeps" ]; then \
"$vdir/bin/pip" install --no-cache-dir --no-deps -r "$nodeps"; \
fi; \
"$vdir/bin/python" -c "import ${name}; print('${name}', getattr(${name}, '__version__', 'ok'), 'in its own venv')"; \
done; \
rm -rf /tmp/venv-profiles rm -rf /tmp/venv-profiles
COPY . /opt/coderai/app COPY . /opt/coderai/app
......
# Installed with --no-deps into the venv described by the .txt beside this file.
# audiocraft's declared pins (torch==2.1.0, torchvision==0.16.0,
# torchtext==0.16.0) cannot all be satisfied on this Python, and its inference
# code does not need those exact versions — the pin is the library being
# conservative. The import check in the build fails the image if that judgement
# turns out to be wrong.
audiocraft==1.3.0
# A SECOND venv inside the audio image, for audiocraft alone. # A SECOND venv inside the audio image, for audiocraft alone.
# #
# audiocraft pins torch==2.1.0, torchvision==0.16.0 and torchtext==0.16.0. The # audiocraft pins torch==2.1.0, torchvision==0.16.0 and torchtext==0.16.0. Two
# server runs torch 2.11 built for a specific CUDA, and torch 2.1.0 has no build # separate problems with that:
# for that CUDA at all — so installing audiocraft beside the server does not
# merely risk a conflict, it leaves the pod with no working GPU torch. Its own
# venv is the only way both can be right.
# #
# Baked at build time, not on first use: a pod is disposable and rented by the # 1. torch 2.1.0 has no build for Python 3.12 at all — support starts at 2.2 —
# second, so paying minutes of GPU rental to pip-install a library on every cold # so the pin is unsatisfiable in this image whatever else we do.
# start would cost more than the image layer does. # 2. Even if it existed, installing it into the server's venv would downgrade
# torch five majors and leave the pod with no working GPU torch, because
# 2.1.0 has no build for this image's CUDA either.
#
# So: its own venv, on the oldest torch this Python can run, with audiocraft
# installed --no-deps (see the .nodeps file beside this one) and the deps it
# actually uses listed here. --no-deps is reckless in a shared venv and routine
# in an isolated one — nothing here can reach the server's stack.
--extra-index-url https://download.pytorch.org/whl/cu121 --extra-index-url https://download.pytorch.org/whl/cu121
torch==2.1.0 torch==2.2.2
torchaudio==2.1.0 torchaudio==2.2.2
audiocraft==1.3.0 torchvision==0.17.2
# audiocraft's real runtime dependencies. Its dev-only ones (pytest, flake8,
# mypy, coverage, pdoc3) and gradio are deliberately absent: a pod serves an
# API, it does not run the library's test suite or its demo UI. torchtext is
# absent too — it is abandoned, unbuildable here, and unused at inference.
av>=11.0.0
einops
flashy>=0.0.1
hydra-core>=1.1
hydra_colorlog
julius
num2words
sentencepiece
spacy>=3.6.1
huggingface_hub
tqdm
librosa
torchmetrics
encodec
protobuf
numpy<2
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment