lipsync: keep the managed fallback out of the packaged install's directory

Correction to the previous commit. The OCI image already ships a complete
lip-sync integration — packaging/linux/launcher/{wav2lip,sadtalker} shims on
PATH, a dedicated lipsync_venv, baked Wav2Lip AND SadTalker source, first-use
checkpoint download and a smoke test. The claim that the binary "essentially
never exists" was wrong for the packaged deployment; _apply_lipsync() uses those
shims and never reaches the managed path there.

The managed install is therefore only for from-source installs (build.sh +
venv_all), where nothing is on PATH and lip sync did silently degrade.

More importantly it had claimed CODERAI_WAV2LIP_DIR — the same variable the shim
uses for its writable working copy, seeded from the baked read-only source and
run against the lipsync venv's pinned deps. Applying our numpy/librosa/torch
rewrites to that tree would have corrupted a working packaged install. The
managed copy now lives under CODERAI_WAV2LIP_MANAGED_DIR
(~/.coderai/wav2lip-managed), whose basename also cannot collide with the shim's
~/.coderai/Wav2Lip on a case-insensitive filesystem.

Re-verified after the move: fetch/patch/run still produces a valid synced mp4.
Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
parent 958d2e09
...@@ -114,7 +114,7 @@ Full guide: [`docs/runpod.md`](docs/runpod.md). ...@@ -114,7 +114,7 @@ Full guide: [`docs/runpod.md`](docs/runpod.md).
- **Character Profiles**: Apply up to 6 saved character profiles for visual consistency. Identity is conditioned via IP-Adapter on pipelines that support it (AnimateDiff and other SD-based video models); pipelines without IP-Adapter (Wan, CogVideoX, LTX, SVD) fall back to the prompt hint plus any per-character LoRA, and say so in the log - **Character Profiles**: Apply up to 6 saved character profiles for visual consistency. Identity is conditioned via IP-Adapter on pipelines that support it (AnimateDiff and other SD-based video models); pipelines without IP-Adapter (Wan, CogVideoX, LTX, SVD) fall back to the prompt hint plus any per-character LoRA, and say so in the log
- **Environment Profiles**: Apply up to 6 saved environment profiles to condition the scene - **Environment Profiles**: Apply up to 6 saved environment profiles to condition the scene
- **Multi-Character Dialog**: Assign spoken lines to individual characters — each line picks a character profile, voice profile or TTS voice ID, and text; lines are **voice-cloned with F5-TTS** from the named profile, mixed with correct timing (sequential or manual), and lip-synced via Wav2Lip - **Multi-Character Dialog**: Assign spoken lines to individual characters — each line picks a character profile, voice profile or TTS voice ID, and text; lines are **voice-cloned with F5-TTS** from the named profile, mixed with correct timing (sequential or manual), and lip-synced via Wav2Lip
- **Lip Sync**: Wav2Lip is a **managed install** — the code and weights are fetched and patched on first use, so no manual setup. If it can't run, the response carries a `warnings` entry saying the audio was muxed without mouth sync rather than passing off an unsynced video as a success - **Lip Sync**: Wav2Lip and SadTalker. The Docker/OCI image ships both (launcher shims plus a dedicated lip-sync venv; checkpoints download on first use). A from-source install has no shims, so CoderAI falls back to a self-managed Wav2Lip that fetches and patches itself on first use. Either way, if lip sync can't run the response carries a `warnings` entry saying the audio was muxed without mouth sync — rather than passing off an unsynced video as a success
### Audio ### Audio
- **Text-to-Speech**: Kokoro TTS with voice selection and speed control - **Text-to-Speech**: Kokoro TTS with voice selection and speed control
......
...@@ -6,18 +6,24 @@ ...@@ -6,18 +6,24 @@
# the Free Software Foundation, either version 3 of the License, or # the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version. # (at your option) any later version.
"""Managed Wav2Lip lip-sync. """Self-contained Wav2Lip lip-sync for installs that don't ship the packaged shims.
Wav2Lip has no pip package and no installable binary; upstream expects you to clone **The OCI image already handles lip sync properly** — ``packaging/linux/launcher/``
the repo and hunt down checkpoints whose original links have long since rotted. So installs ``wav2lip`` and ``sadtalker`` shims on PATH which run the baked repo code in a
this module owns the whole lifecycle the way :mod:`codai.api.ds4_worker` owns ds4: dedicated ``lipsync_venv`` and download the checkpoints on first use. When those shims
it fetches the code *and* the weights from a HuggingFace mirror on first use, patches are present :func:`codai.api.video._apply_lipsync` uses them and never reaches this
the parts that no longer run on modern numpy/librosa/torch, and drives ``inference.py`` module.
as a subprocess.
This module covers the *other* case: a from-source install (``build.sh`` + ``venv_all``)
That replaces the previous behaviour, which looked for a ``wav2lip`` executable on where nothing is on PATH, which is where lip sync silently degraded to a plain audio mux.
PATH — a file that essentially never exists — and silently returned an unsynchronised It owns the lifecycle the way :mod:`codai.api.ds4_worker` owns ds4 — fetching code *and*
video when it was missing. weights from a HuggingFace mirror on first use, patching what no longer runs on modern
numpy/librosa/torch, and driving ``inference.py`` as a subprocess.
It deliberately uses its **own** directory (``CODERAI_WAV2LIP_MANAGED_DIR``) rather than
the packaged ``CODERAI_WAV2LIP_DIR``: that one is the shim's writable working copy, seeded
from the baked read-only source and run against the lipsync venv's pinned dependencies.
Patching it from here would corrupt a working packaged install.
""" """
import os import os
...@@ -43,8 +49,11 @@ _ready: Optional[Tuple[Path, Path]] = None ...@@ -43,8 +49,11 @@ _ready: Optional[Tuple[Path, Path]] = None
def default_install_dir() -> Path: def default_install_dir() -> Path:
return Path(os.environ.get("CODERAI_WAV2LIP_DIR") # NOT CODERAI_WAV2LIP_DIR — that belongs to the packaged launcher shim. The
or os.path.expanduser("~/.coderai/wav2lip")) # distinct basename also avoids colliding with the shim's ~/.coderai/Wav2Lip on a
# case-insensitive filesystem.
return Path(os.environ.get("CODERAI_WAV2LIP_MANAGED_DIR")
or os.path.expanduser("~/.coderai/wav2lip-managed"))
# --------------------------------------------------------------------------- # # --------------------------------------------------------------------------- #
...@@ -131,7 +140,7 @@ def ensure_wav2lip(install_dir: Optional[Path] = None) -> Tuple[Path, Path]: ...@@ -131,7 +140,7 @@ def ensure_wav2lip(install_dir: Optional[Path] = None) -> Tuple[Path, Path]:
raise RuntimeError( raise RuntimeError(
f"could not download Wav2Lip assets from {WAV2LIP_REPO}: {exc}. " f"could not download Wav2Lip assets from {WAV2LIP_REPO}: {exc}. "
f"Place the repo and checkpoints/wav2lip_gan.pth in {root} manually, " f"Place the repo and checkpoints/wav2lip_gan.pth in {root} manually, "
f"or set CODERAI_WAV2LIP_DIR.") from exc f"or set CODERAI_WAV2LIP_MANAGED_DIR.") from exc
if not (root / "inference.py").exists(): if not (root / "inference.py").exists():
raise RuntimeError(f"Wav2Lip inference.py missing under {root}") raise RuntimeError(f"Wav2Lip inference.py missing under {root}")
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment