video: real IP-Adapter conditioning and a managed Wav2Lip install

The last two gaps from the person-to-scene audit.

IP-Adapter on the video path never worked. _apply_character_refs() set
kw['ip_adapter_image'] without ever calling load_ip_adapter(), so a pipeline that
does accept the kwarg (AnimateDiff) failed inside diffusers rather than
transferring identity, and it also passed ip_adapter_scale as a call kwarg, which
no pipeline accepts — a second latent TypeError. It now reuses the image path's
_ensure_ip_adapter_loaded(), sets the scale through set_ip_adapter_scale(), and
returns whether conditioning is actually active. Pipelines with no IP-Adapter at
all (Wan/CogVideoX/LTX/SVD) now say so instead of looking like they worked.

Lip sync required a 'wav2lip' or 'sadtalker' executable on PATH — a file that
essentially never exists, since Wav2Lip ships neither a pip package nor a binary
and its original checkpoint links are dead. codai/api/lipsync.py now owns the
lifecycle the way ds4_worker owns ds4: it fetches code AND weights from the
camenduru/Wav2Lip mirror on first use, patches what no longer runs on modern
numpy/librosa/torch, creates the scratch dirs inference.py expects, and drives it
as a subprocess. An operator-installed binary still takes precedence.

Each patched file is compiled before being kept, so a bad rewrite is caught at
patch time and reverted rather than surfacing as a SyntaxError from a subprocess
(which is exactly how the first attempt failed: two librosa patterns compounded
into 'sr=sr=').

Lip-sync fallback is no longer silent. The reason travels up a per-request
warnings list — not module state, which would cross-talk between concurrent
requests — and appears as a `warnings` field on the video response, so a client can tell
a synchronised result from a plain audio mux.

Verified offline (live container untouched): Wav2Lip downloads, patches, and
lip-syncs a real face to audio, producing a valid 768x512 h264+aac mp4 that
differs from the source; the second run reuses the install with no download;
the fallback populates warnings; IP-Adapter loads its weights and sets the scale
for a supporting pipeline and cleanly declines for a Wan-like one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4

README.md

@@ -111,9 +111,10 @@ Full guide: [`docs/runpod.md`](docs/runpod.md).
 - **Upscaling**: Real-ESRGAN video upscaling
 - **Subtitles**: Whisper transcription + optional translation + burn-in
 - **Dubbing**: Transcribe → translate → TTS → replace audio track
-- **Character Profiles**: Apply up to 6 saved character profiles for visual consistency
+- **Character Profiles**: Apply up to 6 saved character profiles for visual consistency. Identity is conditioned via IP-Adapter on pipelines that support it (AnimateDiff and other SD-based video models); pipelines without IP-Adapter (Wan, CogVideoX, LTX, SVD) fall back to the prompt hint plus any per-character LoRA, and say so in the log
 - **Environment Profiles**: Apply up to 6 saved environment profiles to condition the scene
-- **Multi-Character Dialog**: Assign spoken lines to individual characters — each line picks a character profile, voice profile or TTS voice ID, and text; lines are TTS-synthesised, mixed with correct timing (sequential or manual), and lip-synced to the video via Wav2Lip or SadTalker
+- **Multi-Character Dialog**: Assign spoken lines to individual characters — each line picks a character profile, voice profile or TTS voice ID, and text; lines are **voice-cloned with F5-TTS** from the named profile, mixed with correct timing (sequential or manual), and lip-synced via Wav2Lip
+- **Lip Sync**: Wav2Lip is a **managed install** — the code and weights are fetched and patched on first use, so no manual setup. If it can't run, the response carries a `warnings` entry saying the audio was muxed without mouth sync rather than passing off an unsynced video as a success
 
 ### Audio
 - **Text-to-Speech**: Kokoro TTS with voice selection and speed control

codai/api/lipsync.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+"""Managed Wav2Lip lip-sync.
+
+Wav2Lip has no pip package and no installable binary; upstream expects you to clone
+the repo and hunt down checkpoints whose original links have long since rotted. So
+this module owns the whole lifecycle the way :mod:`codai.api.ds4_worker` owns ds4:
+it fetches the code *and* the weights from a HuggingFace mirror on first use, patches
+the parts that no longer run on modern numpy/librosa/torch, and drives ``inference.py``
+as a subprocess.
+
+That replaces the previous behaviour, which looked for a ``wav2lip`` executable on
+PATH — a file that essentially never exists — and silently returned an unsynchronised
+video when it was missing.
+"""
+
+import os
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Optional, Tuple
+
+#: Mirror carrying the Wav2Lip source tree, the s3fd face detector and the
+#: wav2lip/wav2lip_gan checkpoints in one repo.
+WAV2LIP_REPO = "camenduru/Wav2Lip"
+
+#: Only what inference needs — not the training scripts or the eval corpus.
+_ALLOW = [
+    "inference.py", "audio.py", "hparams.py",
+    "models/*", "face_detection/*",
+    "checkpoints/wav2lip_gan.pth",
+]
+
+_ready: Optional[Tuple[Path, Path]] = None
+
+
+def default_install_dir() -> Path:
+    return Path(os.environ.get("CODERAI_WAV2LIP_DIR")
+                or os.path.expanduser("~/.coderai/wav2lip"))
+
+
+# --------------------------------------------------------------------------- #
+# Compatibility patches
+# --------------------------------------------------------------------------- #
+# Wav2Lip was last updated against numpy 1.x, librosa 0.8 and torch 1.x. Each of
+# these is a hard failure on a current stack, and all of them are mechanical.
+_PATCHES = (
+    # numpy >= 1.24 removed the aliases entirely.
+    (r"\bnp\.float\b(?!\d|_)", "float"),
+    (r"\bnp\.int\b(?!\d|_|p|e)", "int"),
+    (r"\bnp\.bool\b(?!\d|_)", "bool"),
+    # librosa >= 0.10 made the mel arguments keyword-only. The `[^,()=]` classes
+    # exclude '=', so an argument that is already in keyword form cannot match a
+    # second time — without that guard the rewrite compounds into `sr=sr=...`.
+    (r"librosa\.filters\.mel\(\s*([^,()=]+?)\s*,\s*([^,()=]+?)\s*,",
+     r"librosa.filters.mel(sr=\1, n_fft=\2,"),
+    (r"librosa\.core\.stft\(", "librosa.stft("),
+    # torch >= 2.6 defaults weights_only=True, which refuses these checkpoints.
+    (r"torch\.load\((?![^()]*weights_only)([^()]*)\)", r"torch.load(\1, weights_only=False)"),
+)
+
+
+def _patch_tree(root: Path) -> None:
+    """Apply the compatibility rewrites in place.
+
+    Every rewritten file is compiled before it is kept: a pattern that produces
+    invalid Python is reverted and reported, so a bad rewrite surfaces here instead
+    of as a SyntaxError from a subprocess much later.
+    """
+    stamp = root / ".coderai-patched"
+    if stamp.exists():
+        return
+    failed = []
+    for path in list(root.rglob("*.py")):
+        try:
+            src = path.read_text(encoding="utf-8")
+        except (UnicodeDecodeError, OSError):
+            continue
+        out = src
+        for pattern, repl in _PATCHES:
+            out = re.sub(pattern, repl, out)
+        if out == src:
+            continue
+        rel = path.relative_to(root)
+        try:
+            compile(out, str(path), "exec")
+        except SyntaxError as exc:
+            failed.append(f"{rel}: {exc}")
+            continue          # leave the original in place
+        path.write_text(out, encoding="utf-8")
+        print(f"[wav2lip] patched {rel}", flush=True)
+    if failed:
+        raise RuntimeError("Wav2Lip compatibility patching produced invalid Python: "
+                           + "; ".join(failed))
+    stamp.write_text("ok\n", encoding="utf-8")
+
+
+def ensure_wav2lip(install_dir: Optional[Path] = None) -> Tuple[Path, Path]:
+    """Return ``(repo_dir, checkpoint)``, downloading and patching on first use.
+
+    Raises RuntimeError with an actionable message when the assets can't be obtained.
+    """
+    global _ready
+    if _ready is not None:
+        return _ready
+
+    root = Path(install_dir) if install_dir else default_install_dir()
+    ckpt = root / "checkpoints" / "wav2lip_gan.pth"
+
+    if not (root / "inference.py").exists() or not ckpt.exists():
+        try:
+            from huggingface_hub import snapshot_download
+        except ImportError as exc:
+            raise RuntimeError(
+                "lip sync needs huggingface_hub to fetch the Wav2Lip assets") from exc
+        root.mkdir(parents=True, exist_ok=True)
+        print(f"[wav2lip] fetching {WAV2LIP_REPO} into {root} (first use only)…",
+              flush=True)
+        try:
+            snapshot_download(repo_id=WAV2LIP_REPO, allow_patterns=_ALLOW,
+                              local_dir=str(root))
+        except Exception as exc:
+            raise RuntimeError(
+                f"could not download Wav2Lip assets from {WAV2LIP_REPO}: {exc}. "
+                f"Place the repo and checkpoints/wav2lip_gan.pth in {root} manually, "
+                f"or set CODERAI_WAV2LIP_DIR.") from exc
+
+    if not (root / "inference.py").exists():
+        raise RuntimeError(f"Wav2Lip inference.py missing under {root}")
+    if not ckpt.exists():
+        raise RuntimeError(f"Wav2Lip checkpoint missing at {ckpt}")
+
+    _patch_tree(root)
+    # inference.py writes its intermediate to the relative path temp/result.avi and
+    # then muxes that with ffmpeg. The directory is a .gitignore'd scratch dir, so it
+    # isn't in the mirror — without it the run exits 0 having produced nothing.
+    (root / "temp").mkdir(exist_ok=True)
+    (root / "results").mkdir(exist_ok=True)
+    _ready = (root, ckpt)
+    return _ready
+
+
+def run_wav2lip(video_path: str, audio_path: str, out_path: str,
+                timeout: int = 1800) -> str:
+    """Lip-sync ``video_path`` to ``audio_path``. Returns ``out_path``.
+
+    Raises RuntimeError on failure so the caller can report it instead of quietly
+    handing back an unsynchronised video.
+    """
+    root, ckpt = ensure_wav2lip()
+
+    cmd = [sys.executable, "inference.py",
+           "--checkpoint_path", str(ckpt),
+           "--face", os.path.abspath(video_path),
+           "--audio", os.path.abspath(audio_path),
+           "--outfile", os.path.abspath(out_path)]
+    env = dict(os.environ)
+    # inference.py imports audio/hparams/face_detection as top-level modules.
+    env["PYTHONPATH"] = str(root) + os.pathsep + env.get("PYTHONPATH", "")
+
+    proc = subprocess.run(cmd, cwd=str(root), env=env, capture_output=True,
+                          text=True, timeout=timeout)
+    if proc.returncode != 0 or not os.path.exists(out_path):
+        tail = (proc.stderr or proc.stdout or "").strip()[-800:]
+        raise RuntimeError(f"wav2lip failed (rc={proc.returncode}): {tail}")
+    return out_path
+
+
+def is_available() -> bool:
+    """True when the assets are already present (no download attempted)."""
+    root = default_install_dir()
+    return (root / "inference.py").exists() and \
+           (root / "checkpoints" / "wav2lip_gan.pth").exists()
+
+
+def which_sadtalker() -> Optional[str]:
+    return shutil.which("sadtalker")

codai/api/video.py

@@ -1937,18 +1937,47 @@ def _pipeline_supports_ip_adapter(pipe) -> bool:
 
 
 def _apply_character_refs(kw: dict, character_references: List[str], strength: float,
-                           names: Optional[List[str]] = None, pipe=None):
-    """Apply character reference images to pipeline kwargs when supported."""
+                           names: Optional[List[str]] = None, pipe=None) -> bool:
+    """Condition the pipeline on character reference images via IP-Adapter.
+
+    Returns True when identity conditioning is actually active.
+
+    Two things used to be wrong here. The adapter *weights* were never loaded — only
+    the kwarg was set — so a pipeline that accepts ``ip_adapter_image`` (AnimateDiff)
+    would fail inside diffusers instead of transferring identity. And the scale was
+    passed as a call kwarg, which no pipeline accepts; it goes through
+    ``set_ip_adapter_scale()``.
+
+    Models like WanPipeline / CogVideoX / LTX / SVD have no IP-Adapter at all, so for
+    those the only mechanism remains the character-name prompt hint the caller adds —
+    and we say so out loud rather than looking like it worked.
+    """
     if not character_references:
-        return
-    # Only inject ip_adapter_image if the pipeline actually accepts it.
-    # Models like WanPipeline don't support IP-Adapter; for those we rely
-    # solely on the character-name text prompt hint added by the caller.
-    if pipe is not None and not _pipeline_supports_ip_adapter(pipe):
-        return
-    imgs = [_pil_from_b64(r) for r in character_references]
-    kw['ip_adapter_image'] = imgs[0] if len(imgs) == 1 else imgs
-    kw['ip_adapter_scale'] = strength
+        return False
+    if pipe is None or not _pipeline_supports_ip_adapter(pipe):
+        print(f"Note: {type(pipe).__name__ if pipe else 'pipeline'} has no IP-Adapter "
+              f"support — character identity relies on the prompt hint"
+              + (" and any per-character LoRA" if names else ""), flush=True)
+        return False
+
+    try:
+        from codai.api.images import _ensure_ip_adapter_loaded
+        if not _ensure_ip_adapter_loaded(pipe):
+            print("Note: IP-Adapter weights unavailable for this pipeline — "
+                  "character identity relies on the prompt hint/LoRA", flush=True)
+            return False
+        imgs = [_pil_from_b64(r) for r in character_references]
+        if hasattr(pipe, 'set_ip_adapter_scale'):
+            pipe.set_ip_adapter_scale(strength)
+        kw['ip_adapter_image'] = imgs[0] if len(imgs) == 1 else imgs
+        print(f"IP-Adapter conditioning {len(imgs)} character reference(s) "
+              f"at scale {strength}", flush=True)
+        return True
+    except Exception as exc:
+        print(f"Warning: IP-Adapter injection failed ({exc}) — continuing without "
+              f"character references", flush=True)
+        kw.pop('ip_adapter_image', None)
+        return False
 
 
 def _unload_video_loras(pipe):
@@ -2421,8 +2450,11 @@ def _generate_video(pipe, request: VideoGenerationRequest):
 # =============================================================================
 
 def _postprocess_video(mp4_bytes: bytes, request: VideoGenerationRequest,
-                       http_request, temp_paths: list) -> bytes:
-    """Apply upscale / interpolation / audio / dialog steps to a raw mp4 blob."""
+                       http_request, temp_paths: list,
+                       warnings: Optional[list] = None) -> bytes:
+    """Apply upscale / interpolation / audio / dialog steps to a raw mp4 blob.
+
+    ``warnings`` collects per-request notes (e.g. lip sync fell back) for the response."""
     path = _tmp_write(mp4_bytes, '.mp4')
     temp_paths.append(path)
 
@@ -2449,7 +2481,8 @@ def _postprocess_video(mp4_bytes: bytes, request: VideoGenerationRequest,
 
     if request.dialogs:
         path = _process_dialogs(path, request.dialogs,
-                                request.lip_sync_method or 'wav2lip', temp_paths)
+                                request.lip_sync_method or 'wav2lip', temp_paths,
+                                warnings)
 
     if request.generate_subtitles or request.burn_subtitles:
         path = _add_subtitles(path, request, temp_paths)
@@ -3332,14 +3365,21 @@ def _mix_dialog_audio(clips: list, temps: list) -> Optional[str]:
     return out2 if r2.returncode == 0 else None
 
 
-def _apply_lipsync(video_path: str, audio_path: str, method: str, temps: list) -> str:
-    """Apply lip sync to video using wav2lip or sadtalker. Returns new video path."""
+def _apply_lipsync(video_path: str, audio_path: str, method: str, temps: list,
+                    warnings: Optional[list] = None) -> str:
+    """Apply lip sync to video using wav2lip or sadtalker. Returns new video path.
+
+    On fallback the reason is appended to ``warnings`` (a per-request list) so the
+    caller can tell the client the audio was merely muxed — this used to return 200
+    with an unsynchronised video and no signal at all."""
     import logging, shutil
     _log = logging.getLogger(__name__)
+    failure = ''
     out = tempfile.mktemp(suffix='_lipsync.mp4')
     temps.append(out)
 
     if method == 'wav2lip':
+        # An explicit wav2lip binary still wins if the operator installed one.
         wav2lip_bin = shutil.which('wav2lip') or shutil.which('Wav2Lip')
         if wav2lip_bin:
             cmd = [wav2lip_bin, '--face', video_path, '--audio', audio_path, '--outfile', out]
@@ -3348,14 +3388,16 @@ def _apply_lipsync(video_path: str, audio_path: str, method: str, temps: list) -
                 return out
             _log.warning("wav2lip failed (rc=%d): %s", r.returncode, r.stderr.decode(errors='replace'))
         else:
-            # Try wav2lip Python API
+            # Otherwise use the managed install — code and weights are fetched and
+            # patched on first use, so lip sync works out of the box.
             try:
-                import inference as wav2lip_inference  # noqa
-                wav2lip_inference.main(face=video_path, audio=audio_path, outfile=out)
+                from codai.api.lipsync import run_wav2lip
+                run_wav2lip(video_path, audio_path, out)
                 if os.path.exists(out):
                     return out
             except Exception as e:
-                _log.warning("wav2lip Python API failed: %s", e)
+                _log.warning("wav2lip (managed) failed: %s", e)
+                failure = str(e)
 
     elif method == 'sadtalker':
         sadtalker_bin = shutil.which('sadtalker')
@@ -3372,7 +3414,11 @@ def _apply_lipsync(video_path: str, audio_path: str, method: str, temps: list) -
             _log.warning("sadtalker failed (rc=%d): %s", r.returncode, r.stderr.decode(errors='replace'))
 
     # Fallback: just mux audio onto video without lip sync
-    _log.warning("Lip sync unavailable (%s not found/working), merging audio only", method)
+    failure = failure or f"{method} not available"
+    _log.warning("Lip sync unavailable (%s), merging audio only: %s", method, failure)
+    if warnings is not None:
+        warnings.append(f"lip sync skipped ({method}): {failure} — the audio was "
+                        f"muxed onto the video without mouth synchronisation")
     out_fallback = tempfile.mktemp(suffix='_nosync.mp4')
     temps.append(out_fallback)
     cmd = ['ffmpeg', '-y', '-i', video_path, '-i', audio_path,
@@ -3381,7 +3427,8 @@ def _apply_lipsync(video_path: str, audio_path: str, method: str, temps: list) -
     return out_fallback if r.returncode == 0 else video_path
 
 
-def _process_dialogs(path: str, dialogs: list, lip_sync_method: str, temps: list) -> str:
+def _process_dialogs(path: str, dialogs: list, lip_sync_method: str, temps: list,
+                      warnings: Optional[list] = None) -> str:
     """
     Generate TTS for each dialog line, mix with correct timing, apply lip sync to full video.
     Returns new video path.
@@ -3421,7 +3468,7 @@ def _process_dialogs(path: str, dialogs: list, lip_sync_method: str, temps: list
     wants_lip_sync = any(getattr(line, 'lip_sync', True) for line in dialogs if line.text.strip())
 
     if wants_lip_sync and lip_sync_method:
-        return _apply_lipsync(path, mixed_audio, lip_sync_method, temps)
+        return _apply_lipsync(path, mixed_audio, lip_sync_method, temps, warnings)
 
     # No lip sync — just mux audio
     out = tempfile.mktemp(suffix='_dialog.mp4')
@@ -3919,6 +3966,7 @@ async def video_generations(request: VideoGenerationRequest,
 
     # Post-processing pipeline (upscale, audio, subtitles, …)
     temps = []
+    gen_warnings: list = []
     try:
         needs_post = any([
             request.upscale_output,
@@ -3930,7 +3978,8 @@ async def video_generations(request: VideoGenerationRequest,
         ])
         if needs_post:
             mp4_bytes = await asyncio.get_event_loop().run_in_executor(
-                None, _postprocess_video, mp4_bytes, request, http_request, temps)
+                None, _postprocess_video, mp4_bytes, request, http_request, temps,
+                gen_warnings)
     finally:
         for t in temps:
             try:
@@ -3990,7 +4039,12 @@ async def video_generations(request: VideoGenerationRequest,
     except Exception:
         pass
 
-    return VideoGenerationResponse(created=int(time.time()), data=[result])
+    resp = VideoGenerationResponse(created=int(time.time()), data=[result])
+    if gen_warnings:
+        # e.g. lip sync fell back to a plain audio mux — the client must be able to
+        # tell that apart from a genuinely synchronised result.
+        resp.warnings = gen_warnings
+    return resp
 
 
 # =============================================================================