• Stefy Lanza (nextime / spora )'s avatar
    video: real IP-Adapter conditioning and a managed Wav2Lip install · 958d2e09
    Stefy Lanza (nextime / spora ) authored
    The last two gaps from the person-to-scene audit.
    
    IP-Adapter on the video path never worked. _apply_character_refs() set
    kw['ip_adapter_image'] without ever calling load_ip_adapter(), so a pipeline that
    does accept the kwarg (AnimateDiff) failed inside diffusers rather than
    transferring identity, and it also passed ip_adapter_scale as a call kwarg, which
    no pipeline accepts — a second latent TypeError. It now reuses the image path's
    _ensure_ip_adapter_loaded(), sets the scale through set_ip_adapter_scale(), and
    returns whether conditioning is actually active. Pipelines with no IP-Adapter at
    all (Wan/CogVideoX/LTX/SVD) now say so instead of looking like they worked.
    
    Lip sync required a 'wav2lip' or 'sadtalker' executable on PATH — a file that
    essentially never exists, since Wav2Lip ships neither a pip package nor a binary
    and its original checkpoint links are dead. codai/api/lipsync.py now owns the
    lifecycle the way ds4_worker owns ds4: it fetches code AND weights from the
    camenduru/Wav2Lip mirror on first use, patches what no longer runs on modern
    numpy/librosa/torch, creates the scratch dirs inference.py expects, and drives it
    as a subprocess. An operator-installed binary still takes precedence.
    
    Each patched file is compiled before being kept, so a bad rewrite is caught at
    patch time and reverted rather than surfacing as a SyntaxError from a subprocess
    (which is exactly how the first attempt failed: two librosa patterns compounded
    into 'sr=sr=').
    
    Lip-sync fallback is no longer silent. The reason travels up a per-request
    warnings list — not module state, which would cross-talk between concurrent
    requests — and appears as a `warnings` field on the video response, so a client can tell
    a synchronised result from a plain audio mux.
    
    Verified offline (live container untouched): Wav2Lip downloads, patches, and
    lip-syncs a real face to audio, producing a valid 768x512 h264+aac mp4 that
    differs from the source; the second run reuses the install with no download;
    the fallback populates warnings; IP-Adapter loads its weights and sets the scale
    for a supporting pipeline and cleanly declines for a Wan-like one.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    958d2e09
README.md 47.5 KB