• Stefy Lanza (nextime / spora )'s avatar
    video: real IP-Adapter conditioning and a managed Wav2Lip install · 958d2e09
    Stefy Lanza (nextime / spora ) authored
    The last two gaps from the person-to-scene audit.
    
    IP-Adapter on the video path never worked. _apply_character_refs() set
    kw['ip_adapter_image'] without ever calling load_ip_adapter(), so a pipeline that
    does accept the kwarg (AnimateDiff) failed inside diffusers rather than
    transferring identity, and it also passed ip_adapter_scale as a call kwarg, which
    no pipeline accepts — a second latent TypeError. It now reuses the image path's
    _ensure_ip_adapter_loaded(), sets the scale through set_ip_adapter_scale(), and
    returns whether conditioning is actually active. Pipelines with no IP-Adapter at
    all (Wan/CogVideoX/LTX/SVD) now say so instead of looking like they worked.
    
    Lip sync required a 'wav2lip' or 'sadtalker' executable on PATH — a file that
    essentially never exists, since Wav2Lip ships neither a pip package nor a binary
    and its original checkpoint links are dead. codai/api/lipsync.py now owns the
    lifecycle the way ds4_worker owns ds4: it fetches code AND weights from the
    camenduru/Wav2Lip mirror on first use, patches what no longer runs on modern
    numpy/librosa/torch, creates the scratch dirs inference.py expects, and drives it
    as a subprocess. An operator-installed binary still takes precedence.
    
    Each patched file is compiled before being kept, so a bad rewrite is caught at
    patch time and reverted rather than surfacing as a SyntaxError from a subprocess
    (which is exactly how the first attempt failed: two librosa patterns compounded
    into 'sr=sr=').
    
    Lip-sync fallback is no longer silent. The reason travels up a per-request
    warnings list — not module state, which would cross-talk between concurrent
    requests — and appears as a `warnings` field on the video response, so a client can tell
    a synchronised result from a plain audio mux.
    
    Verified offline (live container untouched): Wav2Lip downloads, patches, and
    lip-syncs a real face to audio, producing a valid 768x512 h264+aac mp4 that
    differs from the source; the second run reuses the install with no download;
    the fallback populates warnings; IP-Adapter loads its weights and sets the scale
    for a supporting pipeline and cleanly declines for a Wan-like one.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    958d2e09
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...