• Stefy Lanza (nextime / spora )'s avatar
    video/audio: fix camera_motion 500, dialog voice cloning, and the BigVGAN hub break · cc18ca3e
    Stefy Lanza (nextime / spora ) authored
    Three defects found while auditing the 'person from a video -> scene with their
    voice' path.
    
    1. camera_motion returned 500 on every request that used it.
       _apply_camera_motion() set kw['camera_motion'] unconditionally and nothing
       ever popped it, so pipe(**kw) raised TypeError — no diffusers video pipeline
       accepts that kwarg (the 'CogVideoX supports it natively' comment was wrong).
       tools/videogen.py sends the field, so that path was dead. Now the kwarg is
       passed only when the pipeline signature really takes it, otherwise the motion
       is folded into the prompt, which is the only conditioning these models have.
    
    2. dialogs[].voice never cloned the voice it named.
       _generate_tts_for_line() loaded the profile's reference audio and then used it
       purely as a flag to force a generic voice (en-US-JennyNeural / af_sky) —
       F5-TTS was never invoked, so the selected identity was silently discarded.
       It now clones with F5-TTS when the profile has reference audio AND a
       transcript, and says why in the log when it falls back.
    
    3. seed-vc and F5-TTS could not load their BigVGAN vocoder.
       huggingface_hub 1.x stopped forwarding proxies/resume_download to a subclass
       _from_pretrained; BigVGAN still declares them as required keyword-only args,
       so loading died with 'missing 2 required keyword-only arguments'. Pinning hub
       below 1.0 is not possible (transformers 5.x needs the new one), so
       codai/api/hub_compat.py supplies the dropped arguments for exactly the
       subclasses whose signature still requires them. This also un-breaks
       /v1/audio/convert.
    
    Also fixed on the way:
    - edge-tts never ran: get_event_loop().run_until_complete() inside an executor
      thread raises RuntimeError. In _generate_tts_for_line it was swallowed; in
      _generate_tts only ImportError was caught, so it propagated out of
      /v1/video/dub. Both now use asyncio.run() and verify the output file.
    - music dub adopted a failed Seed-VC result: converted_path was assigned before
      the write succeeded, so a near-empty conversion got remixed instead of falling
      back to the F5-TTS vocal. The result is now validated before adoption.
    
    Verified offline (live container untouched): camera_motion passes through only
    to a supporting pipeline and otherwise reaches the prompt; a dialog line clones
    with the profile's reference audio and transcript and refuses to clone without
    one; seed-vc loads BigVGAN and converts; the music dub falls back to the full
    vocal and reports the reason.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    cc18ca3e
hub_compat.py 3.26 KB