video/audio: fix camera_motion 500, dialog voice cloning, and the BigVGAN hub break

Three defects found while auditing the 'person from a video -> scene with their
voice' path.

1. camera_motion returned 500 on every request that used it.
   _apply_camera_motion() set kw['camera_motion'] unconditionally and nothing
   ever popped it, so pipe(**kw) raised TypeError — no diffusers video pipeline
   accepts that kwarg (the 'CogVideoX supports it natively' comment was wrong).
   tools/videogen.py sends the field, so that path was dead. Now the kwarg is
   passed only when the pipeline signature really takes it, otherwise the motion
   is folded into the prompt, which is the only conditioning these models have.

2. dialogs[].voice never cloned the voice it named.
   _generate_tts_for_line() loaded the profile's reference audio and then used it
   purely as a flag to force a generic voice (en-US-JennyNeural / af_sky) —
   F5-TTS was never invoked, so the selected identity was silently discarded.
   It now clones with F5-TTS when the profile has reference audio AND a
   transcript, and says why in the log when it falls back.

3. seed-vc and F5-TTS could not load their BigVGAN vocoder.
   huggingface_hub 1.x stopped forwarding proxies/resume_download to a subclass
   _from_pretrained; BigVGAN still declares them as required keyword-only args,
   so loading died with 'missing 2 required keyword-only arguments'. Pinning hub
   below 1.0 is not possible (transformers 5.x needs the new one), so
   codai/api/hub_compat.py supplies the dropped arguments for exactly the
   subclasses whose signature still requires them. This also un-breaks
   /v1/audio/convert.

Also fixed on the way:
- edge-tts never ran: get_event_loop().run_until_complete() inside an executor
  thread raises RuntimeError. In _generate_tts_for_line it was swallowed; in
  _generate_tts only ImportError was caught, so it propagated out of
  /v1/video/dub. Both now use asyncio.run() and verify the output file.
- music dub adopted a failed Seed-VC result: converted_path was assigned before
  the write succeeded, so a near-empty conversion got remixed instead of falling
  back to the F5-TTS vocal. The result is now validated before adoption.

Verified offline (live container untouched): camera_motion passes through only
to a supporting pipeline and otherwise reaches the prompt; a dialog line clones
with the profile's reference audio and transcript and refuses to clone without
one; seed-vc loads BigVGAN and converts; the music dub falls back to the full
vocal and reports the reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4

codai/api/custom_pipelines.py

@@ -727,8 +727,17 @@ async def run_full_music_dub(request: AudioMusicDubRequest, http_request: Reques
                     audio_out = audio_out[0]
                 import numpy as _np
                 import soundfile as _sf
-                converted_path = os.path.join(workdir, 'converted_vocals.wav')
-                _sf.write(converted_path, _np.array(audio_out).flatten(), 44100)
+                samples = _np.array(audio_out).flatten()
+                # Seed-VC can return a near-empty buffer when it finds nothing voiced
+                # to track. Only adopt the result once we know it is real audio —
+                # writing straight to converted_path would remix silence.
+                if samples.size < 1000:
+                    raise RuntimeError(
+                        f'conversion returned {samples.size} samples — no voiced '
+                        'content detected in the vocal stem')
+                candidate = os.path.join(workdir, 'converted_vocals.wav')
+                _sf.write(candidate, samples, 44100)
+                converted_path = candidate
                 steps.append({'step': 4, 'type': 'voice_convert',
                               'label': 'Match the original singing voice',
                               'status': 'ok', 'engine': 'seed-vc',

codai/api/hub_compat.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+"""Compatibility shims for huggingface_hub API drift.
+
+``huggingface_hub`` 1.x dropped ``proxies`` and ``resume_download`` from the arguments
+that :class:`PyTorchModelHubMixin.from_pretrained` forwards to a subclass's
+``_from_pretrained``. Models that vendor an older mixin signature — notably **BigVGAN**,
+which both seed-vc and F5-TTS pull in as their vocoder — declare those two as *required*
+keyword-only parameters, so loading them raises::
+
+    BigVGAN._from_pretrained() missing 2 required keyword-only arguments:
+    'proxies' and 'resume_download'
+
+Pinning huggingface_hub back below 1.0 is not an option here: transformers 5.x needs the
+new one. So instead we fill the gap at the call boundary — for the specific subclass being
+loaded, and only when its signature actually still requires the dropped arguments.
+"""
+
+import inspect
+
+_LEGACY_KWARGS = {"proxies": None, "resume_download": None}
+_installed = False
+
+
+def install_hub_legacy_kwargs_shim() -> bool:
+    """Make ``PyTorchModelHubMixin.from_pretrained`` tolerate pre-1.0 subclasses.
+
+    Idempotent, and a no-op on hub versions that still pass the legacy arguments (the
+    shim only fires for a subclass whose ``_from_pretrained`` requires something the
+    installed hub does not supply). Returns True when the shim is in place.
+    """
+    global _installed
+    if _installed:
+        return True
+    try:
+        from huggingface_hub import PyTorchModelHubMixin
+    except ImportError:
+        return False
+
+    original = PyTorchModelHubMixin.from_pretrained
+
+    def _patch_subclass(cls) -> None:
+        """Give ``cls._from_pretrained`` defaults for the arguments hub stopped sending."""
+        if getattr(cls, "_codai_legacy_kwargs_patched", False):
+            return
+        bound = cls._from_pretrained
+        try:
+            params = inspect.signature(bound).parameters
+        except (TypeError, ValueError):
+            return
+        missing = {
+            name: default for name, default in _LEGACY_KWARGS.items()
+            if name in params and params[name].default is inspect.Parameter.empty
+        }
+        if not missing:
+            cls._codai_legacy_kwargs_patched = True
+            return
+
+        def _from_pretrained(**kwargs):
+            for name, default in missing.items():
+                kwargs.setdefault(name, default)
+            return bound(**kwargs)
+
+        cls._from_pretrained = staticmethod(_from_pretrained)
+        cls._codai_legacy_kwargs_patched = True
+        print(f"[hub-compat] {cls.__name__}._from_pretrained: supplied "
+              f"{sorted(missing)} dropped by huggingface_hub", flush=True)
+
+    @classmethod
+    def from_pretrained(cls, *args, **kwargs):
+        try:
+            _patch_subclass(cls)
+        except Exception:
+            pass          # never let the shim itself break a working load
+        return original.__func__(cls, *args, **kwargs)
+
+    PyTorchModelHubMixin.from_pretrained = from_pretrained
+    _installed = True
+    return True

codai/api/video.py

@@ -1851,11 +1851,48 @@ def _build_call_kwargs(request: VideoGenerationRequest) -> dict:
     return kw
 
 
-def _apply_camera_motion(kw: dict, camera_motion: str):
-    """Inject camera motion hint into pipeline kwargs (model-dependent)."""
-    # CogVideoX supports camera_motion natively
-    if camera_motion:
-        kw['camera_motion'] = camera_motion
+# How a camera_motion enum reads as a prompt hint. Text conditioning is the only
+# mechanism the current video pipelines actually have for this.
+_CAMERA_MOTION_HINTS = {
+    'zoom-in':    'the camera slowly zooms in',
+    'zoom-out':   'the camera slowly zooms out',
+    'pan-left':   'the camera pans left',
+    'pan-right':  'the camera pans right',
+    'tilt-up':    'the camera tilts up',
+    'tilt-down':  'the camera tilts down',
+    'rotate':     'the camera orbits around the subject',
+}
+
+
+def _apply_camera_motion(kw: dict, camera_motion: str, pipe=None):
+    """Apply a camera-motion hint in whatever way this pipeline actually supports.
+
+    No diffusers video pipeline accepts a ``camera_motion`` kwarg — passing one
+    unconditionally (as this used to) made every request carrying the field die with a
+    TypeError inside ``pipe(**kw)``. So: pass it through only when the signature really
+    takes it, otherwise fold it into the prompt, which is the only conditioning path
+    these models have for camera movement.
+    """
+    if not camera_motion:
+        return
+    motion = str(camera_motion).strip()
+    if not motion:
+        return
+
+    if pipe is not None:
+        import inspect
+        try:
+            if 'camera_motion' in inspect.signature(pipe.__call__).parameters:
+                kw['camera_motion'] = motion
+                return
+        except Exception:
+            pass
+
+    hint = _CAMERA_MOTION_HINTS.get(motion.lower().replace('_', '-'), f'camera motion: {motion}')
+    if kw.get('prompt'):
+        kw['prompt'] = f"{kw['prompt']}, {hint}"
+    else:
+        kw['prompt'] = hint
 
 
 def _resolve_character_inputs(request) -> tuple[List[str], List[str]]:
@@ -2274,7 +2311,7 @@ def _generate_video(pipe, request: VideoGenerationRequest):
     except Exception:
         pass
 
-    _apply_camera_motion(kw, request.camera_motion)
+    _apply_camera_motion(kw, request.camera_motion, pipe=pipe)
 
     char_images, char_names = _resolve_character_inputs(request)
     if char_images:
@@ -3130,9 +3167,13 @@ def _generate_tts(text: str, voice: Optional[str], speed: float,
         out = tempfile.mktemp(suffix='.mp3')
         temps.append(out)
         tts = edge_tts.Communicate(text, voice_id, rate=f"+{int((speed - 1) * 100)}%")
-        _aio.get_event_loop().run_until_complete(tts.save(out))
-        return out
-    except ImportError:
+        # Called from an executor thread, which has no event loop — get_event_loop()
+        # raised RuntimeError, and only ImportError was caught, so this propagated
+        # out of /v1/video/dub and the tts_text path instead of falling back.
+        _aio.run(tts.save(out))
+        if os.path.exists(out) and os.path.getsize(out) > 0:
+            return out
+    except Exception:
         pass
     try:
         from kokoro import KPipeline
@@ -3167,8 +3208,9 @@ def _generate_tts_for_line(line: CharacterDialogLine, temps: list) -> Optional[s
     speed = line.speed or 1.0
     lang = line.lang
 
-    # Try to load voice profile reference audio first (for kokoro/RVC cloning)
+    # Try to load voice profile reference audio first (for F5-TTS cloning)
     ref_audio = None
+    ref_text = ''
     if voice:
         try:
             from codai.api.voice_clone import _load_voice, _voice_path
@@ -3176,9 +3218,34 @@ def _generate_tts_for_line(line: CharacterDialogLine, temps: list) -> Optional[s
             audio_file = meta.get('audio_file') or meta.get('audio_path')
             if audio_file and os.path.isfile(audio_file):
                 ref_audio = audio_file
+                ref_text = meta.get('transcript') or ''
         except Exception:
             pass
 
+    # A saved voice profile means the caller asked for THAT person's voice, so clone
+    # it. This used to load the reference audio and then never use it — the profile
+    # only flipped the line onto a generic neural voice, silently discarding the
+    # identity the caller selected.
+    if ref_audio:
+        if not ref_text:
+            print(f"[dialog] voice profile '{voice}' has no transcript — "
+                  f"cannot clone, falling back to a generic voice", flush=True)
+        else:
+            try:
+                from codai.api.voice_clone import _f5tts_clone
+                wav = _f5tts_clone(ref_audio, ref_text, text, speed, getattr(line, 'seed', None))
+                out = tempfile.mktemp(suffix='.wav')
+                temps.append(out)
+                with open(out, 'wb') as fh:
+                    fh.write(wav)
+                if os.path.getsize(out) > 0:
+                    return out
+            except ImportError:
+                print("[dialog] f5-tts not installed — cannot clone the voice profile "
+                      f"'{voice}'. Run: pip install f5-tts", flush=True)
+            except Exception as exc:
+                print(f"[dialog] voice cloning failed for '{voice}': {exc}", flush=True)
+
     # edge_tts with voice id
     try:
         import edge_tts, asyncio as _aio
@@ -3190,7 +3257,10 @@ def _generate_tts_for_line(line: CharacterDialogLine, temps: list) -> Optional[s
         out = tempfile.mktemp(suffix='.mp3')
         temps.append(out)
         tts = edge_tts.Communicate(text, voice_id, rate=f"+{int((speed - 1) * 100)}%")
-        _aio.get_event_loop().run_until_complete(tts.save(out))
+        # This runs inside an executor thread, which has no event loop of its own —
+        # get_event_loop() raised RuntimeError here and the failure was swallowed,
+        # so edge-tts never actually ran.
+        _aio.run(tts.save(out))
         if os.path.exists(out) and os.path.getsize(out) > 0:
             return out
     except Exception:

codai/api/voice_clone.py

@@ -122,6 +122,10 @@ def _decode_b64_or_url(data: str) -> bytes:
 def _f5tts_clone(ref_audio_path: str, ref_text: str, gen_text: str,
                   speed: float = 1.0, seed: Optional[int] = None) -> bytes:
     """Run F5-TTS voice cloning, return WAV bytes."""
+    # F5-TTS can be configured with the BigVGAN vocoder, which still expects the
+    # pre-1.0 huggingface_hub mixin signature — see codai.api.hub_compat.
+    from codai.api.hub_compat import install_hub_legacy_kwargs_shim
+    install_hub_legacy_kwargs_shim()
     from f5_tts.api import F5TTS
     import soundfile as sf
     import numpy as np

codai/api/voice_convert.py

@@ -39,6 +39,11 @@ def set_global_file_path(path):
 def _get_wrapper():
     global _wrapper
     if _wrapper is None:
+        # seed-vc's vocoder (BigVGAN) still expects the pre-1.0 huggingface_hub
+        # mixin signature; without this it dies on load with "missing 2 required
+        # keyword-only arguments: 'proxies' and 'resume_download'".
+        from codai.api.hub_compat import install_hub_legacy_kwargs_shim
+        install_hub_legacy_kwargs_shim()
         from seed_vc.seed_vc_wrapper import SeedVCWrapper
         _wrapper = SeedVCWrapper()
     return _wrapper