• Stefy Lanza (nextime / spora )'s avatar
    audio: implement music dub, demucs separation and ML restoration for real · 091fedd2
    Stefy Lanza (nextime / spora ) authored
    Three related placeholders were shipping as working features:
    
    - audio_stems.separate_with_provider() and audio_clean.restore_with_provider()
      were one-line stubs that unconditionally raised 501, so /v1/audio/stems and
      /v1/audio/cleanup only ever worked in ffmpeg fallback_mode despite demucs and
      deepfilternet being declared dependencies and probed by detect_audio_backends.
    - run_full_music_dub() ran only the STT step and returned hardcoded paths
      (vocals.wav, final_mix.wav) plus 'status': 'placeholder' for the other four,
      while the endpoint reported 'status': 'available'.
    
    Now:
    
    - separate_with_provider drives demucs (demucs.api, CLI fallback), keeps the
      vocals stem and sums drums+bass+other into an instrumental bed for
      vocals-instrumental; 4-stem passes the four through.
    - restore_with_provider runs DeepFilterNet (VoiceFixer fallback) and then applies
      the hum/click/loudness DSP with ffmpeg, so those options are no longer lost.
    - audio-music-dub is a real six-stage chain: separate -> transcribe the ISOLATED
      vocal -> adapt lyrics (LLM prompted for singable output with matching syllable
      count, argostranslate fallback) -> re-sing with F5-TTS cloned from the original
      vocal -> optional Seed-VC f0-conditioned singing pass -> ffmpeg remix + limiter.
      New request fields: text_model, voice_name/ref_text, sing_convert,
      diffusion_steps, pitch_shift, speed, seed, fallback_mode.
    
    Missing optional deps now degrade to a reported 'skipped'/'failed' step with a
    reason and a top-level status of 'complete' vs 'degraded' plus warnings[], rather
    than silently producing placeholders. 501s name the package to install.
    
    Verified offline with a fake-model harness: the chain produces a real RIFF wav
    mix end to end, feeds the vocal stem (not the mix) to STT, and correctly reports
    degraded when seed-vc is unavailable.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    091fedd2
audio_stems.py 12.2 KB