• Stefy Lanza (nextime / spora )'s avatar
    audio: implement music dub, demucs separation and ML restoration for real · 091fedd2
    Stefy Lanza (nextime / spora ) authored
    Three related placeholders were shipping as working features:
    
    - audio_stems.separate_with_provider() and audio_clean.restore_with_provider()
      were one-line stubs that unconditionally raised 501, so /v1/audio/stems and
      /v1/audio/cleanup only ever worked in ffmpeg fallback_mode despite demucs and
      deepfilternet being declared dependencies and probed by detect_audio_backends.
    - run_full_music_dub() ran only the STT step and returned hardcoded paths
      (vocals.wav, final_mix.wav) plus 'status': 'placeholder' for the other four,
      while the endpoint reported 'status': 'available'.
    
    Now:
    
    - separate_with_provider drives demucs (demucs.api, CLI fallback), keeps the
      vocals stem and sums drums+bass+other into an instrumental bed for
      vocals-instrumental; 4-stem passes the four through.
    - restore_with_provider runs DeepFilterNet (VoiceFixer fallback) and then applies
      the hum/click/loudness DSP with ffmpeg, so those options are no longer lost.
    - audio-music-dub is a real six-stage chain: separate -> transcribe the ISOLATED
      vocal -> adapt lyrics (LLM prompted for singable output with matching syllable
      count, argostranslate fallback) -> re-sing with F5-TTS cloned from the original
      vocal -> optional Seed-VC f0-conditioned singing pass -> ffmpeg remix + limiter.
      New request fields: text_model, voice_name/ref_text, sing_convert,
      diffusion_steps, pitch_shift, speed, seed, fallback_mode.
    
    Missing optional deps now degrade to a reported 'skipped'/'failed' step with a
    reason and a top-level status of 'complete' vs 'degraded' plus warnings[], rather
    than silently producing placeholders. 501s name the package to install.
    
    Verified offline with a fake-model harness: the chain produces a real RIFF wav
    mix end to end, feeds the vocal stem (not the mix) to STT, and correctly reports
    degraded when seed-vc is unavailable.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    091fedd2
Name
Last commit
Last update
..
__init__.py Loading commit data...
_film_net.py Loading commit data...
_rife_ifnet.py Loading commit data...
app.py Loading commit data...
archive.py Loading commit data...
audio_backends.py Loading commit data...
audio_clean.py Loading commit data...
audio_gen.py Loading commit data...
audio_stems.py Loading commit data...
canary_worker.py Loading commit data...
characters.py Loading commit data...
colibri_worker.py Loading commit data...
crisperwhisper_worker.py Loading commit data...
custom_pipelines.py Loading commit data...
diarization.py Loading commit data...
ds4_kv_janitor.py Loading commit data...
ds4_worker.py Loading commit data...
embeddings.py Loading commit data...
environments.py Loading commit data...
faceswap.py Loading commit data...
fastlog.py Loading commit data...
images.py Loading commit data...
k3_worker.py Loading commit data...
kt_worker.py Loading commit data...
log.py Loading commit data...
loras.py Loading commit data...
ocr.py Loading commit data...
parler_worker.py Loading commit data...
pipelines.py Loading commit data...
prompt_cache.py Loading commit data...
pyannote_worker.py Loading commit data...
ratelimit.py Loading commit data...
rerank.py Loading commit data...
runpod_client.py Loading commit data...
runpod_ledger.py Loading commit data...
runpod_worker.py Loading commit data...
spatial.py Loading commit data...
speaker_embeddings.py Loading commit data...
speaker_registry.py Loading commit data...
state.py Loading commit data...
stt_backends.py Loading commit data...
text.py Loading commit data...
transcriptions.py Loading commit data...
tts.py Loading commit data...
tts_backends.py Loading commit data...
urlutils.py Loading commit data...
video.py Loading commit data...
vllm_worker.py Loading commit data...
voice_clone.py Loading commit data...
voice_convert.py Loading commit data...
vpr_salad_server.py Loading commit data...