• Stefy Lanza (nextime / spora )'s avatar
    audio: implement music dub, demucs separation and ML restoration for real · 091fedd2
    Stefy Lanza (nextime / spora ) authored
    Three related placeholders were shipping as working features:
    
    - audio_stems.separate_with_provider() and audio_clean.restore_with_provider()
      were one-line stubs that unconditionally raised 501, so /v1/audio/stems and
      /v1/audio/cleanup only ever worked in ffmpeg fallback_mode despite demucs and
      deepfilternet being declared dependencies and probed by detect_audio_backends.
    - run_full_music_dub() ran only the STT step and returned hardcoded paths
      (vocals.wav, final_mix.wav) plus 'status': 'placeholder' for the other four,
      while the endpoint reported 'status': 'available'.
    
    Now:
    
    - separate_with_provider drives demucs (demucs.api, CLI fallback), keeps the
      vocals stem and sums drums+bass+other into an instrumental bed for
      vocals-instrumental; 4-stem passes the four through.
    - restore_with_provider runs DeepFilterNet (VoiceFixer fallback) and then applies
      the hum/click/loudness DSP with ffmpeg, so those options are no longer lost.
    - audio-music-dub is a real six-stage chain: separate -> transcribe the ISOLATED
      vocal -> adapt lyrics (LLM prompted for singable output with matching syllable
      count, argostranslate fallback) -> re-sing with F5-TTS cloned from the original
      vocal -> optional Seed-VC f0-conditioned singing pass -> ffmpeg remix + limiter.
      New request fields: text_model, voice_name/ref_text, sing_convert,
      diffusion_steps, pitch_shift, speed, seed, fallback_mode.
    
    Missing optional deps now degrade to a reported 'skipped'/'failed' step with a
    reason and a top-level status of 'complete' vs 'degraded' plus warnings[], rather
    than silently producing placeholders. 501s name the package to install.
    
    Verified offline with a fake-model harness: the chain produces a real RIFF wav
    mix end to end, feeds the vocal stem (not the mix) to STT, and correctly reports
    degraded when seed-vc is unavailable.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    091fedd2
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...