audio: implement music dub, demucs separation and ML restoration for real

Three related placeholders were shipping as working features:

- audio_stems.separate_with_provider() and audio_clean.restore_with_provider()
  were one-line stubs that unconditionally raised 501, so /v1/audio/stems and
  /v1/audio/cleanup only ever worked in ffmpeg fallback_mode despite demucs and
  deepfilternet being declared dependencies and probed by detect_audio_backends.
- run_full_music_dub() ran only the STT step and returned hardcoded paths
  (vocals.wav, final_mix.wav) plus 'status': 'placeholder' for the other four,
  while the endpoint reported 'status': 'available'.

Now:

- separate_with_provider drives demucs (demucs.api, CLI fallback), keeps the
  vocals stem and sums drums+bass+other into an instrumental bed for
  vocals-instrumental; 4-stem passes the four through.
- restore_with_provider runs DeepFilterNet (VoiceFixer fallback) and then applies
  the hum/click/loudness DSP with ffmpeg, so those options are no longer lost.
- audio-music-dub is a real six-stage chain: separate -> transcribe the ISOLATED
  vocal -> adapt lyrics (LLM prompted for singable output with matching syllable
  count, argostranslate fallback) -> re-sing with F5-TTS cloned from the original
  vocal -> optional Seed-VC f0-conditioned singing pass -> ffmpeg remix + limiter.
  New request fields: text_model, voice_name/ref_text, sing_convert,
  diffusion_steps, pitch_shift, speed, seed, fallback_mode.

Missing optional deps now degrade to a reported 'skipped'/'failed' step with a
reason and a top-level status of 'complete' vs 'degraded' plus warnings[], rather
than silently producing placeholders. 501s name the package to install.

Verified offline with a fake-model harness: the chain produces a real RIFF wav
mix end to end, feeds the vocal stem (not the mix) to STT, and correctly reports
degraded when seed-vc is unavailable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4

README.md

@@ -244,6 +244,41 @@ Built-in multi-step pipelines callable from the API or web UI:
 | `POST /v1/pipelines/story` | LLM script → images per scene → video → TTS narration |
 | `POST /v1/pipelines/audio-dub` | Transcribe audio/video → translate → clone voice → replace audio |
 
+#### Music dubbing
+
+`POST /v1/pipelines/audio-music-dub` sings a song in another language over its own
+backing track. Six real stages:
+
+1. **Separate** — Demucs splits the track; the vocal is isolated and drums/bass/other are
+   summed back into an instrumental bed
+2. **Transcribe** — STT runs on the *isolated vocal*, not the full mix, which is what makes
+   the lyrics legible
+3. **Adapt** — with a `text_model`, an LLM adapts the lyrics to be **singable**: same
+   syllable count per line, rhyme preserved where possible. Without one it falls back to a
+   literal argostranslate pass
+4. **Re-sing** — F5-TTS synthesises the new lyrics using the isolated original vocal as the
+   cloning reference, so the dub keeps the original singer's timbre with no voice profile to
+   set up (or pass `voice_name` to use a saved one)
+5. **Match the singing** — an optional Seed-VC pass with `f0_condition` on, conditioned on
+   the original vocal, to pull the result toward sung rather than spoken delivery
+6. **Remix** — the new vocal is mixed back over the instrumental and limited
+
+```json
+{
+  "audio": "<base64 or data URI>",
+  "audio_model": "whisper0",
+  "text_model": "my-llm",
+  "source_lang": "en",
+  "target_lang": "it",
+  "notes": "keep it playful"
+}
+```
+
+Every stage that needs an optional dependency degrades to a reported `skipped` state with
+a reason instead of failing or faking it. The response carries `status` (`complete` or
+`degraded`), a `warnings` list, per-step statuses, and all four artifacts — `vocals`,
+`instrumental`, `converted_vocals` and `final_mix`.
+
 **Custom Pipeline Builder**: Create, save and run your own multi-step pipelines from the web UI or API. Chain any combination of 18 step types with `{{input}}` and `{{stepN.output}}` template variables.
 
 ### Advanced Features
@@ -422,10 +457,8 @@ Notes:
 - `rnnoise` and `voicefixer` are optional alternates / complements.
 - Full music-dub quality depends on separation plus singing-capable conversion; even with this stack, output quality still depends heavily on source material and model/runtime availability.
 
-> **Status note:** `/v1/pipelines/audio-music-dub` is **not complete**. Only the
-> transcription step actually runs today; the separation, translation, voice-conversion and
-> remix steps are returned as placeholders. Use `/v1/audio/stems`, `/v1/audio/convert` and
-> `/v1/pipelines/audio-dub` directly until it is finished.
+Without this stack, `/v1/audio/stems` and `/v1/audio/cleanup` return `501` with the
+package to install — or accept `fallback_mode: true` for the best-effort ffmpeg path.
 
 ### Face Swap
 
@@ -777,7 +810,7 @@ persisted per `config_id`, so the configs don't overwrite each other.
 | `POST /v1/pipelines/story` | LLM → images → video → TTS |
 | `POST /v1/pipelines/audio-dub` | Audio/video dub with voice cloning |
 | `POST /v1/pipelines/audio-understand` | Transcribe audio, then answer a question about it with a text model |
-| `POST /v1/pipelines/audio-music-dub` | Music dub — **partly a stub**, see the note below |
+| `POST /v1/pipelines/audio-music-dub` | Dub a song into another language over its original backing track |
 | `GET /v1/pipelines/custom` | List custom pipelines |
 | `POST /v1/pipelines/custom` | Create custom pipeline |
 | `PUT /v1/pipelines/custom/{id}` | Update custom pipeline |

codai/api/audio_clean.py

@@ -63,8 +63,93 @@ def _run_ffmpeg(command):
         raise HTTPException(status_code=500, detail=detail)
 
 
+def _deepfilter_denoise(src: str, dst: str) -> str:
+    """DeepFilterNet speech enhancement. Raises ImportError when unavailable."""
+    from df.enhance import enhance, init_df, load_audio, save_audio  # noqa: F401
+
+    model, df_state, _ = init_df()
+    audio, _meta = load_audio(src, sr=df_state.sr())
+    enhanced = enhance(model, df_state, audio)
+    # save_audio() appends the suffix itself, so hand it the stem.
+    stem = dst[:-4] if dst.endswith(".wav") else dst
+    save_audio(stem, enhanced, df_state.sr())
+    return dst if os.path.exists(dst) else stem + ".wav"
+
+
+def _voicefixer_restore(src: str, dst: str) -> str:
+    """VoiceFixer general restoration. Raises ImportError when unavailable."""
+    from voicefixer import VoiceFixer
+
+    VoiceFixer().restore(input=src, output=dst, cuda=False, mode=0)
+    return dst
+
+
 def restore_with_provider(audio_bytes: bytes, options: dict, workdir: str) -> dict:
-    raise HTTPException(status_code=501, detail="ML audio restoration backend not installed")
+    """Real ML restoration.
+
+    DeepFilterNet is the primary denoiser; VoiceFixer is the fallback for clips it
+    can't handle. The remaining requested operations (hum removal, click repair,
+    loudness normalisation) are DSP, so they are applied with ffmpeg *after* the
+    learned pass rather than being skipped.
+    """
+    src = os.path.join(workdir, "input.wav")
+    with open(src, "wb") as handle:
+        handle.write(audio_bytes)
+
+    enhanced = os.path.join(workdir, "enhanced.wav")
+    engine = None
+    errors = []
+    if options.get("noise_reduction", True):
+        for name, fn in (("deepfilternet", _deepfilter_denoise),
+                         ("voicefixer", _voicefixer_restore)):
+            try:
+                enhanced = fn(src, enhanced)
+                engine = name
+                break
+            except ImportError as exc:
+                errors.append(f"{name}: {exc}")
+            except Exception as exc:
+                errors.append(f"{name} failed: {exc}")
+        if engine is None:
+            raise HTTPException(
+                status_code=501,
+                detail="ML audio restoration needs deepfilternet or voicefixer "
+                       "(" + "; ".join(errors) + "). Run: pip install deepfilternet — "
+                       "or call /v1/audio/cleanup with fallback_mode=true for the "
+                       "ffmpeg filter chain.")
+        applied = ["noise_reduction"]
+    else:
+        enhanced, applied = src, []
+
+    # DSP touch-ups the learned model doesn't do.
+    filters = []
+    if options.get("remove_hum"):
+        filters.append("highpass=f=60,lowpass=f=15000")
+        applied.append("remove_hum")
+    if options.get("repair_clicks"):
+        filters.append("adeclick=t=40")
+        applied.append("repair_clicks")
+    if options.get("normalize"):
+        filters.append("loudnorm=I=-16:TP=-1.5:LRA=11")
+        applied.append("normalize")
+
+    out = enhanced
+    if filters:
+        out = os.path.join(workdir, "cleaned.wav")
+        _run_ffmpeg([_ffmpeg_binary(), "-y", "-i", enhanced, "-af", ",".join(filters), out])
+
+    if not applied:
+        raise HTTPException(status_code=400, detail="Select at least one cleanup operation")
+
+    return {
+        "path": out,
+        "engine": engine or "ffmpeg-filter-chain",
+        "applied": applied,
+        "limitations": [
+            "tuned for speech; music and wide-band material may be dulled",
+            "severe clipping or codec damage is not fully recoverable",
+        ],
+    }
 
 
 def _cleanup_audio(audio_bytes: bytes, options: dict, workdir: str) -> dict:

codai/api/audio_stems.py

@@ -63,8 +63,126 @@ def _run_ffmpeg(command: List[str]):
         raise HTTPException(status_code=500, detail=detail)
 
 
+#: demucs stem order; "other" carries everything that is not drums/bass/vocals.
+_DEMUCS_STEMS = ("drums", "bass", "other", "vocals")
+
+
+def _demucs_separate(src: str, workdir: str) -> dict:
+    """Run demucs on ``src`` and write one wav per stem into ``workdir``.
+
+    Returns ``{stem_name: path}``. Prefers the in-process ``demucs.api`` (demucs 4.x)
+    and falls back to the CLI for builds that don't ship it. Raises ImportError when
+    demucs isn't installed at all, so the caller can report a clean 501.
+    """
+    try:
+        from demucs.api import Separator, save_audio
+    except ImportError:
+        return _demucs_separate_cli(src, workdir)
+
+    separator = Separator()          # default model (htdemucs)
+    _origin, stems = separator.separate_audio_file(src)
+    out = {}
+    for name, tensor in stems.items():
+        path = os.path.join(workdir, f"{name}.wav")
+        save_audio(tensor, path, samplerate=separator.samplerate)
+        out[name] = path
+    return out
+
+
+def _demucs_separate_cli(src: str, workdir: str) -> dict:
+    """Fallback path: drive demucs through its CLI (`python -m demucs`)."""
+    import sys
+
+    outdir = os.path.join(workdir, "demucs")
+    proc = subprocess.run(
+        [sys.executable, "-m", "demucs", "-o", outdir, src],
+        capture_output=True, text=True,
+    )
+    if proc.returncode != 0:
+        detail = (proc.stderr or proc.stdout or "").strip()
+        if "No module named" in detail:
+            raise ImportError(detail)
+        raise HTTPException(status_code=500, detail=f"demucs failed: {detail[-500:]}")
+
+    # demucs writes <outdir>/<model>/<track-name>/<stem>.wav
+    out = {}
+    for root, _dirs, files in os.walk(outdir):
+        for fn in files:
+            stem = os.path.splitext(fn)[0]
+            if stem in _DEMUCS_STEMS:
+                out[stem] = os.path.join(root, fn)
+    if not out:
+        raise HTTPException(status_code=500, detail="demucs produced no stems")
+    return out
+
+
+def _mix_down(paths: List[str], dest: str) -> str:
+    """Sum several stems into one file (used to rebuild the instrumental)."""
+    ffmpeg = _ffmpeg_binary()
+    if len(paths) == 1:
+        _run_ffmpeg([ffmpeg, "-y", "-i", paths[0], dest])
+        return dest
+    cmd = [ffmpeg, "-y"]
+    for p in paths:
+        cmd += ["-i", p]
+    cmd += ["-filter_complex",
+            f"amix=inputs={len(paths)}:duration=longest:normalize=0",
+            dest]
+    _run_ffmpeg(cmd)
+    return dest
+
+
 def separate_with_provider(audio_bytes: bytes, stem_mode: str, workdir: str) -> dict:
-    raise HTTPException(status_code=501, detail="ML stem separation backend not installed")
+    """Real ML source separation via demucs.
+
+    ``vocals-instrumental`` keeps demucs' vocals stem and sums drums+bass+other back
+    into a single instrumental bed — which is what a music dub needs. ``4-stem``
+    returns demucs' four stems as-is.
+    """
+    src = os.path.join(workdir, "input.wav")
+    with open(src, "wb") as handle:
+        handle.write(audio_bytes)
+
+    try:
+        stems = _demucs_separate(src, workdir)
+    except ImportError:
+        raise HTTPException(
+            status_code=501,
+            detail="ML stem separation needs demucs. Run: pip install demucs "
+                   "(or call /v1/audio/stems with fallback_mode=true for the "
+                   "best-effort ffmpeg split).")
+
+    mode = stem_mode or "vocals-instrumental"
+    if mode == "vocals-instrumental":
+        vocals = stems.get("vocals")
+        backing = [p for name, p in stems.items() if name != "vocals"]
+        if not vocals or not backing:
+            raise HTTPException(status_code=500,
+                                detail="demucs did not return the expected stems")
+        instrumental = _mix_down(sorted(backing),
+                                 os.path.join(workdir, "instrumental.wav"))
+        artifacts = [
+            {"name": "vocals", "path": vocals, "role": "lead-vocal"},
+            {"name": "instrumental", "path": instrumental, "role": "backing-track"},
+        ]
+    elif mode == "4-stem":
+        artifacts = [{"name": n, "path": stems[n], "role": n}
+                     for n in _DEMUCS_STEMS if n in stems]
+        if not artifacts:
+            raise HTTPException(status_code=500, detail="demucs returned no stems")
+    else:
+        raise HTTPException(status_code=400, detail=f"Unsupported stem_mode: {mode}")
+
+    return {
+        "stem_mode": mode,
+        "artifacts": artifacts,
+        "engine": "demucs",
+        "model": "htdemucs",
+        "limitations": [
+            "separation quality depends on the source mix",
+            "bleed between stems is normal on dense mixes",
+        ],
+    }
 
 
 def _split_audio(audio_bytes: bytes, mode: str, workdir: str) -> dict: