voice: cloning engines, multi-clip profiles, watermarking, MeloTTS

Everything OpenVoice does that we didn't, built from models we already run rather
than adopting theirs — plus MeloTTS itself so their exact V2 pairing is available.

Engines on /v1/audio/clone (`engine`, default `auto`):
  f5       today's in-context clone — timbre AND prosody, needs the transcript
  xtts     XTTS-v2 zero-shot from the clip, 17 languages, no transcript
  chain    base TTS -> seed-vc timbre transfer: OpenVoice's architecture with
           seed-vc in place of its tone-colour converter
  melotts  the same chain with MeloTTS as the base voice
`auto` picks a transcript-free engine when the profile has no transcript or the
target language differs from the reference — the case where F5 carries the wrong
accent across. GET /v1/audio/clone/engines reports what is installed.

The transcript is no longer a hard requirement: it is now checked per engine
instead of rejecting the request outright.

Multi-clip profiles: PATCH /v1/audio/voices/{name} takes add_clips/remove_clips,
and a clone prompts on the cleanest take. Scored on clipping, level and crest
factor — explicitly NOT loudest-wins, since a clipped take has the highest RMS of
the set and is the worst possible prompt. Single-clip profiles keep working.

Watermarking with AudioSeal (MIT), on by default, per-request or global override,
detection at POST /v1/audio/watermark/detect. Missing audioseal degrades to
unmarked audio with one warning, never an error.

MeloTTS runs in an isolated venv worker like parler-tts — its pins conflict with
this stack the same way. _RemoteParlerBackend generalised to _RemoteSpeechBackend
since both speak the same POST /speak contract.
Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
parent 1d5ee189
# CoderAI - OpenAI-compatible API server
# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
"""Fully-managed MeloTTS worker.
MeloTTS is the base speaker model OpenVoice V2 synthesises with before its
tone-colour converter re-paints the timbre. coderai uses it the same way: as the
base voice of the `chain` cloning engine (MeloTTS speaks the line, seed-vc moves
the timbre over), and as a plain multilingual TTS engine in its own right —
English, Spanish, French, Chinese, Japanese and Korean.
Its pins (an older transformers/tokenizers line, unidic, a Japanese tokenizer
chain) conflict with this server's stack exactly the way parler-tts's do, so it
runs in an ISOLATED venv and is reached over HTTP. This module owns the venv and
the process; the synthesis lives in tools/melotts_service.py.
"""
import os
import socket
import subprocess
import sys
import threading
import time
from pathlib import Path
_REPO_ROOT = Path(__file__).resolve().parents[2]
_SERVICE_SCRIPT = _REPO_ROOT / "tools" / "melotts_service.py"
# Dedicated venv for the (incompatible) melotts stack. Created with access to
# the base interpreter's packages so torch/numpy aren't re-downloaded; melotts's
# pinned transformers installs into the venv and shadows the system one.
_VENV_DIR = Path(os.environ.get("CODERAI_MELOTTS_VENV")
or os.path.expanduser("~/.coderai/melotts_venv"))
_lock = threading.RLock()
_services: dict[str, dict] = {} # model_name -> {"proc","port","url"}
_bootstrapped = False
def _venv_python() -> Path:
return _VENV_DIR / ("Scripts" if os.name == "nt" else "bin") / (
"python.exe" if os.name == "nt" else "python")
def _pip_ok(py: Path) -> bool:
try:
return subprocess.run([str(py), "-c", "import melo, soundfile"],
capture_output=True).returncode == 0
except Exception:
return False
def _venv_is_system_site() -> bool:
"""True if the venv was built with --system-site-packages (can't isolate)."""
try:
return "include-system-site-packages = true" in \
(_VENV_DIR / "pyvenv.cfg").read_text().lower()
except Exception:
return False
def _bootstrap_venv() -> Path:
"""Create a fully-isolated venv and install melotts (idempotent).
Isolation is the whole point: melotts pins an old transformers/tokenizers
that must NOT be shared with — or shadowed by — the server's stack, so the
venv gets its own copy of everything (torch included). Returns its python."""
global _bootstrapped
py = _venv_python()
if _bootstrapped and py.exists():
return py
# A previously-created shared-site venv leaks the server's transformers in;
# rebuild it isolated.
if py.exists() and _venv_is_system_site():
import shutil
print("[melotts] rebuilding venv as fully isolated …", flush=True)
shutil.rmtree(_VENV_DIR, ignore_errors=True)
if not _venv_python().exists():
print(f"[melotts] creating isolated venv at {_VENV_DIR} …", flush=True)
_VENV_DIR.parent.mkdir(parents=True, exist_ok=True)
subprocess.run([sys.executable, "-m", "venv", str(_VENV_DIR)], check=True)
py = _venv_python()
if not _pip_ok(py):
print("[melotts] installing melotts + torch into the isolated venv "
"(first run, downloads several GB, this can take a while) …", flush=True)
subprocess.run([str(py), "-m", "pip", "install",
"git+https://github.com/huggingface/melotts.git",
"soundfile"], check=True)
if not _pip_ok(py):
raise RuntimeError("melotts install did not yield an importable package")
_bootstrapped = True
return py
def _free_port() -> int:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
def _pump_logs(proc: subprocess.Popen, tail):
for line in proc.stdout:
line = line.rstrip()
if line:
tail.append(line)
print(f"[melotts] {line}", flush=True)
def _health_ok(url: str) -> bool:
import requests
try:
r = requests.get(url + "/health", timeout=3)
return r.ok and bool(r.json().get("ok"))
except Exception:
return False
def ensure_service(model_name: str, ready_timeout: float = 1800.0) -> str:
"""Start (or reuse) the worker for ``model_name`` and return its base URL.
First call bootstraps the venv and downloads the model, so the timeout is
generous. Raises RuntimeError if the service never comes up."""
with _lock:
svc = _services.get(model_name)
if svc and svc["proc"].poll() is None and _health_ok(svc["url"]):
return svc["url"]
if svc and svc["proc"].poll() is not None:
_services.pop(model_name, None) # died — restart below
py = _bootstrap_venv()
port = _free_port()
url = f"http://127.0.0.1:{port}"
env = dict(os.environ)
# The worker must use the model already pulled via coderai's HF download
# interface — it never downloads anything itself. Point it at coderai's
# cache and force offline mode, so a missing model fails fast instead of
# silently fetching.
try:
from codai.models.cache import get_hf_hub_cache_dir
hub = get_hf_hub_cache_dir()
env["HF_HUB_CACHE"] = hub
env["HUGGINGFACE_HUB_CACHE"] = hub
except Exception:
pass
env["HF_HUB_OFFLINE"] = "1"
env["TRANSFORMERS_OFFLINE"] = "1"
proc = subprocess.Popen(
[str(py), str(_SERVICE_SCRIPT), "--model", model_name,
"--host", "127.0.0.1", "--port", str(port)],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
bufsize=1, env=env, cwd=str(_REPO_ROOT),
)
import collections
tail = collections.deque(maxlen=15)
threading.Thread(target=_pump_logs, args=(proc, tail), daemon=True).start()
_services[model_name] = {"proc": proc, "port": port, "url": url}
def _tail_msg():
joined = " | ".join(list(tail)[-5:]).strip()
if "offline" in joined.lower() or "not" in joined.lower() and "found" in joined.lower():
return (f". The model isn't in coderai's cache — download "
f"'{model_name}' from the model interface first. ({joined})")
return f". Last output: {joined}" if joined else ""
# Wait (outside the lock) for the service to load the model and answer.
deadline = time.time() + ready_timeout
while time.time() < deadline:
if proc.poll() is not None:
raise RuntimeError(
f"MeloTTS worker exited (code {proc.returncode}) before becoming ready"
+ _tail_msg())
if _health_ok(url):
print(f"[melotts] service ready for {model_name} at {url}", flush=True)
return url
time.sleep(2)
stop_service(model_name)
raise RuntimeError(f"MeloTTS worker for {model_name} did not become ready in time"
+ _tail_msg())
def stop_service(model_name: str) -> None:
with _lock:
svc = _services.pop(model_name, None)
if not svc:
return
proc = svc["proc"]
if proc.poll() is None:
try:
proc.terminate()
proc.wait(timeout=10)
except Exception:
pass
if proc.poll() is None:
try:
proc.kill()
except Exception:
pass
print(f"[melotts] service for {model_name} stopped", flush=True)
def stop_all() -> None:
for name in list(_services.keys()):
stop_service(name)
import atexit as _atexit
_atexit.register(stop_all)
......@@ -50,6 +50,8 @@ def _family(model_name: str) -> str:
return "parler"
if "bark" in n:
return "bark"
if "melo" in n:
return "melo"
return "transformers"
......@@ -376,11 +378,16 @@ class _BarkBackend:
# (parler-tts pins an old transformers that conflicts with this server's stack).
# --------------------------------------------------------------------------- #
class _RemoteParlerBackend:
family = "parler"
class _RemoteSpeechBackend:
"""A TTS engine that runs in its own venv and answers over HTTP.
Used for the engines whose dependencies can't share this venv: parler-tts and
MeloTTS. Both expose the same tiny contract — POST /speak -> WAV bytes."""
default_voice = ""
def __init__(self, config: dict, managed_model: Optional[str] = None):
def __init__(self, config: dict, family: str = "parler",
managed_model: Optional[str] = None):
self.family = family
self._cfg = config or {}
self._url = str(self._cfg["service_url"]).rstrip("/")
# When coderai launched the worker itself, remember the model so the
......@@ -426,16 +433,26 @@ def load_backend(model_name: str, model_path: Optional[str], config: Optional[di
return _CoquiBackend(model_name, config)
if fam == "bark":
return _BarkBackend(model_name, config)
if fam == "melo":
# MeloTTS is OpenVoice V2's base speaker model. Its pins conflict with this
# stack the way parler's do, so it runs in an isolated venv worker and is
# reached over HTTP — same contract, same managed lifecycle.
if config.get("service_url"):
return _RemoteSpeechBackend(config, family="melo")
from codai.api import melotts_worker
url = melotts_worker.ensure_service(model_name)
return _RemoteSpeechBackend({**config, "service_url": url}, family="melo",
managed_model=model_name)
if fam == "parler":
# An explicit service_url points at an externally-run service. Otherwise
# coderai fully manages the worker: bootstrap its venv, spawn it, and
# route to it — no manual setup needed.
if config.get("service_url"):
return _RemoteParlerBackend(config)
return _RemoteSpeechBackend(config, family="parler")
from codai.api import parler_worker
url = parler_worker.ensure_service(model_name)
return _RemoteParlerBackend({**config, "service_url": url},
managed_model=model_name)
return _RemoteSpeechBackend({**config, "service_url": url},
family="parler", managed_model=model_name)
return _TransformersBackend(model_name, config)
......
......@@ -73,6 +73,76 @@ def _list_voices() -> list:
return sorted(voices, key=lambda v: v.get('created_at', 0))
def _clip_score(path: str) -> float:
"""Rough usability score for a reference clip. 0.0 if unreadable.
Judged on three things a clone actually suffers from: clipping (a hard
distortion no engine can undo), level (too quiet means noise dominates), and
crest factor (speech sits around 4-8; flatter means compressed or clipped).
Deliberately NOT "loudest wins" — a clipped take has the highest RMS of all
and is the worst possible prompt."""
try:
import numpy as np
import soundfile as sf
data, _sr = sf.read(path, dtype='float32')
if data.ndim > 1:
data = data.mean(axis=1)
if not len(data):
return 0.0
peak = float(np.max(np.abs(data)))
rms = float(np.sqrt(np.mean(data ** 2)))
if rms <= 1e-6 or peak <= 1e-6:
return 0.0
# 1. clipping: fraction of samples pinned at full scale. 0.1% is already
# audible, 1% is ruinous.
clipped = float(np.mean(np.abs(data) >= 0.995))
clip_term = max(0.0, 1.0 - clipped * 300.0)
# 2. level: best around -20 dBFS RMS (~0.1), falling off either side.
level_term = float(np.exp(-((np.log10(rms) - np.log10(0.1)) ** 2) / 0.5))
# 3. crest factor: speech ~4-8. Flat (clipped/compressed) or spiky
# (silence with a bang) both score worse.
crest = peak / rms
crest_term = float(np.exp(-((crest - 6.0) ** 2) / 32.0))
return clip_term * level_term * crest_term
except Exception:
return 0.0
def _profile_clips(meta: dict) -> list:
"""Every reference clip on a profile, newest first, as [{path, transcript}].
Profiles used to hold exactly one clip (`audio_file` + `transcript`); that shape
is still honoured so existing profiles keep working."""
clips = []
for c in (meta.get('clips') or []):
if isinstance(c, dict) and c.get('path') and os.path.isfile(c['path']):
clips.append(c)
if not clips and meta.get('audio_file') and os.path.isfile(meta['audio_file']):
clips.append({'path': meta['audio_file'], 'transcript': meta.get('transcript', '')})
return clips
def _best_clip(meta: dict) -> Optional[dict]:
"""The clip a clone should prompt on.
More references only help engines that can average them (see `--engine chain`);
F5-TTS prompts on ONE, so picking the cleanest take beats picking the first."""
clips = _profile_clips(meta)
if not clips:
return None
if len(clips) == 1:
return clips[0]
scored = sorted(clips, key=lambda c: _clip_score(c['path']), reverse=True)
best = scored[0]
print(f" [voice-clone] {len(clips)} reference clips — using "
f"{os.path.basename(best['path'])} (cleanest of the set)", flush=True)
return best
def _save_voice(name: str, audio_bytes: bytes, audio_ext: str, transcript: str, description: str = '') -> dict:
vdir = _voice_path(name)
os.makedirs(vdir, exist_ok=True)
......@@ -85,6 +155,7 @@ def _save_voice(name: str, audio_bytes: bytes, audio_ext: str, transcript: str,
'transcript': transcript,
'audio_file': audio_file,
'audio_ext': audio_ext,
'clips': [{'path': audio_file, 'transcript': transcript}],
'created_at': int(time.time()),
}
with open(os.path.join(vdir, 'meta.json'), 'w') as f:
......@@ -239,6 +310,243 @@ def _f5tts_clone(ref_audio_path: str, ref_text: str, gen_text: str,
return buf.getvalue()
# ---------------------------------------------------------------------------
# Watermarking
#
# Cloned speech of real people leaves this server at scale — dubbed videos,
# character dialogue, whole generated scenes. OpenVoice marks its output;
# nothing here did. AudioSeal (MIT, Meta) is the stronger option: an inaudible,
# localised mark that survives the usual re-encoding, plus a detector.
#
# On by default, because a watermark nobody enables protects nobody. Disable per
# request with watermark=false, or globally with CODERAI_AUDIO_WATERMARK=0.
# Degrades gracefully: if audioseal isn't installed the audio is returned
# unchanged with a one-time warning, never an error.
# ---------------------------------------------------------------------------
WATERMARK_DEFAULT = os.environ.get("CODERAI_AUDIO_WATERMARK", "1") not in ("0", "false", "no")
DEFAULT_CHAIN_BASE = os.environ.get("CODERAI_CLONE_BASE_TTS", "hexgrad/Kokoro-82M")
_watermarker = None
_watermark_warned = False
_watermark_lock = threading.Lock()
def _get_watermarker():
"""Cached AudioSeal generator, or None when the package isn't available."""
global _watermarker, _watermark_warned
with _watermark_lock:
if _watermarker is not None:
return _watermarker
try:
from audioseal import AudioSeal
_watermarker = AudioSeal.load_generator("audioseal_wm_16bits")
print(" [voice-clone] AudioSeal watermarking active", flush=True)
return _watermarker
except Exception as exc:
if not _watermark_warned:
_watermark_warned = True
print(f" [voice-clone] watermarking unavailable ({exc}) — synthesised "
f"audio will NOT be marked. pip install audioseal", flush=True)
return None
def apply_watermark(wav_bytes: bytes, enabled: Optional[bool] = None) -> bytes:
"""Return the audio with an inaudible watermark, or unchanged if unavailable."""
if enabled is False or (enabled is None and not WATERMARK_DEFAULT):
return wav_bytes
model = _get_watermarker()
if model is None:
return wav_bytes
try:
import io as _io
import numpy as np
import soundfile as sf
import torch
data, sr = sf.read(_io.BytesIO(wav_bytes), dtype="float32")
if data.ndim > 1:
data = data.mean(axis=1)
tensor = torch.from_numpy(np.ascontiguousarray(data))[None, None, :]
with torch.no_grad():
marked = model(tensor, sample_rate=sr, alpha=1.0)
out = (tensor + marked).squeeze().cpu().numpy()
buf = _io.BytesIO()
sf.write(buf, out, sr, format="WAV")
return buf.getvalue()
except Exception as exc:
print(f" [voice-clone] watermarking failed ({exc}) — returning unmarked audio",
flush=True)
return wav_bytes
def detect_watermark(wav_bytes: bytes) -> dict:
"""Was this audio produced here? Returns {available, watermarked, confidence}."""
try:
import io as _io
import numpy as np
import soundfile as sf
import torch
from audioseal import AudioSeal
detector = AudioSeal.load_detector("audioseal_detector_16bits")
data, sr = sf.read(_io.BytesIO(wav_bytes), dtype="float32")
if data.ndim > 1:
data = data.mean(axis=1)
tensor = torch.from_numpy(np.ascontiguousarray(data))[None, None, :]
with torch.no_grad():
result, _msg = detector.detect_watermark(tensor, sample_rate=sr), None
score = float(result if isinstance(result, (int, float)) else result[0])
return {"available": True, "watermarked": score > 0.5, "confidence": round(score, 4)}
except ImportError:
return {"available": False, "watermarked": None,
"detail": "audioseal not installed"}
except Exception as exc:
return {"available": True, "watermarked": None, "detail": str(exc)}
# ---------------------------------------------------------------------------
# Cloning engines
#
# F5-TTS clones by CONTINUING a prompt: reference audio + its transcript, one
# model, so prosody and accent come along with the timbre. That is the best
# choice when the reference speaks the target language — and useless without a
# transcript.
#
# The alternatives split the job the way OpenVoice does — synthesise first, then
# move the timbre — which costs some prosody fidelity but needs no transcript and
# lets the language be chosen independently of the reference:
#
# xtts XTTS-v2 zero-shot from the clip (17 languages, no transcript)
# chain any base TTS -> seed-vc timbre transfer (OpenVoice's architecture,
# with seed-vc in place of its tone-colour converter)
# melotts MeloTTS as the base voice of that chain
# ---------------------------------------------------------------------------
CLONE_ENGINES = ("auto", "f5", "xtts", "chain", "melotts")
def _pick_engine(requested: str, ref_text: str, language: str, ref_language: str) -> str:
"""Resolve engine='auto' to a concrete engine, and say why."""
engine = (requested or "auto").strip().lower()
if engine not in CLONE_ENGINES:
raise HTTPException(status_code=400, detail=(
f"Unknown clone engine '{engine}'. Use one of: {', '.join(CLONE_ENGINES)}"))
if engine != "auto":
return engine
lang = (language or "").strip().lower()[:2]
ref_lang = (ref_language or "").strip().lower()[:2]
if not ref_text:
print(" [voice-clone] auto -> xtts (no transcript for the reference)", flush=True)
return "xtts"
if lang and ref_lang and lang != ref_lang:
print(f" [voice-clone] auto -> xtts (target '{lang}' differs from the "
f"reference's '{ref_lang}'; F5 would carry the accent over)", flush=True)
return "xtts"
return "f5"
_tts_backends: dict = {}
_tts_backend_lock = threading.Lock()
def _get_tts_backend(model_name: str):
"""Cached TTS backend, for the same reason F5 is cached: these are loaded per
call otherwise, and a dialog is many calls."""
with _tts_backend_lock:
be = _tts_backends.get(model_name)
if be is None:
from codai.api.tts_backends import load_backend
print(f" [voice-clone] loading TTS backend '{model_name}'", flush=True)
be = load_backend(model_name, None, {})
_tts_backends[model_name] = be
return be
def release_tts_backends() -> None:
"""Drop cached TTS backends (VRAM eviction hook)."""
with _tts_backend_lock:
if not _tts_backends:
return
_tts_backends.clear()
try:
import gc
import torch
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
except Exception:
pass
print(" [voice-clone] TTS backends released", flush=True)
def _seedvc_convert(wrapper, source_path: str, target_path: str,
diffusion_steps: int = 10):
"""Timbre transfer with seed-vc — the role OpenVoice gives its tone-colour
converter. Returns (audio, sample_rate)."""
out = wrapper.convert_voice(
source=source_path,
target=target_path,
diffusion_steps=diffusion_steps,
length_adjust=1.0,
inference_cfg_rate=0.7,
f0_condition=False,
pitch_shift=0,
stream_output=False,
)
if isinstance(out, tuple):
out = out[0]
import numpy as np
return np.asarray(out).flatten(), 22050 # f0_condition=False -> 22.05 kHz
def _xtts_clone(ref_audio_path: str, gen_text: str, speed: float = 1.0,
language: str = "en") -> bytes:
"""Zero-shot clone with XTTS-v2 — no transcript needed, language selectable."""
import io as _io
import numpy as np
import soundfile as sf
from codai.api.tts_backends import load_backend
backend = _get_tts_backend("coqui/XTTS-v2")
# XTTS treats a `voice` that is an existing file as the clone source.
wav, sr = backend.synthesize(gen_text, ref_audio_path, speed or 1.0,
(language or "en")[:2])
buf = _io.BytesIO()
sf.write(buf, np.asarray(wav, dtype="float32"), sr, format="WAV")
return buf.getvalue()
def _base_tts_wav(gen_text: str, speed: float, language: str, model: str,
temps: list) -> str:
"""Synthesise the line in a base voice; return a WAV path for conversion."""
import numpy as np
import soundfile as sf
backend = _get_tts_backend(model)
wav, sr = backend.synthesize(gen_text, "", speed or 1.0, (language or "en")[:2])
out = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
out.close()
sf.write(out.name, np.asarray(wav, dtype="float32"), sr)
temps.append(out.name)
return out.name
def _chain_clone(ref_audio_path: str, gen_text: str, speed: float, language: str,
base_model: str, temps: list) -> bytes:
"""OpenVoice's pattern: base TTS speaks it, seed-vc moves the timbre over.
seed-vc is a zero-shot converter, so like OpenVoice's tone-colour converter it
needs only the reference audio — no transcript, and the base voice decides the
language and delivery."""
import io as _io
import numpy as np
import soundfile as sf
src = _base_tts_wav(gen_text, speed, language, base_model, temps)
from codai.api.voice_convert import _get_wrapper
wrapper = _get_wrapper()
audio, sr = _seedvc_convert(wrapper, src, ref_audio_path)
buf = _io.BytesIO()
sf.write(buf, np.asarray(audio, dtype="float32"), sr, format="WAV")
return buf.getvalue()
def _save_audio_response(audio_bytes: bytes, http_request: Request) -> dict:
import uuid
filename = f"{uuid.uuid4().hex}.wav"
......@@ -269,6 +577,8 @@ class VoicePatchRequest(BaseModel):
description: Optional[str] = None
transcript: Optional[str] = None
audio: Optional[str] = None # replace reference audio (base64)
add_clips: Optional[list] = None # [{audio: b64, transcript: str}] — extra references
remove_clips: Optional[list] = None # 0-based indices into the clip list
model_config = ConfigDict(extra="allow")
......@@ -344,10 +654,42 @@ async def patch_voice(name: str, req: VoicePatchRequest):
f.write(audio_bytes)
meta['audio_file'] = audio_file
meta['audio_ext'] = ext
meta['clips'] = [{'path': audio_file, 'transcript': meta.get('transcript', '')}]
# Extra reference clips. One take is a gamble — a cough, a room, a bad mic and
# every cloned line inherits it. Several let the clone prompt on the cleanest,
# and give the averaging engines something to average.
if req.add_clips:
clips = _profile_clips(meta)
for i, c in enumerate(req.add_clips):
if not isinstance(c, dict) or not c.get('audio'):
continue
raw, ext = _decode_audio(c['audio'])
dest = os.path.join(vdir, f'clip_{int(time.time())}_{i:02d}{ext}')
with open(dest, 'wb') as fh:
fh.write(raw)
clips.append({'path': dest, 'transcript': (c.get('transcript') or '').strip()})
meta['clips'] = clips
if req.remove_clips:
clips = _profile_clips(meta)
drop = {int(i) for i in req.remove_clips if isinstance(i, (int, float, str)) and str(i).isdigit()}
kept = []
for i, c in enumerate(clips):
if i in drop:
try:
os.unlink(c['path'])
except OSError:
pass
continue
kept.append(c)
meta['clips'] = kept
if kept and meta.get('audio_file') not in [c['path'] for c in kept]:
meta['audio_file'] = kept[0]['path']
meta['transcript'] = kept[0].get('transcript') or meta.get('transcript', '')
with open(os.path.join(vdir, 'meta.json'), 'w') as f:
json.dump(meta, f)
return {"updated": True, "voice": meta}
return {"updated": True, "voice": meta, "clips": len(_profile_clips(meta))}
@router.get("/v1/audio/voices/{name}", summary="Get a voice profile")
......@@ -445,10 +787,61 @@ class VoiceCloneRequest(BaseModel):
ref_text: Optional[str] = None # transcript of ref_audio
speed: Optional[float] = 1.0
seed: Optional[int] = None
engine: Optional[str] = "auto" # auto | f5 | xtts | chain | melotts
language: Optional[str] = None # target language (xtts/chain/melotts)
ref_language: Optional[str] = None # language the reference speaks
base_model: Optional[str] = None # base voice for engine=chain
watermark: Optional[bool] = None # override the global watermark setting
response_format: Optional[str] = "url"
model_config = ConfigDict(extra="allow")
@router.post("/v1/audio/watermark/detect", summary="Detect a coderai audio watermark")
async def watermark_detect(request: Request):
"""Was this audio synthesised here? Accepts base64/data-URI/URL audio.
Answers the question the watermark exists for: given a clip in the wild, did it
come out of this server."""
body = await request.json()
data = body.get("audio") or body.get("file") or ""
if not data:
raise HTTPException(status_code=400, detail="Provide 'audio' (base64, data URI or URL)")
try:
raw = _decode_b64_or_url(data)
except Exception as exc:
raise HTTPException(status_code=400, detail=f"Could not decode audio: {exc}")
return detect_watermark(raw)
@router.get("/v1/audio/clone/engines", summary="List voice cloning engines")
async def list_clone_engines():
"""Which cloning engines this install can actually run, and what each needs."""
def _installed(mod: str) -> bool:
import importlib.util
try:
return importlib.util.find_spec(mod) is not None
except Exception:
return False
return {"default": "auto", "engines": [
{"id": "f5", "installed": _installed("f5_tts"),
"needs_transcript": True, "cross_lingual": False,
"detail": "In-context clone: copies timbre AND prosody/accent. Best when "
"the reference speaks the target language."},
{"id": "xtts", "installed": _installed("TTS"),
"needs_transcript": False, "cross_lingual": True,
"detail": "XTTS-v2 zero-shot from the clip, 17 languages."},
{"id": "chain", "installed": _installed("seed_vc"),
"needs_transcript": False, "cross_lingual": True,
"detail": f"Base TTS ({DEFAULT_CHAIN_BASE}) then seed-vc timbre transfer — "
f"OpenVoice's architecture with seed-vc as the converter."},
{"id": "melotts", "installed": _installed("melo") or _installed("MeloTTS"),
"needs_transcript": False, "cross_lingual": True,
"detail": "Same chain with MeloTTS as the base voice (OpenVoice V2's base)."},
], "watermark": {"default_on": WATERMARK_DEFAULT,
"available": _installed("audioseal")}}
@router.post("/v1/audio/clone", summary="Clone a voice / synthesize cloned speech")
async def clone_voice(request: VoiceCloneRequest, http_request: Request = None):
"""
......@@ -464,12 +857,15 @@ async def clone_voice(request: VoiceCloneRequest, http_request: Request = None):
temps = []
try:
ref_language = request.ref_language or ''
if request.voice_name:
meta = _load_voice(request.voice_name)
if not meta:
raise HTTPException(status_code=404, detail=f"Voice '{request.voice_name}' not found")
ref_audio_path = meta['audio_file']
ref_text = ref_text or meta.get('transcript', '')
clip = _best_clip(meta) or {}
ref_audio_path = clip.get('path') or meta['audio_file']
ref_text = ref_text or clip.get('transcript') or meta.get('transcript', '')
ref_language = ref_language or meta.get('language', '')
elif request.ref_audio:
audio_bytes, ext = _decode_audio(request.ref_audio)
tmp = tempfile.NamedTemporaryFile(suffix=ext, delete=False)
......@@ -480,25 +876,40 @@ async def clone_voice(request: VoiceCloneRequest, http_request: Request = None):
else:
raise HTTPException(status_code=400, detail="Provide voice_name or ref_audio")
if not ref_text:
raise HTTPException(status_code=400, detail="ref_text (transcript of reference audio) is required for voice cloning")
# A saved profile can hold minutes of audio; F5-TTS wants ~15 s of prompt.
# A saved profile can hold minutes of audio; every engine here prompts on
# a short clip.
ref_audio_path = _trim_reference(ref_audio_path, temps)
engine = _pick_engine(request.engine, ref_text, request.language, ref_language)
speed = request.speed or 1.0
lang = (request.language or ref_language or "en")[:2]
def _run():
if engine == "f5":
if not ref_text:
raise HTTPException(status_code=400, detail=(
"engine=f5 needs ref_text (the reference transcript). "
"Use engine=xtts or engine=chain to clone without one."))
return _f5tts_clone(ref_audio_path, ref_text, request.text, speed, request.seed)
if engine == "xtts":
return _xtts_clone(ref_audio_path, request.text, speed, lang)
base = request.base_model or ("melotts/MeloTTS" if engine == "melotts"
else DEFAULT_CHAIN_BASE)
return _chain_clone(ref_audio_path, request.text, speed, lang, base, temps)
try:
audio_bytes = await asyncio.get_event_loop().run_in_executor(
None, _f5tts_clone,
ref_audio_path, ref_text, request.text,
request.speed or 1.0, request.seed,
)
except ImportError:
raise HTTPException(status_code=501, detail="f5-tts not installed. Run: pip install f5-tts")
audio_bytes = await asyncio.get_event_loop().run_in_executor(None, _run)
except HTTPException:
raise
except ImportError as e:
raise HTTPException(status_code=501, detail=(
f"engine '{engine}' is not installed: {e}"))
except Exception as e:
raise HTTPException(status_code=500, detail=f"Voice cloning failed: {e}")
raise HTTPException(status_code=500, detail=f"Voice cloning failed ({engine}): {e}")
audio_bytes = apply_watermark(audio_bytes, request.watermark)
result = _save_audio_response(audio_bytes, http_request)
return {"created": int(time.time()), "data": [result]}
return {"created": int(time.time()), "data": [result], "engine": engine}
finally:
for t in temps:
......
# Reference projects
Outside projects we've studied, what each does that we don't, and what was taken
from them. Kept as a record because several of these are meant to seed **promo
tools** built on coderai — the point is to know what exists before rebuilding it.
Each entry says plainly whether it was *evaluated* (read the code / the paper) or
only *noted* (a link to look at later), so nothing here reads as a conclusion it
hasn't earned.
---
## OpenVoice — myshell-ai/OpenVoice · MIT · EVALUATED 2026-09-13
https://github.com/myshell-ai/OpenVoice · paper: arXiv:2312.01479
Instant voice cloning. Read from `openvoice/api.py`, the README and the paper
abstract.
**Architecture.** Two stages rather than one. `BaseSpeakerTTS.tts()` speaks the
text in a *base* voice with the chosen language and style (V2 uses **MeloTTS**:
English, Spanish, French, Chinese, Japanese, Korean). Then
`ToneColorConverter.convert(source, src_se, tgt_se, tau=0.3)` re-paints only the
timbre. The target embedding comes from `extract_se(audio_paths)` — **audio only,
no transcript anywhere in the pipeline** — and it averages across several clips.
`add_watermark()` / `detect_watermark()` mark the output.
**Why the split matters.** Language and delivery belong to the base TTS, timbre to
the converter, so cross-lingual cloning needs no multilingual data for the target
speaker. The cost is prosody: it copies how someone *sounds*, not how they *speak*.
**How it compares to what we run.** We clone with F5-TTS (in-context: reference
audio + its transcript, one model, so accent and prosody come along) and convert
with seed-vc. At the model level we were already ahead — F5-TTS over MeloTTS for
naturalness, XTTS-v2 over their base for language coverage (17 vs 6), seed-vc over
their tone-colour converter. What we lacked was the *wiring*, plus watermarking.
**Taken from it** (commits `45cc730`, and the engine work that followed):
- `engine` on `/v1/audio/clone`: `f5` | `xtts` | `chain` | `melotts`, plus `auto`,
which picks a transcript-free engine when the profile has no transcript or the
target language differs from the reference.
- `chain` = base TTS → seed-vc. That IS OpenVoice's architecture, with seed-vc in
place of its converter and any of our TTS engines as the base.
- MeloTTS itself, as an isolated-venv worker (`requirements-melotts.txt`,
`tools/melotts_service.py`, `codai/api/melotts_worker.py`) — so the exact
OpenVoice V2 pairing is reproducible here, and MeloTTS is also a TTS engine in
its own right.
- Multi-clip voice profiles: `PATCH /v1/audio/voices/{name}` takes `add_clips` /
`remove_clips`, and a clone prompts on the cleanest take (scored on clipping,
level and crest factor — deliberately not "loudest wins", since a clipped take
has the highest RMS and is the worst prompt).
- Watermarking, via **AudioSeal** (MIT, Meta) rather than their silentcipher:
on by default, `CODERAI_AUDIO_WATERMARK=0` or `watermark=false` to disable,
detection at `POST /v1/audio/watermark/detect`.
**Not taken.** Their base TTS as a default (F5/XTTS are better) and their
converter (seed-vc is newer and stronger).
**Still open.** True embedding *averaging* across clips — we pick the best clip
rather than average, because neither XTTS nor seed-vc exposes an averaged speaker
embedding. Worth revisiting if we ever store our own embeddings on the profile.
---
## MoneyPrinterTurbo — harry0703/MoneyPrinterTurbo · NOTED, not yet evaluated
https://github.com/harry0703/MoneyPrinterTurbo/blob/main/README-en.md
Automated short-video generation: script → stock/generated footage → subtitles →
TTS → assembled clip. Directly relevant to the promo-tool goal, and overlapping
with `tools/videogen.py`, `tools/video_editor.py` and `tools/character_studio.py`.
To check when we get to it: how it plans a whole video from one prompt, how it
sources and cuts footage, subtitle generation and burn-in, its TTS/voice pipeline,
and how its web UI is organised. Nothing here has been verified yet.
---
## Playtime-AI (Hugging Face) — EVALUATED 2026-09-10
https://huggingface.co/Playtime-AI
72 LoRA repos, catalogued via the HF API: 33 for LTX-2.3, 24 for MiniMax H3,
7 for Krea 2, plus Wan/Z-Image collections. Studied for **how** the LoRAs are
trained, which is not secret — two public recipes match the catalogue:
- **Stills path** (ai-toolkit, 12 GB VRAM): ~32 face crops, captions reduced to
`(trigger), 1girl` so identity collapses into the trigger token, 512×512,
rank/alpha 16, LR 1e-4 adamw8bit, 1000 steps, INT8 quantised transformer with
CPU offload, `flowmatch` + `timestep_type: shift`, `audio_loss_multiplier: 0.0`,
~6-7 h per LoRA.
- **Clips path** (fal's H3 trainer): 50-200 clips of 3-15 s at exactly 24.000 fps,
bucketed to multiples of 32, `frames % 17 == 5`, rank 16, 1500-5000 steps, audio
kept. Their finding: training resolution matters more than rank or step count.
Recorded because the *technique* is what we need for per-character identity LoRAs
(`codai/api/loras.py`, the character studio's `train` step). The catalogue itself
pairs named-celebrity likenesses with explicit concept LoRAs; we build the generic
trainer for characters we have rights to and don't reproduce that.
Related: [minimax-h3.md](minimax-h3.md) for the H3 facts gathered at the same time.
# Isolated venv for MeloTTS — the base speaker model OpenVoice V2 uses.
#
# MeloTTS pins an older transformers/tokenizers line and pulls unidic + a Japanese
# tokenizer chain, the same class of conflict that makes parler-tts unsafe to
# install into the shared venv (see requirements.txt). So it runs in its own venv:
# codai/api/melotts_worker.py builds it (baked at /opt/coderai/melotts_venv when
# present, else on demand under ~/.coderai/melotts_venv) and runs
# tools/melotts_service.py inside it.
# Do NOT merge these into requirements.txt.
melotts>=0.1.2
# MeloTTS resolves its dictionaries through cached_path at first use.
cached_path
# English g2p; the package pulls language-specific extras itself on first run.
txtsplit
soundfile>=0.12.0
numpy
......@@ -96,3 +96,7 @@ llama-cpp-python>=0.2.0
causal-conv1d
# flash-linear-attention requires CUDA and must be installed from source:
# pip install git+https://github.com/fla-org/flash-linear-attention --no-build-isolation
# Inaudible watermarking for synthesised speech (codai/api/voice_clone.py).
# MIT, Meta. Without it cloning still works, just unmarked.
audioseal>=0.1.4
......@@ -35,3 +35,7 @@ onnxruntime>=1.20.0 # CPU ONNX runtime (use onnxruntime-gpu for GPU a
# PDF rasterisation for the OCR subsystem (/v1/ocr); self-contained, no system poppler
pypdfium2>=4.20.0
# Inaudible watermarking for synthesised speech (codai/api/voice_clone.py).
# MIT, Meta. Without it cloning still works, just unmarked.
audioseal>=0.1.4
......@@ -128,3 +128,7 @@ onnxruntime-gpu>=1.20.0 # GPU-accelerated ONNX runtime for insightface
#
# If you see "No such file or directory: '0.0'" errors, you forgot to use quotes!
# The shell interprets >= as redirection. Fix: pip install "torch>=2.0.0" (with quotes)
# Inaudible watermarking for synthesised speech (codai/api/voice_clone.py).
# MIT, Meta. Without it cloning still works, just unmarked.
audioseal>=0.1.4
......@@ -24,6 +24,9 @@ Read the existing implementations before proposing changes to understand the cur
# TODO: review MoneyPrinterTurbo for ideas worth borrowing
(Tracked alongside the other reference projects in docs/reference-projects.md.)
https://github.com/harry0703/MoneyPrinterTurbo/blob/main/README-en.md
An automated short-video generation stack (script → stock/generated footage →
......@@ -36,6 +39,10 @@ is organised. Not yet evaluated — this entry is only a reminder to look.
# TODO: OpenVoice features missing from our voice cloning
**Update 2026-09-13: mostly DONE — see docs/reference-projects.md for the record.**
Remaining: true multi-clip embedding averaging (we pick the best clip instead).
https://github.com/myshell-ai/OpenVoice (MIT). We clone with F5-TTS
(`codai/api/voice_clone.py`, in-context: reference audio + its transcript) and
convert with seed-vc (`/v1/audio/convert`). OpenVoice splits the job — a base TTS
......
#!/usr/bin/env python3
# CoderAI - OpenAI-compatible API server
# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
"""MeloTTS speech service (runs inside the isolated MeloTTS venv).
MeloTTS is what OpenVoice V2 synthesises with before its tone-colour converter
re-paints the timbre. coderai uses it the same way — as the base voice of the
`chain` cloning engine — and as a plain multilingual TTS engine.
Same contract as tools/parler_tts_service.py so the shared remote backend in
codai/api/tts_backends.py can drive either:
GET /health -> {"ok": true, "model": …, "languages": […]}
POST /speak -> audio/wav (body: {text, voice, speed, language})
`voice` selects a speaker within the language pack (MeloTTS ships several per
language, e.g. EN-US / EN-BR / EN-Default); unknown names fall back to the first.
"""
import argparse
import io
import json
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import numpy as np
import soundfile as sf
ENGINE = None
#: MeloTTS ships one model per language; the request's `language` picks the pack.
LANGUAGES = ("EN", "ES", "FR", "ZH", "JP", "KR")
#: Two-letter codes the rest of coderai speaks -> MeloTTS language packs.
LANG_MAP = {"en": "EN", "es": "ES", "fr": "FR", "zh": "ZH", "ja": "JP",
"jp": "JP", "ko": "KR", "kr": "KR"}
class _Engine:
"""Lazily loads one TTS model per language and keeps them around."""
def __init__(self, device: str = "auto"):
self._device = device
self._models = {}
self.sr = 44100
def _model_for(self, language: str):
lang = LANG_MAP.get((language or "en").lower()[:2], "EN")
if lang not in self._models:
from melo.api import TTS
print(f"Loading MeloTTS language pack {lang} …", flush=True)
model = TTS(language=lang, device=self._device)
self._models[lang] = model
self.sr = int(getattr(model, "hps", None).data.sampling_rate)
return self._models[lang], lang
def speakers(self, language: str = "EN") -> list:
model, _ = self._model_for(language)
return list(getattr(model, "hps").data.spk2id.keys())
def speak(self, text: str, voice: str, speed: float, language: str) -> np.ndarray:
model, lang = self._model_for(language)
spk2id = getattr(model, "hps").data.spk2id
names = list(spk2id.keys())
name = voice if voice in spk2id else names[0]
buf = io.BytesIO()
# MeloTTS writes through soundfile; ask for a buffer rather than a path.
model.tts_to_file(text, spk2id[name], buf, speed=float(speed or 1.0),
format="WAV")
buf.seek(0)
data, sr = sf.read(buf, dtype="float32")
self.sr = int(sr)
return data
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def _send(self, code, body=b"", ctype="application/json"):
self.send_response(code)
self.send_header("Content-Type", ctype)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
try:
self.wfile.write(body)
except BrokenPipeError:
pass
def log_message(self, fmt, *args): # quieter logs
return
def do_GET(self):
if self.path.split("?")[0] == "/health":
self._send(200, json.dumps({
"ok": True, "model": "melotts", "languages": list(LANGUAGES),
"speakers": ENGINE.speakers("EN") if ENGINE else [],
}).encode())
else:
self._send(404, b'{"error":"not found"}')
def do_POST(self):
if self.path.split("?")[0] != "/speak":
self._send(404, b'{"error":"not found"}')
return
try:
length = int(self.headers.get("Content-Length") or 0)
body = json.loads(self.rfile.read(length) or b"{}")
text = (body.get("text") or "").strip()
if not text:
self._send(400, b'{"error":"text is required"}')
return
wav = ENGINE.speak(text, body.get("voice") or "",
body.get("speed") or 1.0,
body.get("language") or "en")
buf = io.BytesIO()
sf.write(buf, np.asarray(wav, dtype="float32"), ENGINE.sr, format="WAV")
self._send(200, buf.getvalue(), "audio/wav")
except Exception as exc:
self._send(500, json.dumps({"error": str(exc)}).encode())
def main(argv=None):
global ENGINE
ap = argparse.ArgumentParser(description="MeloTTS speech service")
ap.add_argument("--model", default="melotts", help="ignored; language picks the pack")
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, required=True)
ap.add_argument("--device", default="auto", help="auto | cpu | cuda")
args = ap.parse_args(argv)
ENGINE = _Engine(args.device)
print(f"MeloTTS service on http://{args.host}:{args.port} "
f"(languages: {', '.join(LANGUAGES)})", flush=True)
ThreadingHTTPServer((args.host, args.port), Handler).serve_forever()
return 0
if __name__ == "__main__":
sys.exit(main())
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment