voice: cloning engines, multi-clip profiles, watermarking, MeloTTS

Everything OpenVoice does that we didn't, built from models we already run rather
than adopting theirs — plus MeloTTS itself so their exact V2 pairing is available.

Engines on /v1/audio/clone (`engine`, default `auto`):
  f5       today's in-context clone — timbre AND prosody, needs the transcript
  xtts     XTTS-v2 zero-shot from the clip, 17 languages, no transcript
  chain    base TTS -> seed-vc timbre transfer: OpenVoice's architecture with
           seed-vc in place of its tone-colour converter
  melotts  the same chain with MeloTTS as the base voice
`auto` picks a transcript-free engine when the profile has no transcript or the
target language differs from the reference — the case where F5 carries the wrong
accent across. GET /v1/audio/clone/engines reports what is installed.

The transcript is no longer a hard requirement: it is now checked per engine
instead of rejecting the request outright.

Multi-clip profiles: PATCH /v1/audio/voices/{name} takes add_clips/remove_clips,
and a clone prompts on the cleanest take. Scored on clipping, level and crest
factor — explicitly NOT loudest-wins, since a clipped take has the highest RMS of
the set and is the worst possible prompt. Single-clip profiles keep working.

Watermarking with AudioSeal (MIT), on by default, per-request or global override,
detection at POST /v1/audio/watermark/detect. Missing audioseal degrades to
unmarked audio with one warning, never an error.

MeloTTS runs in an isolated venv worker like parler-tts — its pins conflict with
this stack the same way. _RemoteParlerBackend generalised to _RemoteSpeechBackend
since both speak the same POST /speak contract.
Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
parent 1d5ee189
# CoderAI - OpenAI-compatible API server
# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
"""Fully-managed MeloTTS worker.
MeloTTS is the base speaker model OpenVoice V2 synthesises with before its
tone-colour converter re-paints the timbre. coderai uses it the same way: as the
base voice of the `chain` cloning engine (MeloTTS speaks the line, seed-vc moves
the timbre over), and as a plain multilingual TTS engine in its own right —
English, Spanish, French, Chinese, Japanese and Korean.
Its pins (an older transformers/tokenizers line, unidic, a Japanese tokenizer
chain) conflict with this server's stack exactly the way parler-tts's do, so it
runs in an ISOLATED venv and is reached over HTTP. This module owns the venv and
the process; the synthesis lives in tools/melotts_service.py.
"""
import os
import socket
import subprocess
import sys
import threading
import time
from pathlib import Path
_REPO_ROOT = Path(__file__).resolve().parents[2]
_SERVICE_SCRIPT = _REPO_ROOT / "tools" / "melotts_service.py"
# Dedicated venv for the (incompatible) melotts stack. Created with access to
# the base interpreter's packages so torch/numpy aren't re-downloaded; melotts's
# pinned transformers installs into the venv and shadows the system one.
_VENV_DIR = Path(os.environ.get("CODERAI_MELOTTS_VENV")
or os.path.expanduser("~/.coderai/melotts_venv"))
_lock = threading.RLock()
_services: dict[str, dict] = {} # model_name -> {"proc","port","url"}
_bootstrapped = False
def _venv_python() -> Path:
return _VENV_DIR / ("Scripts" if os.name == "nt" else "bin") / (
"python.exe" if os.name == "nt" else "python")
def _pip_ok(py: Path) -> bool:
try:
return subprocess.run([str(py), "-c", "import melo, soundfile"],
capture_output=True).returncode == 0
except Exception:
return False
def _venv_is_system_site() -> bool:
"""True if the venv was built with --system-site-packages (can't isolate)."""
try:
return "include-system-site-packages = true" in \
(_VENV_DIR / "pyvenv.cfg").read_text().lower()
except Exception:
return False
def _bootstrap_venv() -> Path:
"""Create a fully-isolated venv and install melotts (idempotent).
Isolation is the whole point: melotts pins an old transformers/tokenizers
that must NOT be shared with — or shadowed by — the server's stack, so the
venv gets its own copy of everything (torch included). Returns its python."""
global _bootstrapped
py = _venv_python()
if _bootstrapped and py.exists():
return py
# A previously-created shared-site venv leaks the server's transformers in;
# rebuild it isolated.
if py.exists() and _venv_is_system_site():
import shutil
print("[melotts] rebuilding venv as fully isolated …", flush=True)
shutil.rmtree(_VENV_DIR, ignore_errors=True)
if not _venv_python().exists():
print(f"[melotts] creating isolated venv at {_VENV_DIR} …", flush=True)
_VENV_DIR.parent.mkdir(parents=True, exist_ok=True)
subprocess.run([sys.executable, "-m", "venv", str(_VENV_DIR)], check=True)
py = _venv_python()
if not _pip_ok(py):
print("[melotts] installing melotts + torch into the isolated venv "
"(first run, downloads several GB, this can take a while) …", flush=True)
subprocess.run([str(py), "-m", "pip", "install",
"git+https://github.com/huggingface/melotts.git",
"soundfile"], check=True)
if not _pip_ok(py):
raise RuntimeError("melotts install did not yield an importable package")
_bootstrapped = True
return py
def _free_port() -> int:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
def _pump_logs(proc: subprocess.Popen, tail):
for line in proc.stdout:
line = line.rstrip()
if line:
tail.append(line)
print(f"[melotts] {line}", flush=True)
def _health_ok(url: str) -> bool:
import requests
try:
r = requests.get(url + "/health", timeout=3)
return r.ok and bool(r.json().get("ok"))
except Exception:
return False
def ensure_service(model_name: str, ready_timeout: float = 1800.0) -> str:
"""Start (or reuse) the worker for ``model_name`` and return its base URL.
First call bootstraps the venv and downloads the model, so the timeout is
generous. Raises RuntimeError if the service never comes up."""
with _lock:
svc = _services.get(model_name)
if svc and svc["proc"].poll() is None and _health_ok(svc["url"]):
return svc["url"]
if svc and svc["proc"].poll() is not None:
_services.pop(model_name, None) # died — restart below
py = _bootstrap_venv()
port = _free_port()
url = f"http://127.0.0.1:{port}"
env = dict(os.environ)
# The worker must use the model already pulled via coderai's HF download
# interface — it never downloads anything itself. Point it at coderai's
# cache and force offline mode, so a missing model fails fast instead of
# silently fetching.
try:
from codai.models.cache import get_hf_hub_cache_dir
hub = get_hf_hub_cache_dir()
env["HF_HUB_CACHE"] = hub
env["HUGGINGFACE_HUB_CACHE"] = hub
except Exception:
pass
env["HF_HUB_OFFLINE"] = "1"
env["TRANSFORMERS_OFFLINE"] = "1"
proc = subprocess.Popen(
[str(py), str(_SERVICE_SCRIPT), "--model", model_name,
"--host", "127.0.0.1", "--port", str(port)],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
bufsize=1, env=env, cwd=str(_REPO_ROOT),
)
import collections
tail = collections.deque(maxlen=15)
threading.Thread(target=_pump_logs, args=(proc, tail), daemon=True).start()
_services[model_name] = {"proc": proc, "port": port, "url": url}
def _tail_msg():
joined = " | ".join(list(tail)[-5:]).strip()
if "offline" in joined.lower() or "not" in joined.lower() and "found" in joined.lower():
return (f". The model isn't in coderai's cache — download "
f"'{model_name}' from the model interface first. ({joined})")
return f". Last output: {joined}" if joined else ""
# Wait (outside the lock) for the service to load the model and answer.
deadline = time.time() + ready_timeout
while time.time() < deadline:
if proc.poll() is not None:
raise RuntimeError(
f"MeloTTS worker exited (code {proc.returncode}) before becoming ready"
+ _tail_msg())
if _health_ok(url):
print(f"[melotts] service ready for {model_name} at {url}", flush=True)
return url
time.sleep(2)
stop_service(model_name)
raise RuntimeError(f"MeloTTS worker for {model_name} did not become ready in time"
+ _tail_msg())
def stop_service(model_name: str) -> None:
with _lock:
svc = _services.pop(model_name, None)
if not svc:
return
proc = svc["proc"]
if proc.poll() is None:
try:
proc.terminate()
proc.wait(timeout=10)
except Exception:
pass
if proc.poll() is None:
try:
proc.kill()
except Exception:
pass
print(f"[melotts] service for {model_name} stopped", flush=True)
def stop_all() -> None:
for name in list(_services.keys()):
stop_service(name)
import atexit as _atexit
_atexit.register(stop_all)
......@@ -50,6 +50,8 @@ def _family(model_name: str) -> str:
return "parler"
if "bark" in n:
return "bark"
if "melo" in n:
return "melo"
return "transformers"
......@@ -376,11 +378,16 @@ class _BarkBackend:
# (parler-tts pins an old transformers that conflicts with this server's stack).
# --------------------------------------------------------------------------- #
class _RemoteParlerBackend:
family = "parler"
class _RemoteSpeechBackend:
"""A TTS engine that runs in its own venv and answers over HTTP.
Used for the engines whose dependencies can't share this venv: parler-tts and
MeloTTS. Both expose the same tiny contract — POST /speak -> WAV bytes."""
default_voice = ""
def __init__(self, config: dict, managed_model: Optional[str] = None):
def __init__(self, config: dict, family: str = "parler",
managed_model: Optional[str] = None):
self.family = family
self._cfg = config or {}
self._url = str(self._cfg["service_url"]).rstrip("/")
# When coderai launched the worker itself, remember the model so the
......@@ -426,16 +433,26 @@ def load_backend(model_name: str, model_path: Optional[str], config: Optional[di
return _CoquiBackend(model_name, config)
if fam == "bark":
return _BarkBackend(model_name, config)
if fam == "melo":
# MeloTTS is OpenVoice V2's base speaker model. Its pins conflict with this
# stack the way parler's do, so it runs in an isolated venv worker and is
# reached over HTTP — same contract, same managed lifecycle.
if config.get("service_url"):
return _RemoteSpeechBackend(config, family="melo")
from codai.api import melotts_worker
url = melotts_worker.ensure_service(model_name)
return _RemoteSpeechBackend({**config, "service_url": url}, family="melo",
managed_model=model_name)
if fam == "parler":
# An explicit service_url points at an externally-run service. Otherwise
# coderai fully manages the worker: bootstrap its venv, spawn it, and
# route to it — no manual setup needed.
if config.get("service_url"):
return _RemoteParlerBackend(config)
return _RemoteSpeechBackend(config, family="parler")
from codai.api import parler_worker
url = parler_worker.ensure_service(model_name)
return _RemoteParlerBackend({**config, "service_url": url},
managed_model=model_name)
return _RemoteSpeechBackend({**config, "service_url": url},
family="parler", managed_model=model_name)
return _TransformersBackend(model_name, config)
......
This diff is collapsed.
# Reference projects
Outside projects we've studied, what each does that we don't, and what was taken
from them. Kept as a record because several of these are meant to seed **promo
tools** built on coderai — the point is to know what exists before rebuilding it.
Each entry says plainly whether it was *evaluated* (read the code / the paper) or
only *noted* (a link to look at later), so nothing here reads as a conclusion it
hasn't earned.
---
## OpenVoice — myshell-ai/OpenVoice · MIT · EVALUATED 2026-09-13
https://github.com/myshell-ai/OpenVoice · paper: arXiv:2312.01479
Instant voice cloning. Read from `openvoice/api.py`, the README and the paper
abstract.
**Architecture.** Two stages rather than one. `BaseSpeakerTTS.tts()` speaks the
text in a *base* voice with the chosen language and style (V2 uses **MeloTTS**:
English, Spanish, French, Chinese, Japanese, Korean). Then
`ToneColorConverter.convert(source, src_se, tgt_se, tau=0.3)` re-paints only the
timbre. The target embedding comes from `extract_se(audio_paths)` — **audio only,
no transcript anywhere in the pipeline** — and it averages across several clips.
`add_watermark()` / `detect_watermark()` mark the output.
**Why the split matters.** Language and delivery belong to the base TTS, timbre to
the converter, so cross-lingual cloning needs no multilingual data for the target
speaker. The cost is prosody: it copies how someone *sounds*, not how they *speak*.
**How it compares to what we run.** We clone with F5-TTS (in-context: reference
audio + its transcript, one model, so accent and prosody come along) and convert
with seed-vc. At the model level we were already ahead — F5-TTS over MeloTTS for
naturalness, XTTS-v2 over their base for language coverage (17 vs 6), seed-vc over
their tone-colour converter. What we lacked was the *wiring*, plus watermarking.
**Taken from it** (commits `45cc730`, and the engine work that followed):
- `engine` on `/v1/audio/clone`: `f5` | `xtts` | `chain` | `melotts`, plus `auto`,
which picks a transcript-free engine when the profile has no transcript or the
target language differs from the reference.
- `chain` = base TTS → seed-vc. That IS OpenVoice's architecture, with seed-vc in
place of its converter and any of our TTS engines as the base.
- MeloTTS itself, as an isolated-venv worker (`requirements-melotts.txt`,
`tools/melotts_service.py`, `codai/api/melotts_worker.py`) — so the exact
OpenVoice V2 pairing is reproducible here, and MeloTTS is also a TTS engine in
its own right.
- Multi-clip voice profiles: `PATCH /v1/audio/voices/{name}` takes `add_clips` /
`remove_clips`, and a clone prompts on the cleanest take (scored on clipping,
level and crest factor — deliberately not "loudest wins", since a clipped take
has the highest RMS and is the worst prompt).
- Watermarking, via **AudioSeal** (MIT, Meta) rather than their silentcipher:
on by default, `CODERAI_AUDIO_WATERMARK=0` or `watermark=false` to disable,
detection at `POST /v1/audio/watermark/detect`.
**Not taken.** Their base TTS as a default (F5/XTTS are better) and their
converter (seed-vc is newer and stronger).
**Still open.** True embedding *averaging* across clips — we pick the best clip
rather than average, because neither XTTS nor seed-vc exposes an averaged speaker
embedding. Worth revisiting if we ever store our own embeddings on the profile.
---
## MoneyPrinterTurbo — harry0703/MoneyPrinterTurbo · NOTED, not yet evaluated
https://github.com/harry0703/MoneyPrinterTurbo/blob/main/README-en.md
Automated short-video generation: script → stock/generated footage → subtitles →
TTS → assembled clip. Directly relevant to the promo-tool goal, and overlapping
with `tools/videogen.py`, `tools/video_editor.py` and `tools/character_studio.py`.
To check when we get to it: how it plans a whole video from one prompt, how it
sources and cuts footage, subtitle generation and burn-in, its TTS/voice pipeline,
and how its web UI is organised. Nothing here has been verified yet.
---
## Playtime-AI (Hugging Face) — EVALUATED 2026-09-10
https://huggingface.co/Playtime-AI
72 LoRA repos, catalogued via the HF API: 33 for LTX-2.3, 24 for MiniMax H3,
7 for Krea 2, plus Wan/Z-Image collections. Studied for **how** the LoRAs are
trained, which is not secret — two public recipes match the catalogue:
- **Stills path** (ai-toolkit, 12 GB VRAM): ~32 face crops, captions reduced to
`(trigger), 1girl` so identity collapses into the trigger token, 512×512,
rank/alpha 16, LR 1e-4 adamw8bit, 1000 steps, INT8 quantised transformer with
CPU offload, `flowmatch` + `timestep_type: shift`, `audio_loss_multiplier: 0.0`,
~6-7 h per LoRA.
- **Clips path** (fal's H3 trainer): 50-200 clips of 3-15 s at exactly 24.000 fps,
bucketed to multiples of 32, `frames % 17 == 5`, rank 16, 1500-5000 steps, audio
kept. Their finding: training resolution matters more than rank or step count.
Recorded because the *technique* is what we need for per-character identity LoRAs
(`codai/api/loras.py`, the character studio's `train` step). The catalogue itself
pairs named-celebrity likenesses with explicit concept LoRAs; we build the generic
trainer for characters we have rights to and don't reproduce that.
Related: [minimax-h3.md](minimax-h3.md) for the H3 facts gathered at the same time.
# Isolated venv for MeloTTS — the base speaker model OpenVoice V2 uses.
#
# MeloTTS pins an older transformers/tokenizers line and pulls unidic + a Japanese
# tokenizer chain, the same class of conflict that makes parler-tts unsafe to
# install into the shared venv (see requirements.txt). So it runs in its own venv:
# codai/api/melotts_worker.py builds it (baked at /opt/coderai/melotts_venv when
# present, else on demand under ~/.coderai/melotts_venv) and runs
# tools/melotts_service.py inside it.
# Do NOT merge these into requirements.txt.
melotts>=0.1.2
# MeloTTS resolves its dictionaries through cached_path at first use.
cached_path
# English g2p; the package pulls language-specific extras itself on first run.
txtsplit
soundfile>=0.12.0
numpy
......@@ -96,3 +96,7 @@ llama-cpp-python>=0.2.0
causal-conv1d
# flash-linear-attention requires CUDA and must be installed from source:
# pip install git+https://github.com/fla-org/flash-linear-attention --no-build-isolation
# Inaudible watermarking for synthesised speech (codai/api/voice_clone.py).
# MIT, Meta. Without it cloning still works, just unmarked.
audioseal>=0.1.4
......@@ -35,3 +35,7 @@ onnxruntime>=1.20.0 # CPU ONNX runtime (use onnxruntime-gpu for GPU a
# PDF rasterisation for the OCR subsystem (/v1/ocr); self-contained, no system poppler
pypdfium2>=4.20.0
# Inaudible watermarking for synthesised speech (codai/api/voice_clone.py).
# MIT, Meta. Without it cloning still works, just unmarked.
audioseal>=0.1.4
......@@ -128,3 +128,7 @@ onnxruntime-gpu>=1.20.0 # GPU-accelerated ONNX runtime for insightface
#
# If you see "No such file or directory: '0.0'" errors, you forgot to use quotes!
# The shell interprets >= as redirection. Fix: pip install "torch>=2.0.0" (with quotes)
# Inaudible watermarking for synthesised speech (codai/api/voice_clone.py).
# MIT, Meta. Without it cloning still works, just unmarked.
audioseal>=0.1.4
......@@ -24,6 +24,9 @@ Read the existing implementations before proposing changes to understand the cur
# TODO: review MoneyPrinterTurbo for ideas worth borrowing
(Tracked alongside the other reference projects in docs/reference-projects.md.)
https://github.com/harry0703/MoneyPrinterTurbo/blob/main/README-en.md
An automated short-video generation stack (script → stock/generated footage →
......@@ -36,6 +39,10 @@ is organised. Not yet evaluated — this entry is only a reminder to look.
# TODO: OpenVoice features missing from our voice cloning
**Update 2026-09-13: mostly DONE — see docs/reference-projects.md for the record.**
Remaining: true multi-clip embedding averaging (we pick the best clip instead).
https://github.com/myshell-ai/OpenVoice (MIT). We clone with F5-TTS
(`codai/api/voice_clone.py`, in-context: reference audio + its transcript) and
convert with seed-vc (`/v1/audio/convert`). OpenVoice splits the job — a base TTS
......
#!/usr/bin/env python3
# CoderAI - OpenAI-compatible API server
# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
"""MeloTTS speech service (runs inside the isolated MeloTTS venv).
MeloTTS is what OpenVoice V2 synthesises with before its tone-colour converter
re-paints the timbre. coderai uses it the same way — as the base voice of the
`chain` cloning engine — and as a plain multilingual TTS engine.
Same contract as tools/parler_tts_service.py so the shared remote backend in
codai/api/tts_backends.py can drive either:
GET /health -> {"ok": true, "model": …, "languages": […]}
POST /speak -> audio/wav (body: {text, voice, speed, language})
`voice` selects a speaker within the language pack (MeloTTS ships several per
language, e.g. EN-US / EN-BR / EN-Default); unknown names fall back to the first.
"""
import argparse
import io
import json
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import numpy as np
import soundfile as sf
ENGINE = None
#: MeloTTS ships one model per language; the request's `language` picks the pack.
LANGUAGES = ("EN", "ES", "FR", "ZH", "JP", "KR")
#: Two-letter codes the rest of coderai speaks -> MeloTTS language packs.
LANG_MAP = {"en": "EN", "es": "ES", "fr": "FR", "zh": "ZH", "ja": "JP",
"jp": "JP", "ko": "KR", "kr": "KR"}
class _Engine:
"""Lazily loads one TTS model per language and keeps them around."""
def __init__(self, device: str = "auto"):
self._device = device
self._models = {}
self.sr = 44100
def _model_for(self, language: str):
lang = LANG_MAP.get((language or "en").lower()[:2], "EN")
if lang not in self._models:
from melo.api import TTS
print(f"Loading MeloTTS language pack {lang} …", flush=True)
model = TTS(language=lang, device=self._device)
self._models[lang] = model
self.sr = int(getattr(model, "hps", None).data.sampling_rate)
return self._models[lang], lang
def speakers(self, language: str = "EN") -> list:
model, _ = self._model_for(language)
return list(getattr(model, "hps").data.spk2id.keys())
def speak(self, text: str, voice: str, speed: float, language: str) -> np.ndarray:
model, lang = self._model_for(language)
spk2id = getattr(model, "hps").data.spk2id
names = list(spk2id.keys())
name = voice if voice in spk2id else names[0]
buf = io.BytesIO()
# MeloTTS writes through soundfile; ask for a buffer rather than a path.
model.tts_to_file(text, spk2id[name], buf, speed=float(speed or 1.0),
format="WAV")
buf.seek(0)
data, sr = sf.read(buf, dtype="float32")
self.sr = int(sr)
return data
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def _send(self, code, body=b"", ctype="application/json"):
self.send_response(code)
self.send_header("Content-Type", ctype)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
try:
self.wfile.write(body)
except BrokenPipeError:
pass
def log_message(self, fmt, *args): # quieter logs
return
def do_GET(self):
if self.path.split("?")[0] == "/health":
self._send(200, json.dumps({
"ok": True, "model": "melotts", "languages": list(LANGUAGES),
"speakers": ENGINE.speakers("EN") if ENGINE else [],
}).encode())
else:
self._send(404, b'{"error":"not found"}')
def do_POST(self):
if self.path.split("?")[0] != "/speak":
self._send(404, b'{"error":"not found"}')
return
try:
length = int(self.headers.get("Content-Length") or 0)
body = json.loads(self.rfile.read(length) or b"{}")
text = (body.get("text") or "").strip()
if not text:
self._send(400, b'{"error":"text is required"}')
return
wav = ENGINE.speak(text, body.get("voice") or "",
body.get("speed") or 1.0,
body.get("language") or "en")
buf = io.BytesIO()
sf.write(buf, np.asarray(wav, dtype="float32"), ENGINE.sr, format="WAV")
self._send(200, buf.getvalue(), "audio/wav")
except Exception as exc:
self._send(500, json.dumps({"error": str(exc)}).encode())
def main(argv=None):
global ENGINE
ap = argparse.ArgumentParser(description="MeloTTS speech service")
ap.add_argument("--model", default="melotts", help="ignored; language picks the pack")
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, required=True)
ap.add_argument("--device", default="auto", help="auto | cpu | cuda")
args = ap.parse_args(argv)
ENGINE = _Engine(args.device)
print(f"MeloTTS service on http://{args.host}:{args.port} "
f"(languages: {', '.join(LANGUAGES)})", flush=True)
ThreadingHTTPServer((args.host, args.port), Handler).serve_forever()
return 0
if __name__ == "__main__":
sys.exit(main())
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment