tools: make every client tool work over the API alone

Audit for detaching the bundled tools into their own containers, talking to
coderai only through the HTTP API (aisbf proxying 1:1, assumed).

One coupling existed: video_editor.py imported codai.api.tts_backends to fill its
emotion and style pickers, which only works while it shares a process tree with
the server. Added GET /v1/audio/speech/capabilities and the editor now asks over
HTTP, falling back to the local import on an older server and to empty lists when
neither answers — the pickers simply don't appear rather than the tool breaking.

Everything else was already clean: no tool sends a local path into the API
expecting the server to read it (media goes base64/multipart), none reads coderai's
own state (models.json, /AI, ~/.coderai), and generated media is always fetched
over HTTP rather than from a shared filesystem.

docs/tools-as-services.md records the audit, what each container would need
(ffmpeg, own volumes, base-url/api-key, long proxy timeouts, large bodies) and
which tools are not API clients at all — review_outputs.py reviews the tools' own
output dirs, and the *_service.py workers are coderai's own subprocesses.
Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
parent 461e9690
...@@ -79,6 +79,25 @@ class TTSResponse(BaseModel): ...@@ -79,6 +79,25 @@ class TTSResponse(BaseModel):
model_config = ConfigDict(extra="allow") model_config = ConfigDict(extra="allow")
@router.get("/v1/audio/speech/capabilities", summary="What a TTS model can steer")
async def speech_capabilities(model: str = ""):
"""Emotions and delivery styles the given TTS model supports.
Exposed over HTTP because the bundled tools need it: the video editor used to
import codai.api.tts_backends directly to populate its emotion/style pickers,
which only works while the tool shares a process tree with the server. Over the
API it works from anywhere — including a tool running in its own container.
"""
from codai.api import tts_backends
name = (model or "").strip()
return {
"model": name,
"family": tts_backends._family(name),
"emotions": tts_backends.family_emotions(name),
"styles": tts_backends.family_styles(name),
}
@router.post("/v1/audio/speech", summary="Text-to-speech synthesis") @router.post("/v1/audio/speech", summary="Text-to-speech synthesis")
async def create_speech(request: TTSRequest, http_request: Request = None): async def create_speech(request: TTSRequest, http_request: Request = None):
""" """
......
# Running the bundled tools as separate services
Audit of whether each bundled tool can run **detached** — its own container,
reaching coderai only over the HTTP API (assumed proxied 1:1 by aisbf).
Verdict: **every client tool is detachable.** One coupling existed and is now
fixed; the rest is deployment plumbing (volumes, ffmpeg, timeouts).
## Client tools — detachable
| tool | reaches coderai via | needs |
|---|---|---|
| `character_studio.py` | `--base-url` / `CODERAI_BASE_URL`, `/v1/*` only | ffmpeg, own out-dir volume |
| `videogen.py` | same | ffmpeg, own out-dir volume |
| `video_dubber.py` | same (multipart uploads) | ffmpeg + ffprobe |
| `gen_township_fighters.py` | same | ffmpeg, own out-dir volume |
| `video_editor.py` | same | ffmpeg, own media-dir + output-dir volumes |
| `manual_multimodal_test_client.py` | `--url` | — |
Checked for, and clean:
- **No tool imports `codai.*` any more.** `video_editor.py` used to import
`codai.api.tts_backends.family_emotions/family_styles` to fill its emotion and
style pickers — the one thing that only worked inside the server's process tree.
It now asks `GET /v1/audio/speech/capabilities` (added for this), falling back to
the local import when the server is older, and to empty lists when neither works
— the pickers just don't appear.
- **No tool sends a local path into the API** expecting the server to open it.
Media goes as base64 or multipart. The `path` fields that do appear are the
tools' own paths, returned to their own browser UIs.
- **No tool reads coderai's own state** — no `models.json`, no `/AI`, no
`~/.coderai`. (`gen_township_fighters.py` mentions `models.json` only in
`--help` text; `video_editor.py`'s `/cache/` is its own config path.)
- **Generated media is fetched over HTTP.** The clients resolve `b64_json` /
`b64_mp4` / URLs / `/v1/...` paths, never a filesystem path from the server.
## Not API clients (different class)
- `review_outputs.py` — a review UI over the *tools'* output directories, not a
coderai client. Detaching it means mounting those output volumes, not proxying.
- `*_service.py` (`canary`, `crisperwhisper`, `parler_tts`, `pyannote`, `melotts`,
`h3`), `h3_common.py`, `vpr_salad_server.py` — these are coderai's own
subprocesses, launched inside its isolated venvs by the matching `*_worker.py`.
They are not detachable tools and must stay with the server.
## What each container needs
1. **ffmpeg + ffprobe** on PATH — every media tool shells out to them.
2. **Its own writable volume** for output; `video_editor.py` also needs its
`media_dir` (its source library, which today is a local directory).
3. **`CODERAI_BASE_URL` + `CODERAI_API_KEY`** — all of them accept both as flags
and as env vars.
4. **Generous proxy timeouts.** Video generation and LoRA training are single
requests that run for minutes; the studio and videogen poll progress endpoints,
but the generation call itself blocks. The nginx block for `/character/` uses
`proxy_read_timeout 3600s` — detached tools need the same on their own routes.
5. **Large request bodies.** Uploads are base64 or multipart; the container nginx
already allows 4 GB globally, and the `/character/` proxy on lisa sets
`client_max_body_size 1024m` with `proxy_request_buffering off`.
6. **Sub-path awareness, if proxied under a prefix.** `character_studio.py` and
`video_editor.py` honour `X-Forwarded-Prefix`; `videogen.py`,
`gen_township_fighters.py` and `review_outputs.py` assume they own `/` (see
reverse-proxy-nginx.md). Detaching those three behind a prefix needs the same
`ROOT_PATH` treatment the character studio got.
## Not verified
The assumption that aisbf proxies the coderai API 1:1 was taken as given, not
tested. Nothing here has been run as an actual separate container either — this is
a source-level audit plus the one fix it turned up.
...@@ -82,25 +82,53 @@ DEFAULT_VOICE = {"feminine": "af_sarah", "masculine": "am_michael"} ...@@ -82,25 +82,53 @@ DEFAULT_VOICE = {"feminine": "af_sarah", "masculine": "am_michael"}
SPEED_FACTORS = [2, 4, 8, 16, 32, 64, 128, 240] SPEED_FACTORS = [2, 4, 8, 16, 32, 64, 128, 240]
def tts_emotions(model: str) -> list: _TTS_CAPS_CACHE: dict = {}
"""Emotions the configured TTS model can steer, or [] when unavailable.
Looked up from the CoderAI backend's family map when importable; the editor def _tts_caps(model: str, cfg=None) -> dict:
only offers an emotion picker when this is non-empty.""" """Emotions/styles for a TTS model, asked of the CoderAI API.
try:
from codai.api.tts_backends import family_emotions Over HTTP rather than by importing codai.api.tts_backends, so the editor works
return family_emotions(model or "") when it runs somewhere else entirely — its own container, talking to coderai
except Exception: through a proxy. Falls back to the local import when the endpoint isn't there
return [] (older server), and to {} when neither works: the pickers simply don't appear.
"""
key = model or ""
def tts_styles(model: str) -> list: base = (getattr(cfg, "base_url", "") or "").rstrip("/")
cache_key = (base, key)
if cache_key in _TTS_CAPS_CACHE:
return _TTS_CAPS_CACHE[cache_key]
caps = {}
if base:
try:
import requests
api_key = getattr(cfg, "api_key", "") or ""
r = requests.get(f"{base}/v1/audio/speech/capabilities",
params={"model": key},
headers={"Authorization": f"Bearer {api_key}"} if api_key else {},
timeout=15)
if r.ok:
caps = r.json() or {}
except Exception:
caps = {}
if not caps:
try: # same process as the server (bundled deployment)
from codai.api.tts_backends import family_emotions, family_styles
caps = {"emotions": family_emotions(key), "styles": family_styles(key)}
except Exception:
caps = {}
_TTS_CAPS_CACHE[cache_key] = caps
return caps
def tts_emotions(model: str, cfg=None) -> list:
"""Emotions the configured TTS model can steer, or [] when unavailable."""
return list(_tts_caps(model, cfg).get("emotions") or [])
def tts_styles(model: str, cfg=None) -> list:
"""Delivery styles (whisper/shout/tone/…) the model can steer, or [].""" """Delivery styles (whisper/shout/tone/…) the model can steer, or []."""
try: return list(_tts_caps(model, cfg).get("styles") or [])
from codai.api.tts_backends import family_styles
return family_styles(model or "")
except Exception:
return []
# --------------------------------------------------------------------------- # # --------------------------------------------------------------------------- #
...@@ -1431,8 +1459,8 @@ def make_handler(editor: Editor): ...@@ -1431,8 +1459,8 @@ def make_handler(editor: Editor):
"voices": VOICES, "voices": VOICES,
"speedFactors": SPEED_FACTORS, "speedFactors": SPEED_FACTORS,
"ttsModel": cfg.tts_model, "ttsModel": cfg.tts_model,
"ttsEmotions": tts_emotions(cfg.tts_model), "ttsEmotions": tts_emotions(cfg.tts_model, cfg),
"ttsStyles": tts_styles(cfg.tts_model), "ttsStyles": tts_styles(cfg.tts_model, cfg),
"sttModel": cfg.stt_model, "sttModel": cfg.stt_model,
"audioModel": cfg.audio_model, "audioModel": cfg.audio_model,
"defaultVideo": cfg.default_video, "defaultVideo": cfg.default_video,
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment