compose: POST /v1/video/compose and POST /v1/files/upload (v0.2.22)

CoderAI could make every piece of a short video and nothing turned the pieces
into a reel — assembly needs ffmpeg and two facts only the server has (how long
the narration it just synthesised is, and when each word falls), so a detached
client cannot do it. Requested by the Motus project over the HTTP API.

- codai/api/compose.py: scenes -> one uploadable video. Scene length = narration
  + padding (min_duration floor), visuals split it evenly; async job with
  stage/progress/message/warnings, cancel (the ffmpeg child is killed), a queue
  that reports its position, and a blocking form for tests. Narration, music and
  word timings come from this server's own endpoints, called on a loop this
  module owns — never the request's loop, which may be closed before the render
  needs it. 400 names the offending scene/field; 501 when a sub-feature has no
  model (music.generate, timing=stt, TTS for narration) or ffmpeg is missing.
- codai/compose/render.py: one normalised clip per visual (so a failed visual
  becomes a gradient and a warning instead of a lost render), concat or xfade
  that preserves the total duration, narration padded per scene and
  loudness-normalised, music looped/faded/ducked with sidechaincompress, burned
  captions + overlays with the final H.264 High/AAC/+faststart encode, thumbnail.
- codai/compose/captions.py: word timings from STT aligned onto the CALLER's
  words (a mis-heard word never reaches the screen) or estimated; line breaking;
  ASS with six presets inside a 10%/6% safe area; SRT/VTT with the same text.
- codai/compose/media.py: resolve a URL / /v1/files path / data URI / base64
  (never a server path), probe with ffprobe, resolve fonts through fontconfig.
- codai/api/uploads.py: POST /v1/files/upload, content-addressed by sha256 with
  GET /v1/files/blob/{hash} so a client can skip a re-upload; type decided by
  content, not by the client's claim.
- compose and uploads are pinned local in the capability gateway (a pod has
  ffmpeg but not this install's models or files); both advertised in the
  capability document; caption fonts (Montserrat/Inter/Noto CJK) added to the
  three image builds.
- docs/video-compose.md; tests/test_compose.py (30, real ffmpeg, no model/GPU).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7EnDW6RSd2nhEPZMzMkqi

README.md

@@ -205,6 +205,7 @@ Full guide: [`docs/cluster.md`](docs/cluster.md).
 - **Character Profiles**: Apply up to 6 saved character profiles for visual consistency. Pipelines with IP-Adapter (AnimateDiff and other SD-based video models) are conditioned directly. For the rest — Wan, CogVideoX, LTX, SVD, i.e. most video models — CoderAI renders an **identity keyframe** on the image endpoint (which does have real IP-Adapter) and runs the video as `ti2v` from that frame, so `character_profiles` locks a face on every model. Controlled by `keyframe_identity` (`auto` / `always` / `never`) with optional `keyframe_model` and `keyframe_steps`; combine with a per-character LoRA for the strongest consistency
 - **Environment Profiles**: Apply up to 6 saved environment profiles to condition the scene
 - **Multi-Character Dialog**: Assign spoken lines to individual characters — each line picks a character profile, voice profile or TTS voice ID, and text; lines are **voice-cloned with F5-TTS** from the named profile, mixed with correct timing (sequential or manual), and lip-synced via Wav2Lip
+- **Composition (reels)**: `POST /v1/video/compose` turns a list of **scenes** — narration plus visuals — into one finished, uploadable video: the scene's length comes from its narration, visuals are Ken-Burns'd or trimmed to fit, music is looped and ducked under the voice, captions are burned in with word timings (karaoke presets) inside the platforms' safe area, and a thumbnail, SRT/VTT and the narration track come back with it. Async jobs with stage/progress/cancel; a visual that fails to download becomes a gradient and a warning instead of a failed render. See [`docs/video-compose.md`](docs/video-compose.md)
 - **Lip Sync**: Wav2Lip and SadTalker. The Docker/OCI image ships both (launcher shims plus a dedicated lip-sync venv; checkpoints download on first use). A from-source install has no shims, so CoderAI falls back to a self-managed Wav2Lip that fetches and patches itself on first use. Either way, if lip sync can't run the response carries a `warnings` entry saying the audio was muxed without mouth sync — rather than passing off an unsynced video as a success
 
 ### Audio
@@ -802,6 +803,8 @@ persisted per `config_id`, so the configs don't overwrite each other.
 | `POST /v1/embeddings` | Embeddings — text, image, multi-vector (dense/sparse/colbert), geolocation, VPR |
 | `POST /v1/rerank` | Cross-encoder reranking of documents against a query |
 | `GET /v1/files/{filename}` | Fetch a generated file |
+| `POST /v1/files/upload` | Upload your own video/image/audio (content-addressed); returns a `/v1/files/...` URL to reference in later requests |
+| `GET /v1/files/blob/{sha256}` | Does this install already hold these bytes? (skip a re-upload) |
 
 ### Image
 
@@ -826,6 +829,8 @@ persisted per `config_id`, so the configs don't overwrite each other.
 | Endpoint | Description |
 |---|---|
 | `POST /v1/video/generations` | Generate video (t2v/i2v/v2v/ti2v/interp) — supports `character_profiles`, `environment_profiles`, `dialogs` |
+| `POST /v1/video/compose` | Compose scenes into a finished reel (narration timing, visuals, ducked music, burned captions, thumbnail) — async job |
+| `GET /v1/video/compose/{id}` | Composition progress / result; `POST …/{id}/cancel` stops it |
 | `POST /v1/video/upscale` | Upscale video |
 | `POST /v1/video/subtitle` | Generate/burn subtitles |
 | `POST /v1/video/interpolate` | Frame interpolation |

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.2.21"
+__version__ = "0.2.22"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/app.py

@@ -199,6 +199,8 @@ from codai.api.images import router as images_router
 from codai.api.tts import router as tts_router
 from codai.api.text import router as text_router
 from codai.api.video import router as video_router
+from codai.api.compose import router as compose_router
+from codai.api.uploads import router as uploads_router
 from codai.api.audio_gen import router as audio_gen_router
 from codai.api.audio_stems import router as audio_stems_router
 from codai.api.audio_clean import router as audio_clean_router
@@ -691,6 +693,11 @@ app.include_router(images_router, tags=["Images"])
 app.include_router(tts_router, tags=["Audio"])
 app.include_router(text_router, tags=["Text"])
 app.include_router(video_router, tags=["Video"])
+# Composition and generic media uploads. Registered BEFORE the /v1/files/{filename}
+# download route defined further down so /v1/files/upload and /v1/files/blob/{hash}
+# are matched as themselves rather than as a filename.
+app.include_router(compose_router, tags=["Video"])
+app.include_router(uploads_router, tags=["Files"])
 app.include_router(audio_gen_router, tags=["Audio"])
 app.include_router(audio_stems_router, tags=["Audio"])
 app.include_router(audio_clean_router, tags=["Audio"])

codai/api/remote_gateway.py

@@ -115,7 +115,16 @@ _SKIP_PATHS = ("/v1/chat/completions", "/v1/completions", "/v1/models",
                # is only meaningful locally. Forwarded, it would report the pod's
                # view of a config the pod does not have — the exact confusion it
                # was added to end.
-               "/v1/models/test/state")
+               "/v1/models/test/state",
+               # This install's own media store: an upload must land where THIS
+               # front serves /v1/files from, and the existence check answers for
+               # this store. Forwarding either would put the bytes on a pod that
+               # is gone an hour later.
+               "/v1/files/upload", "/v1/files/blob/{hash}")
+
+#: Paths that are this install's own, matched by prefix at run time (the literal
+#: route templates above are what the coverage test reads).
+_LOCAL_PREFIXES = ("/v1/files/upload", "/v1/files/blob")
 
 #: Orchestration endpoints: a sequence of calls to other endpoints, not a model.
 #: They are NEVER forwarded as a whole — the chain stays here and each step goes
@@ -123,7 +132,11 @@ _SKIP_PATHS = ("/v1/chat/completions", "/v1/completions", "/v1/models",
 #: every step is placed by its OWN model's configuration. Forwarding the
 #: orchestration instead would move the logic to a pod and place every step by
 #: that pod's catalogue, which is the opposite of what these are for.
-_ORCHESTRATION_PREFIXES = ("/v1/pipelines", "/v1/characters", "/v1/environments")
+#: ``/v1/video/compose`` is one of these: it calls speech, music and transcription
+#: endpoints itself, and the visuals it was given may be files only this install
+#: holds. Forwarded to a video pod it would find neither the models nor the files.
+_ORCHESTRATION_PREFIXES = ("/v1/pipelines", "/v1/characters", "/v1/environments",
+                           "/v1/video/compose")
 
 #: `model` field in a multipart body, without paying for a full form parse.
 _MULTIPART_MODEL = re.compile(
@@ -505,7 +518,7 @@ def resolve_target(path: str, method: str, query: str, body: bytes,
         return None
     if not path.startswith("/v1/") or path in _SKIP_PATHS:
         return None
-    if path.startswith(_ORCHESTRATION_PREFIXES):
+    if path.startswith(_ORCHESTRATION_PREFIXES) or path.startswith(_LOCAL_PREFIXES):
         return None
     # A generated file is fetched from whichever remote rendered it.
     if path.startswith("/v1/files/"):

codai/api/uploads.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+"""Generic media upload — ``POST /v1/files/upload``.
+
+A client that is not on this machine has media of its own: the user's footage,
+a logo, a music bed. It could inline all of it as base64 in every request, and
+a 40 MB clip sent again for each of ten renders is exactly the kind of waste
+that makes an API unusable. So: upload once, get a ``/v1/files/...`` URL, name
+that URL in as many compose requests as you like.
+
+Storage is content-addressed (sha256 of the bytes), which makes re-uploading
+the same file free and makes ``GET /v1/files/blob/{hash}`` a useful question —
+a client can ask before sending. This mirrors ``/v1/loras/upload`` for LoRA
+weights; the difference is that these files are served back by
+``/v1/files/{name}``, so they live in the ordinary output directory under their
+hash, not in a private blob store.
+"""
+
+import os
+from typing import Optional
+
+from fastapi import APIRouter, Depends, HTTPException, Request
+
+from codai.api.loras import _require_api_auth
+from codai.compose import media
+
+router = APIRouter()
+
+# The name a hash is stored under, e.g. up-3f9c…-<12 hex>.mp4. The prefix keeps
+# uploads recognisable in the archive listing and in /v1/files URLs.
+_PREFIX = "up-"
+_MAX_BYTES = 2 << 30          # 2 GiB: a minute of 4K is ~400 MB
+
+
+def _files_dir() -> str:
+    d = media.files_dir()
+    if not d:
+        raise HTTPException(status_code=501, detail=(
+            "this install has no output directory configured, so uploads cannot be "
+            "stored (start the server with --file-path / files.path)"))
+    os.makedirs(d, exist_ok=True)
+    return d
+
+
+def _stored_name(hexhash: str, ext: str) -> str:
+    return f"{_PREFIX}{hexhash}{ext}"
+
+
+def find_blob(hexhash: str) -> Optional[str]:
+    """The stored path for a hash, whatever extension it landed under."""
+    h = (hexhash or "").strip().lower()
+    if h.startswith("sha256:"):
+        h = h[7:]
+    if not h or not all(c in "0123456789abcdef" for c in h) or len(h) != 64:
+        return None
+    try:
+        d = _files_dir()
+    except HTTPException:
+        return None
+    try:
+        for name in os.listdir(d):
+            if name.startswith(_PREFIX + h):
+                p = os.path.join(d, name)
+                if os.path.isfile(p):
+                    return p
+    except OSError:
+        return None
+    return None
+
+
+def store(data: bytes, filename: str = "", content_type: str = "") -> dict:
+    """Put bytes in the content-addressed store; returns the response document."""
+    if not data:
+        raise HTTPException(status_code=400, detail="empty upload")
+    if len(data) > _MAX_BYTES:
+        raise HTTPException(status_code=413,
+                            detail=f"upload larger than {_MAX_BYTES // (1 << 20)} MiB")
+    mime = media.sniff_mime(data, content_type)
+    kind = media.kind_of(mime)
+    if kind not in ("video", "image", "audio"):
+        raise HTTPException(status_code=415, detail=(
+            f"unsupported media type {mime!r}: uploads must be video "
+            f"(mp4/mov/webm/mkv), image (png/jpg/webp) or audio (mp3/wav/m4a/ogg/flac)"))
+    h = media.sha256_hex(data)
+    existing = find_blob(h)
+    if existing:
+        return {"id": f"sha256:{h}", "name": os.path.basename(existing),
+                "bytes": os.path.getsize(existing), "mime": mime, "kind": kind,
+                "existed": True}
+    name = _stored_name(h, media.ext_for(mime, filename))
+    path = os.path.join(_files_dir(), name)
+    tmp = path + ".part"
+    with open(tmp, "wb") as f:
+        f.write(data)
+    os.replace(tmp, path)
+    return {"id": f"sha256:{h}", "name": name, "bytes": len(data), "mime": mime,
+            "kind": kind, "existed": False}
+
+
+def _with_urls(doc: dict, http_request: Optional[Request]) -> dict:
+    from codai.api.urlutils import build_file_url
+    name = doc.pop("name")
+    doc["path"] = f"/v1/files/{name}"
+    doc["url"] = build_file_url(name, http_request)
+    return doc
+
+
+@router.post("/v1/files/upload", summary="Upload media (content-addressed)", tags=["Files"])
+async def upload_file(request: Request, _auth=Depends(_require_api_auth)):
+    """Store a media file and return the URL to reference it by.
+
+    Accepts the file as ``multipart/form-data`` (field ``file``), as JSON
+    ``{"file": "<base64 or data: URI>"}``, or as a raw request body — the three
+    forms ``/v1/loras/upload`` takes. Returns ``{id, url, path, bytes, mime,
+    kind, existed}``; ``existed: true`` means these exact bytes were already
+    here and nothing was written. Video, image and audio only."""
+    ctype = request.headers.get("content-type", "")
+    data = b""
+    filename = ""
+    part_type = ""
+    if "multipart/form-data" in ctype:
+        form = await request.form()
+        up = form.get("file") or form.get("upload") or form.get("media")
+        if up is None:
+            raise HTTPException(status_code=400,
+                                detail="multipart upload missing 'file' field")
+        if hasattr(up, "read"):
+            data = await up.read()
+            filename = getattr(up, "filename", "") or ""
+            part_type = getattr(up, "content_type", "") or ""
+        else:
+            data = bytes(up)
+    elif "application/json" in ctype:
+        body = await request.json()
+        blob = body.get("file") or body.get("data")
+        if not blob:
+            raise HTTPException(status_code=400,
+                                detail="JSON upload missing 'file'/'data' (base64)")
+        filename = str(body.get("filename") or "")
+        try:
+            data = media.fetch(str(blob))
+        except media.MediaError as exc:
+            raise HTTPException(status_code=400, detail=str(exc))
+    else:
+        data = await request.body()
+        part_type = ctype
+    return _with_urls(store(data, filename, part_type), request)
+
+
+@router.get("/v1/files/blob/{hash}", summary="Check an uploaded file exists", tags=["Files"])
+async def file_blob_info(hash: str, request: Request, _auth=Depends(_require_api_auth)):
+    """Does this install already hold these bytes? 200 with the URL, else 404.
+
+    Lets a client skip re-uploading: hash the file locally, ask, and only send
+    it when the answer is 404. ``hash`` is a hex sha256, with or without the
+    ``sha256:`` prefix."""
+    p = find_blob(hash)
+    if not p:
+        raise HTTPException(status_code=404, detail="blob not found")
+    name = os.path.basename(p)
+    h = name[len(_PREFIX):].split(".")[0]
+    doc = {"id": f"sha256:{h}", "name": name, "bytes": os.path.getsize(p),
+           "mime": media.sniff_mime(open(p, "rb").read(32)), "exists": True}
+    return _with_urls(doc, request)

codai/broker/capabilities.py

@@ -39,6 +39,13 @@ DEFAULT_STUDIO_ENDPOINTS = [
     "v1/video/dub",
     "v1/video/to3d",
     "v1/video/from3d",
+    # Server-side composition (codai/api/compose.py) and the media upload it
+    # references: a client discovers these the same way it discovers the rest.
+    "v1/video/compose",
+    "v1/video/compose/{id}",
+    "v1/video/compose/{id}/cancel",
+    "v1/files/upload",
+    "v1/files/blob/{hash}",
 ]
 
 

codai/compose/__init__.py

+# CoderAI - OpenAI-compatible API server
+# Copyright (C) 2026 Stefy Lanza <stefy@nexlab.net>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+"""Server-side video composition — turning the pieces into a finished reel.
+
+CoderAI can generate every *piece* of a short video (narration, stills, clips,
+music) but a client that is not on this machine cannot assemble them: that
+needs ffmpeg, the narration's exact duration, and word timings. This package
+is that assembly, driven by ``POST /v1/video/compose``
+(:mod:`codai.api.compose`):
+
+* :mod:`codai.compose.media` — resolving a media reference (URL, ``/v1/files``
+  path, data URI, base64) to a local file, probing duration, finding fonts;
+* :mod:`codai.compose.captions` — word timings (from STT or estimated) and the
+  ASS / SRT / VTT the renderer burns in and returns;
+* :mod:`codai.compose.render` — the ffmpeg pipeline: one normalised clip per
+  visual, concat or crossfade, narration + ducked music, burned captions,
+  overlays, thumbnail.
+
+Everything here is synchronous and side-effect-free apart from the files it
+writes into the job's scratch directory, so the whole pipeline is testable
+without a server and without a GPU.
+"""

codai/main.py

@@ -1281,6 +1281,11 @@ def main():
     if global_file_path:
         set_video_file_path(global_file_path)
 
+    # Composition writes its artefacts where /v1/files reads (codai/compose).
+    from codai.api.compose import set_global_file_path as set_compose_file_path
+    if global_file_path:
+        set_compose_file_path(global_file_path)
+
     # Set audio_gen module global args
     from codai.api.audio_gen import set_global_args as set_audiogen_global_args, set_global_file_path as set_audiogen_file_path
     set_audiogen_global_args(global_args)

docs/README.md

@@ -24,6 +24,7 @@ reference; these are the deep dives.
 
 | Doc | Subject |
 |---|---|
+| [video-compose.md](video-compose.md) | `POST /v1/video/compose`: scenes → a finished, uploadable reel (narration timing, Ken Burns, ducked music, burned captions, thumbnail) and `/v1/files/upload` |
 | [ocr.md](ocr.md) | The `/v1/ocr` document-transcription subsystem |
 | [expressive-tts.md](expressive-tts.md) | Expressive text-to-speech |
 | [zimage-lora-training.md](zimage-lora-training.md) | LoRA training for image models |

packaging/linux/Dockerfile.oci

@@ -169,6 +169,11 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
       libvulkan1 \
       lm-sensors \
       espeak \
+      fonts-montserrat \
+      fonts-inter \
+      fonts-noto-core \
+      fonts-noto-cjk \
+      fonts-liberation \
       mesa-vulkan-drivers \
       pciutils \
       openssl \

packaging/linux/Dockerfile.update

@@ -65,4 +65,13 @@ RUN /opt/coderai/python/bin/python3 -m pip install --no-input --no-cache-dir "ze
 RUN apt-get update && apt-get install -y --no-install-recommends openssh-client espeak \
     && rm -rf /var/lib/apt/lists/* || true
 
+# Fonts for burned-in captions and overlays (codai/compose): Montserrat and
+# Inter are what the caption presets ask for by name, Noto (with CJK) is what
+# makes a non-Latin script render as glyphs instead of boxes. The base image
+# carried DejaVu only, so a request for "Montserrat" silently got DejaVu.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        fonts-montserrat fonts-inter fonts-noto-core fonts-noto-cjk fonts-liberation \
+    && fc-cache -f >/dev/null 2>&1 \
+    && rm -rf /var/lib/apt/lists/* || true
+
 # ENTRYPOINT / EXPOSE / VOLUME / ENV / WORKDIR are inherited from the base image.

packaging/runpod/Dockerfile.capability-base-light

@@ -27,6 +27,7 @@ ENV PYTHONUNBUFFERED=1 \
 # at install time; libgl/libglib for opencv.
 RUN apt-get update && apt-get install -y --no-install-recommends \
         ffmpeg git curl ca-certificates libgl1 libglib2.0-0 \
+        fonts-montserrat fonts-inter fonts-noto-core fonts-noto-cjk \
     && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /opt/coderai/app