compose: talking presenter (lip-synced character) and /v1/video/talking-head

Motus request 4. The most common reel format is a person talking to camera, and
only this server can build it: composition synthesises the narration, so it holds
the exact audio of each scene, and the lip-sync engines and identity-locked video
generation are here too.

- codai/compose/presenter.py: per scene — a face (a character profile's front
  reference, or a portrait), something to move in (a still, the caller's clip, or
  a short identity-locked shot generated through the ordinary video endpoint so
  its model config/acceleration/keyframe identity all apply), then an engine
  driven by THAT scene's narration, held to the scene's exact length and the
  canvas size. wav2lip and sadtalker are known by name; anything else installed
  under its own name can be selected by that name, so a newer engine needs no API
  change. engine=auto prefers a video-driven engine when there is a clip and a
  portrait engine otherwise.
- Layouts in render.py: full (replaces the scene's visuals — the scene may then
  have none), pip (a masked circle/rounded/rect bubble with an optional border,
  inside the caption safe area) and split (stacked halves). With a presenter a
  scene's visuals are joined without an internal transition and composited as one
  clip; without one nothing changes.
- Bottom captions move above a bottom bubble: ASS takes a per-event MarginV, so
  only the scenes that need it move (captions.py carries margin_v through).
- Failure never loses a render: no engine, no face, a crash or a failed composite
  falls back to the scene's visuals (a gradient when there are none) and warns,
  naming the scene. New stage "presenter"; result.scenes[i].presenter says which
  engine and layout, or null.
- Lip-sync is the only GPU step in a composition, so it takes a lease from the
  central scheduler instead of fighting a generation for the card.
- POST /v1/video/talking-head (+ /{id}, /{id}/cancel) is the same pipeline as a
  one-scene full-screen composition, for a character-page preview, capped at 15 s;
  GET /v1/video/presenter/engines tells a client what is installed so it can grey
  the feature out instead of meeting a 501. Both stay local in the capability
  gateway (a video pod has the shims but not this install's characters).
- 400 for an unknown character / invalid layout, position, size, shape, motion, or
  a pip scene with no visuals; 501 with what to install when no engine is present.
- docs/video-compose.md, README; tests/test_compose.py 32 -> 43 (engines stubbed,
  compositing verified on real pixels).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7EnDW6RSd2nhEPZMzMkqi

README.md

@@ -205,6 +205,11 @@ Full guide: [`docs/cluster.md`](docs/cluster.md).
 - **Character Profiles**: Apply up to 6 saved character profiles for visual consistency. Pipelines with IP-Adapter (AnimateDiff and other SD-based video models) are conditioned directly. For the rest — Wan, CogVideoX, LTX, SVD, i.e. most video models — CoderAI renders an **identity keyframe** on the image endpoint (which does have real IP-Adapter) and runs the video as `ti2v` from that frame, so `character_profiles` locks a face on every model. Controlled by `keyframe_identity` (`auto` / `always` / `never`) with optional `keyframe_model` and `keyframe_steps`; combine with a per-character LoRA for the strongest consistency
 - **Environment Profiles**: Apply up to 6 saved environment profiles to condition the scene
 - **Multi-Character Dialog**: Assign spoken lines to individual characters — each line picks a character profile, voice profile or TTS voice ID, and text; lines are **voice-cloned with F5-TTS** from the named profile, mixed with correct timing (sequential or manual), and lip-synced via Wav2Lip
+- **Talking presenter**: a scene can be a character talking to camera, lip-synced to that
+  scene's narration — full screen, a bubble over the B-roll (captions move above it), or a
+  split screen; the face comes from a character profile, the motion from a short
+  identity-locked clip, a still or footage you supply. `POST /v1/video/talking-head` renders
+  one line for a preview. Any installed lip-sync engine (wav2lip, sadtalker, …) can be named
 - **Composition (reels)**: `POST /v1/video/compose` turns a list of **scenes** — narration plus visuals — into one finished, uploadable video: the scene's length comes from its narration, visuals are Ken-Burns'd or trimmed to fit, music is looped and ducked under the voice, captions are burned in with word timings (karaoke presets) inside the platforms' safe area, and a thumbnail, SRT/VTT and the narration track come back with it. Async jobs with stage/progress/cancel; a visual that fails to download becomes a gradient and a warning instead of a failed render. See [`docs/video-compose.md`](docs/video-compose.md)
 - **Lip Sync**: Wav2Lip and SadTalker. The Docker/OCI image ships both (launcher shims plus a dedicated lip-sync venv; checkpoints download on first use). A from-source install has no shims, so CoderAI falls back to a self-managed Wav2Lip that fetches and patches itself on first use. Either way, if lip sync can't run the response carries a `warnings` entry saying the audio was muxed without mouth sync — rather than passing off an unsynced video as a success
 
@@ -831,6 +836,8 @@ persisted per `config_id`, so the configs don't overwrite each other.
 | `POST /v1/video/generations` | Generate video (t2v/i2v/v2v/ti2v/interp) — supports `character_profiles`, `environment_profiles`, `dialogs` |
 | `POST /v1/video/compose` | Compose scenes into a finished reel (narration timing, visuals, ducked music, burned captions, thumbnail) — async job |
 | `GET /v1/video/compose/{id}` | Composition progress / result; `POST …/{id}/cancel` stops it |
+| `POST /v1/video/talking-head` | Lip-sync one line onto a character's face (preview); same job store |
+| `GET /v1/video/presenter/engines` | Which lip-sync engines this install has |
 | `POST /v1/video/upscale` | Upscale video |
 | `POST /v1/video/subtitle` | Generate/burn subtitles |
 | `POST /v1/video/interpolate` | Frame interpolation |

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.2.22"
+__version__ = "0.2.23"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/remote_gateway.py

@@ -136,7 +136,8 @@ _LOCAL_PREFIXES = ("/v1/files/upload", "/v1/files/blob")
 #: endpoints itself, and the visuals it was given may be files only this install
 #: holds. Forwarded to a video pod it would find neither the models nor the files.
 _ORCHESTRATION_PREFIXES = ("/v1/pipelines", "/v1/characters", "/v1/environments",
-                           "/v1/video/compose")
+                           "/v1/video/compose", "/v1/video/talking-head",
+                           "/v1/video/presenter")
 
 #: `model` field in a multipart body, without paying for a full form parse.
 _MULTIPART_MODEL = re.compile(

codai/broker/capabilities.py

@@ -46,6 +46,10 @@ DEFAULT_STUDIO_ENDPOINTS = [
     "v1/video/compose/{id}/cancel",
     "v1/files/upload",
     "v1/files/blob/{hash}",
+    # The talking presenter (codai/compose/presenter.py).
+    "v1/video/talking-head",
+    "v1/video/talking-head/{id}",
+    "v1/video/presenter/engines",
 ]
 
 

codai/compose/captions.py

@@ -194,13 +194,19 @@ def break_lines(words: List[dict], max_words_per_line: int = 4,
         if not chunk:
             continue
         rows = [chunk[k:k + per_line] for k in range(0, len(chunk), per_line)]
-        out.append({
+        ev = {
             "start": chunk[0]["start"],
             "end": chunk[-1]["end"],
             "words": chunk,
             "rows": [[w["word"] for w in row] for row in rows],
             "text": " ".join(w["word"] for w in chunk),
-        })
+        }
+        # Words carry the margin their scene needs (a bottom presenter bubble);
+        # an event takes the largest of the ones it groups.
+        mv = max((int(w.get("margin_v") or 0) for w in chunk), default=0)
+        if mv:
+            ev["margin_v"] = mv
+        out.append(ev)
     return out
 
 
@@ -273,11 +279,14 @@ def build_ass(events: List[dict], *, width: int, height: int, font: str = "Monts
     body = []
     for ev in events:
         rows = ev.get("rows") or [[w["word"] for w in ev.get("words", [])]]
+        # A scene with a presenter bubble at the bottom pushes its captions up:
+        # ASS takes a per-event MarginV, so only those events move.
+        mv = int(ev.get("margin_v") or 0)
         if not p["karaoke"]:
             text = "\\N".join(" ".join(_ass_escape(w.upper() if upper else w) for w in row)
                               for row in rows)
             body.append(f"Dialogue: 0,{_ass_time(ev['start'])},{_ass_time(ev['end'])},"
-                        f"cap,,0,0,0,,{text}")
+                        f"cap,,0,0,{mv},,{text}")
             continue
         # Karaoke: one event per word, the whole line drawn each time with the
         # active word in the highlight colour. Simpler than \k timings and it
@@ -299,7 +308,7 @@ def build_ass(events: List[dict], *, width: int, height: int, font: str = "Monts
             if end <= start:
                 end = start + 0.04
             body.append(f"Dialogue: 0,{_ass_time(start)},{_ass_time(end)},"
-                        f"cap,,0,0,0,,{{\\c{primary}}}" + "\\N".join(parts))
+                        f"cap,,0,0,{mv},,{{\\c{primary}}}" + "\\N".join(parts))
     return "\n".join(head + body) + "\n"
 
 

codai/compose/render.py

@@ -248,6 +248,148 @@ def build_visual(visual: dict, out_path: str, *, width: int, height: int, fps: i
     run(pre + ["-i", src, "-vf", vf] + common_out, cancel=cancel, log=log)
 
 
+def still_video(image: str, out_path: str, *, duration: float, width: int, height: int,
+                fps: int, cancel=None, log=None) -> str:
+    """A motionless clip of a portrait — what a video-driven lip-sync engine needs
+    when there is no footage to drive (and what a generated shot falls back to).
+    No Ken Burns: a presenter's head must not drift while it talks."""
+    run(["-loop", "1", "-i", image, "-t", f"{max(0.08, duration):.3f}",
+         "-vf", fit_filter(width, height, "cover") + f",fps={fps},format=yuv420p",
+         "-r", str(fps), "-an"] + _INTER_ARGS + [out_path], cancel=cancel, log=log)
+    return out_path
+
+
+def hold_to(src: str, out_path: str, *, duration: float, fps: int,
+            width: Optional[int] = None, height: Optional[int] = None,
+            fit: str = "cover", cancel=None, log=None) -> str:
+    """Make a clip exactly ``duration`` long, and the canvas's size when asked.
+
+    A lip-sync engine returns the length of the audio it was given and whatever
+    frame size it likes (SadTalker squares its output), while the scene is the
+    narration plus its padding at the canvas size — so: hold the final frame (what
+    a presenter does when they stop talking), trim if longer, and scale/crop to
+    the canvas."""
+    chain = [f"fps={fps}"]
+    if width and height:
+        chain.append(fit_filter(int(width), int(height), fit))
+    chain += [f"tpad=stop_mode=clone:stop_duration={max(0.05, duration):.3f}",
+              f"trim=0:{max(0.08, duration):.3f}", "setpts=PTS-STARTPTS",
+              "format=yuv420p"]
+    run(["-i", src, "-vf", ",".join(chain), "-an", "-r", str(fps)]
+        + _INTER_ARGS + [out_path], cancel=cancel, log=log)
+    return out_path
+
+
+# ------------------------------------------------------------ presenter layout
+#: The presenter bubble keeps the captions' safe area: clear of the platform UI.
+SAFE_X = 0.06
+SAFE_Y = 0.10
+
+
+def bubble_mask(path: str, w: int, h: int, shape: str = "circle",
+                border_width: int = 0, border_colour: str = "#FFFFFF") -> tuple:
+    """(mask, ring) PNGs for a picture-in-picture bubble.
+
+    The mask is greyscale — white where the presenter shows — and becomes the
+    clip's alpha; the ring is an RGBA outline drawn on top afterwards, so the
+    border is not eaten by the same mask that rounds the corners."""
+    from PIL import Image, ImageDraw
+    w, h = max(2, int(w)), max(2, int(h))
+    shape = (shape or "circle").lower()
+    radius = int(min(w, h) * 0.18)
+    mask = Image.new("L", (w, h), 0)
+    md = ImageDraw.Draw(mask)
+    if shape == "circle":
+        md.ellipse((0, 0, w - 1, h - 1), fill=255)
+    elif shape in ("rounded", "round"):
+        md.rounded_rectangle((0, 0, w - 1, h - 1), radius=radius, fill=255)
+    else:
+        md.rectangle((0, 0, w - 1, h - 1), fill=255)
+    mask_path = path + ".mask.png"
+    mask.save(mask_path)
+
+    ring_path = None
+    if border_width and border_width > 0:
+        col = (border_colour or "#FFFFFF").lstrip("#")
+        if len(col) == 3:
+            col = "".join(c * 2 for c in col)
+        rgb = tuple(int(col[i:i + 2], 16) for i in (0, 2, 4)) if len(col) == 6 else (255, 255, 255)
+        ring = Image.new("RGBA", (w, h), (0, 0, 0, 0))
+        rd = ImageDraw.Draw(ring)
+        bw = max(1, int(border_width))
+        box = (bw // 2, bw // 2, w - 1 - bw // 2, h - 1 - bw // 2)
+        if shape == "circle":
+            rd.ellipse(box, outline=rgb + (255,), width=bw)
+        elif shape in ("rounded", "round"):
+            rd.rounded_rectangle(box, radius=radius, outline=rgb + (255,), width=bw)
+        else:
+            rd.rectangle(box, outline=rgb + (255,), width=bw)
+        ring_path = path + ".ring.png"
+        ring.save(ring_path)
+    return mask_path, ring_path
+
+
+def bubble_geometry(width: int, height: int, size: float, position: str,
+                    shape: str = "circle") -> tuple:
+    """(w, h, x, y) of the bubble inside the safe area."""
+    size = max(0.1, min(0.95, float(size or 0.38)))
+    bw = int(round(width * size / 2) * 2)
+    bh = bw if (shape or "circle").lower() in ("circle", "rounded", "round")         else int(round(bw * height / max(1, width) / 2) * 2)
+    bh = min(bh, int(height * 0.8))
+    mx, my = int(width * SAFE_X), int(height * SAFE_Y)
+    pos = (position or "bottom-right").lower().replace("_", "-")
+    vert, _, horiz = pos.partition("-") if "-" in pos else (pos, "", "center")
+    x = {"left": mx, "center": (width - bw) // 2, "centre": (width - bw) // 2,
+         "right": width - bw - mx}.get(horiz, width - bw - mx)
+    y = {"top": my, "center": (height - bh) // 2, "centre": (height - bh) // 2,
+         "middle": (height - bh) // 2,
+         "bottom": height - bh - my}.get(vert, height - bh - my)
+    return bw, bh, max(0, x), max(0, y)
+
+
+def composite_pip(base: str, presenter: str, out_path: str, *, width: int, height: int,
+                  fps: int, size: float = 0.38, position: str = "bottom-right",
+                  shape: str = "circle", border: Optional[dict] = None,
+                  workdir: str = "", cancel=None, log=None) -> str:
+    """The presenter as a bubble over the scene's visuals."""
+    bw, bh, x, y = bubble_geometry(width, height, size, position, shape)
+    border = border or {}
+    mask, ring = bubble_mask(os.path.join(workdir or ".", f"bubble-{bw}x{bh}"), bw, bh,
+                             shape, int(border.get("width") or 0),
+                             border.get("color") or border.get("colour") or "#FFFFFF")
+    inputs = ["-i", base, "-i", presenter, "-i", mask]
+    graph = [f"[1:v]scale={bw}:{bh}:force_original_aspect_ratio=increase,"
+             f"crop={bw}:{bh},format=rgba[pv]",
+             "[2:v]format=gray[pm]",
+             "[pv][pm]alphamerge[pa]"]
+    top = "pa"
+    if ring:
+        inputs += ["-i", ring]
+        graph.append("[pa][3:v]overlay=0:0[pb]")
+        top = "pb"
+    graph.append(f"[0:v][{top}]overlay={x}:{y}:format=auto,format=yuv420p,fps={fps}[vout]")
+    run(inputs + ["-filter_complex", ";".join(graph), "-map", "[vout]", "-an"]
+        + _INTER_ARGS + [out_path], cancel=cancel, log=log)
+    return out_path
+
+
+def composite_split(base: str, presenter: str, out_path: str, *, width: int, height: int,
+                    fps: int, presenter_half: str = "bottom", cancel=None,
+                    log=None) -> str:
+    """Presenter and B-roll stacked, each filling its half."""
+    half = int(height / 2 / 2) * 2
+    fill = (f"scale={width}:{half}:force_original_aspect_ratio=increase,"
+            f"crop={width}:{half},setsar=1")
+    top_in, bottom_in = ((1, 0) if (presenter_half or "bottom").lower() == "top"
+                         else (0, 1))
+    graph = [f"[{top_in}:v]{fill}[t]", f"[{bottom_in}:v]{fill}[b]",
+             f"[t][b]vstack=inputs=2,pad={width}:{height}:0:0,"
+             f"format=yuv420p,fps={fps}[vout]"]
+    run(["-i", base, "-i", presenter, "-filter_complex", ";".join(graph),
+         "-map", "[vout]", "-an"] + _INTER_ARGS + [out_path], cancel=cancel, log=log)
+    return out_path
+
+
 # --------------------------------------------------------------------- join
 def concat_clips(clips: List[str], out_path: str, workdir: str,
                  cancel: Optional[threading.Event] = None,

docs/video-compose.md

@@ -182,6 +182,78 @@ The body may also be JSON (`{"file": "<base64 or data: URI>"}`) or the raw bytes
 with a `Content-Type`. Video, image and audio only; the type is decided by the
 content, not by the client's claim.
 
+## The talking presenter
+
+A person talking to camera is the format most short videos use, and it is
+something only this server can assemble: composition synthesises the narration,
+so it holds the exact audio of every scene, and the lip-sync engines and
+identity-locked video generation are here too.
+
+```jsonc
+{
+  "presenter": {
+    "character": "nova",            // a saved character profile (its front reference is used)
+    "image": null,                  // or an explicit portrait (URL / /v1/files / data URI / base64)
+    "engine": "auto",               // auto | wav2lip | sadtalker | <any installed engine name>
+    "motion": "generate",           // generate: a short identity-locked clip per scene
+                                    // still: animate the portrait   src: use base_src
+    "base_src": null,
+    "prompt": "talking to the camera, natural head movement, soft studio light",
+    "layout": "full",               // full | pip | split
+    "position": "bottom-right",     // pip: top/bottom + left/right/center
+    "size": 0.38,                   // pip: bubble width as a fraction of the canvas
+    "shape": "circle",              // pip: circle | rounded | rect
+    "split": "bottom",              // split: which half the presenter takes
+    "border": {"color": "#FFFFFF", "width": 6},
+    "remove_background": false      // ignored (with a warning) when no matting is installed
+  },
+  "scenes": [
+    {"text": "Hi! Today: three hearts.", "visuals": [...]},                      // the default presenter
+    {"text": "Here's how they work.",    "visuals": [...], "presenter": false},  // B-roll only
+    {"text": "Follow for more!",         "visuals": [], "presenter": {"layout": "full"}}
+  ]
+}
+```
+
+* **Timing** is the scene's: the engine is driven by *that scene's* narration —
+  synthesised or supplied — and the clip is held to narration + padding (or
+  `min_duration`). Captions, music and ducking are unchanged.
+* **`engine: "auto"`** takes a video-driven engine (wav2lip) when there is a clip
+  to drive and a portrait engine (sadtalker) otherwise. Any other engine installed
+  under its own name can be named instead — no API change for a new one. Ask
+  `GET /v1/video/presenter/engines` what this install has.
+* **`layout`**: `full` replaces the scene's visuals (a scene may then have none);
+  `pip` composites a bubble over them inside the caption safe area — and captions
+  at the bottom move above it automatically; `split` stacks presenter and B-roll,
+  each filling its half. With a presenter, a scene's visuals are joined without a
+  transition *between them* and composited as one clip; without one, nothing
+  changes.
+* **Nothing fails the render**: no engine, no face, a crash, a failed composite —
+  each falls back to the scene's visuals (or a gradient when there are none) and
+  adds a warning naming the scene. `stage` is `presenter`, with
+  `message: "scene 2/6: lip-sync"`, and `result.scenes[i].presenter` is
+  `{"engine": …, "layout": …}` or `null`.
+* **Errors at submission**: `400` for an unknown character, an invalid layout,
+  position, size, shape, motion or a `pip` scene with no visuals; `501` when a
+  presenter is asked for and no lip-sync engine is installed, naming what to
+  install.
+* **GPU**: lip-sync is the only GPU step in a composition (the rest is ffmpeg), so
+  it takes a lease from the same scheduler ordinary model requests use instead of
+  fighting a generation for the card.
+
+### `POST /v1/video/talking-head`
+
+The presenter on its own, for a preview on a character page: a character (or a
+portrait), `text` synthesised with `voice` — or `audio` you already have — and the
+same job semantics as compose (same job store, so either status URL works). Capped
+at 15 s of audio. `result.video` as usual.
+
+```bash
+curl -H "Authorization: Bearer $TOKEN" -X POST …/v1/video/talking-head \
+  -d '{"character":"nova","text":"Hi, I am Nova.","canvas":{"width":720,"height":720}}'
+# {"id":"cmp_…","status":"queued"}
+```
+
 ## The same references elsewhere
 
 The extractors that build characters and voice profiles from a user's own media
@@ -231,6 +303,7 @@ claimed.
 |---|---|
 | Endpoints, validation, job queue, orchestration | `codai/api/compose.py` |
 | Uploads (`/v1/files/upload`, `/v1/files/blob/{hash}`) | `codai/api/uploads.py` |
+| The talking presenter (engines, face, base clip) | `codai/compose/presenter.py` |
 | Reference resolving, probing, fonts | `codai/compose/media.py` |
 | Word timings, line breaking, ASS/SRT/VTT | `codai/compose/captions.py` |
 | The ffmpeg pipeline | `codai/compose/render.py` |