• Stefy Lanza (nextime / spora )'s avatar
    compose: talking presenter (lip-synced character) and /v1/video/talking-head · b8b5ba3a
    Stefy Lanza (nextime / spora ) authored
    Motus request 4. The most common reel format is a person talking to camera, and
    only this server can build it: composition synthesises the narration, so it holds
    the exact audio of each scene, and the lip-sync engines and identity-locked video
    generation are here too.
    
    - codai/compose/presenter.py: per scene — a face (a character profile's front
      reference, or a portrait), something to move in (a still, the caller's clip, or
      a short identity-locked shot generated through the ordinary video endpoint so
      its model config/acceleration/keyframe identity all apply), then an engine
      driven by THAT scene's narration, held to the scene's exact length and the
      canvas size. wav2lip and sadtalker are known by name; anything else installed
      under its own name can be selected by that name, so a newer engine needs no API
      change. engine=auto prefers a video-driven engine when there is a clip and a
      portrait engine otherwise.
    - Layouts in render.py: full (replaces the scene's visuals — the scene may then
      have none), pip (a masked circle/rounded/rect bubble with an optional border,
      inside the caption safe area) and split (stacked halves). With a presenter a
      scene's visuals are joined without an internal transition and composited as one
      clip; without one nothing changes.
    - Bottom captions move above a bottom bubble: ASS takes a per-event MarginV, so
      only the scenes that need it move (captions.py carries margin_v through).
    - Failure never loses a render: no engine, no face, a crash or a failed composite
      falls back to the scene's visuals (a gradient when there are none) and warns,
      naming the scene. New stage "presenter"; result.scenes[i].presenter says which
      engine and layout, or null.
    - Lip-sync is the only GPU step in a composition, so it takes a lease from the
      central scheduler instead of fighting a generation for the card.
    - POST /v1/video/talking-head (+ /{id}, /{id}/cancel) is the same pipeline as a
      one-scene full-screen composition, for a character-page preview, capped at 15 s;
      GET /v1/video/presenter/engines tells a client what is installed so it can grey
      the feature out instead of meeting a 501. Both stay local in the capability
      gateway (a video pod has the shims but not this install's characters).
    - 400 for an unknown character / invalid layout, position, size, shape, motion, or
      a pip scene with no visuals; 501 with what to install when no engine is present.
    - docs/video-compose.md, README; tests/test_compose.py 32 -> 43 (engines stubbed,
      compositing verified on real pixels).
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01B7EnDW6RSd2nhEPZMzMkqi
    b8b5ba3a
video-compose.md 15.3 KB