• Stefy Lanza (nextime / spora )'s avatar
    compose: talking presenter (lip-synced character) and /v1/video/talking-head · b8b5ba3a
    Stefy Lanza (nextime / spora ) authored
    Motus request 4. The most common reel format is a person talking to camera, and
    only this server can build it: composition synthesises the narration, so it holds
    the exact audio of each scene, and the lip-sync engines and identity-locked video
    generation are here too.
    
    - codai/compose/presenter.py: per scene — a face (a character profile's front
      reference, or a portrait), something to move in (a still, the caller's clip, or
      a short identity-locked shot generated through the ordinary video endpoint so
      its model config/acceleration/keyframe identity all apply), then an engine
      driven by THAT scene's narration, held to the scene's exact length and the
      canvas size. wav2lip and sadtalker are known by name; anything else installed
      under its own name can be selected by that name, so a newer engine needs no API
      change. engine=auto prefers a video-driven engine when there is a clip and a
      portrait engine otherwise.
    - Layouts in render.py: full (replaces the scene's visuals — the scene may then
      have none), pip (a masked circle/rounded/rect bubble with an optional border,
      inside the caption safe area) and split (stacked halves). With a presenter a
      scene's visuals are joined without an internal transition and composited as one
      clip; without one nothing changes.
    - Bottom captions move above a bottom bubble: ASS takes a per-event MarginV, so
      only the scenes that need it move (captions.py carries margin_v through).
    - Failure never loses a render: no engine, no face, a crash or a failed composite
      falls back to the scene's visuals (a gradient when there are none) and warns,
      naming the scene. New stage "presenter"; result.scenes[i].presenter says which
      engine and layout, or null.
    - Lip-sync is the only GPU step in a composition, so it takes a lease from the
      central scheduler instead of fighting a generation for the card.
    - POST /v1/video/talking-head (+ /{id}, /{id}/cancel) is the same pipeline as a
      one-scene full-screen composition, for a character-page preview, capped at 15 s;
      GET /v1/video/presenter/engines tells a client what is installed so it can grey
      the feature out instead of meeting a 501. Both stay local in the capability
      gateway (a video pod has the shims but not this install's characters).
    - 400 for an unknown character / invalid layout, position, size, shape, motion, or
      a pip scene with no visuals; 501 with what to install when no engine is present.
    - docs/video-compose.md, README; tests/test_compose.py 32 -> 43 (engines stubbed,
      compositing verified on real pixels).
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01B7EnDW6RSd2nhEPZMzMkqi
    b8b5ba3a
Name
Last commit
Last update
..
__init__.py Loading commit data...
asgi_bridge.py Loading commit data...
capabilities.py Loading commit data...
client.py Loading commit data...
config.py Loading commit data...
dispatcher.py Loading commit data...
models.py Loading commit data...
service.py Loading commit data...
streaming.py Loading commit data...