video: make character_profiles actually lock identity on every video model

Loading the IP-Adapter (958d2e09) fixed the mechanism, but only AnimateDiff-class
pipelines have one. On Wan / CogVideoX / LTX / SVD — the models people actually
use — character_profiles still did nothing except prepend the profile name to the
prompt, so identity was not working where it mattered.

The image endpoint does have real IP-Adapter. So when a video model can't
condition on references itself, render the first frame there against the same
profiles and run the video as ti2v from that keyframe. This is the recipe
tools/videogen.py already drives by hand, now done server-side.

It runs BEFORE the video pipeline is loaded, so an image model and a video model
are never resident at once, and it dispatches through the front (execute_api_request)
so the image model may live on a different engine.

New request fields: keyframe_identity (auto | always | never), keyframe_model,
keyframe_steps. 'auto' skips the bridge when the model has native IP-Adapter or a
first frame was already supplied.

If the keyframe can't be rendered the request still proceeds — identity degrades
to the prompt hint plus any per-character LoRA and the reason is reported in the
response warnings — unless the caller asked for 'always', which is an explicit
requirement and so returns 502 rather than silently producing a stranger.

Verified offline across every branch: bridge fires for Wan and rewrites to ti2v
with the keyframe; skipped for AnimateDiff; forced by 'always'; disabled by
'never'; no-op with no profiles or an existing first frame; degrades with a
warning on image-endpoint failure and raises 502 under 'always'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4

README.md

@@ -111,7 +111,7 @@ Full guide: [`docs/runpod.md`](docs/runpod.md).
 - **Upscaling**: Real-ESRGAN video upscaling
 - **Subtitles**: Whisper transcription + optional translation + burn-in
 - **Dubbing**: Transcribe → translate → TTS → replace audio track
-- **Character Profiles**: Apply up to 6 saved character profiles for visual consistency. Identity is conditioned via IP-Adapter on pipelines that support it (AnimateDiff and other SD-based video models); pipelines without IP-Adapter (Wan, CogVideoX, LTX, SVD) fall back to the prompt hint plus any per-character LoRA, and say so in the log
+- **Character Profiles**: Apply up to 6 saved character profiles for visual consistency. Pipelines with IP-Adapter (AnimateDiff and other SD-based video models) are conditioned directly. For the rest — Wan, CogVideoX, LTX, SVD, i.e. most video models — CoderAI renders an **identity keyframe** on the image endpoint (which does have real IP-Adapter) and runs the video as `ti2v` from that frame, so `character_profiles` locks a face on every model. Controlled by `keyframe_identity` (`auto` / `always` / `never`) with optional `keyframe_model` and `keyframe_steps`; combine with a per-character LoRA for the strongest consistency
 - **Environment Profiles**: Apply up to 6 saved environment profiles to condition the scene
 - **Multi-Character Dialog**: Assign spoken lines to individual characters — each line picks a character profile, voice profile or TTS voice ID, and text; lines are **voice-cloned with F5-TTS** from the named profile, mixed with correct timing (sequential or manual), and lip-synced via Wav2Lip
 - **Lip Sync**: Wav2Lip and SadTalker. The Docker/OCI image ships both (launcher shims plus a dedicated lip-sync venv; checkpoints download on first use). A from-source install has no shims, so CoderAI falls back to a self-managed Wav2Lip that fetches and patches itself on first use. Either way, if lip sync can't run the response carries a `warnings` entry saying the audio was muxed without mouth sync — rather than passing off an unsynced video as a success

codai/api/video.py

@@ -3575,6 +3575,96 @@ async def get_video_progress():
 # Main generation endpoint
 # =============================================================================
 
+#: Video pipeline families that can do IP-Adapter themselves, so they don't need the
+#: keyframe bridge. Everything else (Wan, CogVideoX, LTX, SVD, …) does.
+_NATIVE_IP_ADAPTER_VIDEO = ('animatediff', 'animate-diff')
+
+
+async def _maybe_build_identity_keyframe(request: VideoGenerationRequest,
+                                          http_request) -> Optional[str]:
+    """Lock character identity on video models that have no IP-Adapter.
+
+    ``character_profiles`` can only condition identity where the pipeline supports
+    IP-Adapter — which the video pipelines people actually use (Wan, CogVideoX, LTX,
+    SVD) do not. On those, the profile previously did nothing but prepend a name to
+    the prompt.
+
+    The image endpoint *does* have real IP-Adapter, so we render the first frame
+    there, conditioned on the profiles, and hand it to the video model as a ti2v
+    keyframe — the same recipe tools/videogen.py drives by hand, done server-side.
+    It runs BEFORE the video pipeline is loaded, so the image model isn't fighting
+    the video model for VRAM, and it goes through the front so the image model may
+    live on another engine entirely.
+
+    Returns a short note when the bridge ran (for the response warnings), else None.
+    """
+    mode = (request.mode or 't2v').lower()
+    setting = str(getattr(request, 'keyframe_identity', 'auto') or 'auto').lower()
+    if setting in ('never', 'off', 'false', 'no', '0'):
+        return None
+    # Only useful when there is no first frame yet.
+    if mode != 't2v' or request.init_image or request.image:
+        return None
+    if not (request.character_profiles or request.characters or request.character_references):
+        return None
+    if setting == 'auto' and any(f in (request.model or '').lower()
+                                 for f in _NATIVE_IP_ADAPTER_VIDEO):
+        return None          # the pipeline can condition on the refs directly
+
+    payload = {
+        "prompt": request.prompt or "",
+        "n": 1,
+        "response_format": "b64_json",
+        "size": f"{request.width or 512}x{request.height or 512}",
+        "character_strength": request.character_strength or 0.8,
+    }
+    if request.character_profiles:
+        payload["character_profiles"] = list(request.character_profiles)
+    if request.character_references:
+        payload["character_references"] = list(request.character_references)
+    if getattr(request, 'environment_profiles', None):
+        payload["environment_profiles"] = list(request.environment_profiles)
+    if getattr(request, 'keyframe_model', None):
+        payload["model"] = request.keyframe_model
+    if getattr(request, 'keyframe_steps', None):
+        payload["steps"] = request.keyframe_steps
+    if request.negative_prompt:
+        payload["negative_prompt"] = request.negative_prompt
+    if request.seed is not None:
+        payload["seed"] = request.seed
+
+    try:
+        import json as _json
+        from codai.broker.asgi_bridge import execute_api_request
+        why = "forced by keyframe_identity=always" if setting == 'always' \
+              else "this video model has no IP-Adapter"
+        print(f"  [video][identity] rendering IP-Adapter keyframe for "
+              f"{request.character_profiles or request.character_names or 'references'}"
+              f" ({why})", flush=True)
+        resp = await execute_api_request(
+            http_request, method="POST", path="/v1/images/generations",
+            headers={"Content-Type": "application/json"},
+            body=_json.dumps(payload).encode())
+        if resp["status_code"] >= 400:
+            raise RuntimeError(resp["body"].decode()[:300])
+        data = _json.loads(resp["body"]).get("data", [])
+        b64 = (data[0].get("b64_json") if data else None)
+        if not b64:
+            raise RuntimeError("image endpoint returned no b64_json")
+    except Exception as exc:
+        msg = (f"character identity keyframe could not be rendered ({exc}); "
+               f"identity falls back to the prompt hint and any per-character LoRA")
+        print(f"  [video][identity] {msg}", flush=True)
+        if setting == 'always':
+            raise HTTPException(status_code=502, detail=msg)
+        return msg
+
+    request.init_image = f"data:image/png;base64,{b64}"
+    request.mode = 'ti2v'
+    print("  [video][identity] keyframe ready — running ti2v from it", flush=True)
+    return None
+
+
 @router.post("/v1/video/generations", response_model=VideoGenerationResponse, summary="Generate video")
 async def video_generations(request: VideoGenerationRequest,
                              http_request: Request = None):
@@ -3601,6 +3691,11 @@ async def video_generations(request: VideoGenerationRequest,
         elif request.video:
             request.mode = 'v2v'
 
+    # Character identity bridge. Must run BEFORE the video model is loaded: it
+    # generates an image, and doing that afterwards would put an image model and a
+    # video model on the card at once. May rewrite mode → ti2v + init_image.
+    _identity_note = await _maybe_build_identity_keyframe(request, http_request)
+
     # Run in a thread: request_model may block while waiting for a busy text
     # model to finish its in-flight request before evicting it.  Blocking here
     # on the event loop would deadlock that very request.
@@ -3967,6 +4062,8 @@ async def video_generations(request: VideoGenerationRequest,
     # Post-processing pipeline (upscale, audio, subtitles, …)
     temps = []
     gen_warnings: list = []
+    if _identity_note:
+        gen_warnings.append(_identity_note)
     try:
         needs_post = any([
             request.upscale_output,

codai/pydantic/videorequest.py

@@ -97,6 +97,16 @@ class VideoGenerationRequest(BaseModel):
     character_strength: Optional[float] = Field(0.8, description="IP-Adapter scale for character references.")
     character_names: Optional[List[str]] = Field(None, description="Optional names aligned with character_references.")
     character_profiles: Optional[List[str]] = Field(None, description="Saved character profile names to load (resolved server-side).")
+    keyframe_identity: Optional[str] = Field("auto", description=(
+        "How to lock character identity on video models that have no IP-Adapter "
+        "(Wan, CogVideoX, LTX, SVD — i.e. most of them). 'auto' (default) renders an "
+        "identity-conditioned first frame with the IMAGE endpoint, which does have real "
+        "IP-Adapter, then runs the video as ti2v from that keyframe; it is skipped when "
+        "the model can do IP-Adapter itself or a first frame was already supplied. "
+        "'always' forces the bridge, 'never' disables it (identity then relies on the "
+        "prompt hint plus any per-character LoRA)."))
+    keyframe_model: Optional[str] = Field(None, description="Image model for the identity keyframe (defaults to the server's image model).")
+    keyframe_steps: Optional[int] = Field(None, description="Denoising steps for the identity keyframe.")
 
     loras: Optional[List[VideoLoraConfig]] = Field(None, description="Per-request LoRA adapters (e.g. trained per-character identity LoRAs) fused into the pipeline.")