video: serve MiniMax-H3 through an isolated worker, and clone voices in the studio

MiniMax-H3 is a 33B flow-matching DiT that denoises video and audio in ONE packed
sequence. diffusers ships it as Modular blocks only (MiniMaxH3ModularPipeline,
0.40+) — there is no DiffusionPipeline half, so it fits neither
_detect_pipeline_class() nor this venv's diffusers 0.38. It runs in an isolated
venv, the same way pyannote, NeMo-Canary and vLLM do:

  tools/h3_service.py   generation inside the isolated venv (HTTP)
  codai/api/h3_worker.py venv bootstrap, process lifecycle, eviction handle
  requirements-h3.txt   the venv's pins (diffusers>=0.40, transformers 5.x)

_generate_h3() adapts the request to the checkpoint's contracts instead of
forwarding it, and reports every adaptation in the response warnings: guidance is
baked in (negative_prompt and guidance_scale dropped), 24 fps is fixed and
num_frames snaps up to the next 17n+5 within 5-15s, canvas axes round to 32, and
v2v/interp are rejected. character_profiles become native ref2va references (max
9) so H3 skips the identity-keyframe bridge; an explicit keyframe wins, since
fl2va and ref2va are separate transformer partitions. The worker picks a workflow
per request and never loads without one — that pulls both 61.7GB partitions.

The worker registers as an evictable, VRAM-tracked model; eviction stops the
subprocess, the only way the transformer's pages actually come back.

Character studio: clone the voice off the same footage the face came from
(/v1/audio/voices/extract) with `extract --voice`, the new `voice` subcommand or
the UI checkbox, then `--say` a line that is lip-synced in that cloned voice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY

codai/api/video.py

@@ -3576,9 +3576,157 @@ async def get_video_progress():
 # Main generation endpoint
 # =============================================================================
 
-#: Video pipeline families that can do IP-Adapter themselves, so they don't need the
-#: keyframe bridge. Everything else (Wan, CogVideoX, LTX, SVD, …) does.
-_NATIVE_IP_ADAPTER_VIDEO = ('animatediff', 'animate-diff')
+#: Video pipeline families that condition on character references themselves, so they
+#: don't need the keyframe bridge: AnimateDiff has real IP-Adapter, and MiniMax-H3's
+#: ref2va workflow takes the reference images directly (up to 9). Everything else
+#: (Wan, CogVideoX, LTX, SVD, …) needs the bridge.
+_NATIVE_IP_ADAPTER_VIDEO = ('animatediff', 'animate-diff', 'minimax-h3', 'minimax_h3')
+
+
+async def _generate_h3(request: VideoGenerationRequest, model_name: str,
+                       model_cfg: dict, http_request,
+                       identity_note: Optional[str] = None):
+    """Generate on MiniMax-H3 through the isolated worker.
+
+    H3 differs from every other video model here in ways the request has to be
+    translated for, not just forwarded:
+
+      * It is guidance-distilled — no negative prompt, no guidance scale. Both are
+        dropped (with a warning, so a caller that set them isn't left guessing).
+      * Video and audio are denoised jointly, so the returned mp4 already HAS a
+        soundtrack. The post-processing chain still runs on top for lip sync,
+        dialogs, subtitles and upscaling.
+      * Character identity is native: the profile's reference images go in as
+        ref2va references (up to 9) instead of through the keyframe bridge.
+      * Frame count and canvas are checkpoint contracts (24 fps, 17n+5 frames,
+        5-15 s, axes multiple of 32); the worker snaps them and reports what it
+        actually generated.
+    """
+    from codai.api import h3_worker
+
+    warnings: list = []
+    if identity_note:
+        warnings.append(identity_note)
+
+    mode = (request.mode or 't2v').lower()
+    if mode in ('v2v', 'interp'):
+        raise HTTPException(status_code=400, detail=(
+            f"MiniMax-H3 has no '{mode}' workflow — it supports t2va (prompt), "
+            "fl2va (first and/or last keyframe) and ref2va (reference media)."))
+    if request.negative_prompt:
+        warnings.append("MiniMax-H3 is guidance-distilled: negative_prompt ignored")
+    if request.guidance_scale is not None:
+        warnings.append("MiniMax-H3 is guidance-distilled: guidance_scale ignored")
+
+    def _as_b64(ref: str) -> str:
+        """Normalise a base64 / data-URI / URL reference to plain base64."""
+        return base64.b64encode(_decode_b64_or_url(ref)).decode()
+
+    payload = {
+        "prompt": request.prompt or "",
+        "num_inference_steps": request.num_inference_steps or 50,
+    }
+    if request.num_frames:
+        payload["num_frames"] = int(request.num_frames)
+    if request.fps and int(request.fps) != 24:
+        warnings.append("MiniMax-H3 generates at a fixed 24 fps; fps ignored")
+    if request.width and request.height:
+        payload["width"], payload["height"] = int(request.width), int(request.height)
+    if request.seed is not None:
+        payload["seed"] = int(request.seed)
+
+    first = request.init_image or request.image
+    if first:
+        payload["image"] = _as_b64(first)
+    if request.end_image:
+        payload["last_image"] = _as_b64(request.end_image)
+
+    # ── native identity conditioning (ref2va) ───────────────────────────────
+    refs: list = []
+    try:
+        if request.character_profiles:
+            from codai.api.characters import resolve_character_profiles
+            for img in resolve_character_profiles(list(request.character_profiles)):
+                refs.append({"type": "image", "data": _as_b64(img)})
+    except Exception as e:
+        print(f"  [h3] character profile resolution failed: {e}")
+    for img in (request.character_references or []):
+        refs.append({"type": "image", "data": _as_b64(img)})
+    for entry in (request.characters or []):
+        for img in (entry.get("images") or []):
+            refs.append({"type": "image", "data": _as_b64(img)})
+    if refs:
+        # The checkpoint takes at most 9 image references; keep the first ones,
+        # which is the order the caller considered most representative.
+        if len(refs) > 9:
+            warnings.append(f"MiniMax-H3 takes 9 image references; used the first 9 of {len(refs)}")
+            refs = refs[:9]
+        if payload.get("image") or payload.get("last_image"):
+            # fl2va and ref2va are different transformer partitions — a request
+            # can't be both. The explicit keyframe wins; say so.
+            warnings.append("MiniMax-H3: keyframe supplied, so character references were dropped "
+                            "(fl2va and ref2va are separate checkpoints)")
+        else:
+            payload["references"] = refs
+
+    if getattr(request, 'loras', None):
+        adapters = []
+        for lora in request.loras:
+            path = getattr(lora, 'model', None) or getattr(lora, 'path', None)
+            if path:
+                adapters.append({"path": path, "weight": float(getattr(lora, 'weight', 1.0) or 1.0)})
+        if adapters:
+            payload["loras"] = adapters
+
+    _vid_progress_reset(int(payload["num_inference_steps"]))
+    try:
+        data = await asyncio.to_thread(h3_worker.generate, model_name, payload, model_cfg)
+    finally:
+        _vid_progress_done()
+
+    mp4_bytes = base64.b64decode(data.get("mp4_b64") or "")
+    if not mp4_bytes:
+        raise HTTPException(status_code=500, detail="MiniMax-H3 returned no video")
+    print(f"  [h3] {data.get('workflow')}: {data.get('num_frames')} frames "
+          f"({data.get('seconds')}s @ {data.get('fps')}fps), {len(mp4_bytes)/1e6:.1f} MB")
+
+    temps: list = []
+    try:
+        needs_post = any([
+            request.upscale_output, request.interpolate_output, request.add_audio,
+            request.tts_text, request.dialogs, request.lip_sync,
+            request.generate_subtitles, request.burn_subtitles,
+        ])
+        if needs_post:
+            mp4_bytes = await asyncio.get_event_loop().run_in_executor(
+                None, _postprocess_video, mp4_bytes, request, http_request, temps, warnings)
+    finally:
+        for path in temps:
+            try:
+                if os.path.isfile(path):
+                    os.unlink(path)
+            except Exception:
+                pass
+
+    result = _save_file(mp4_bytes, 'mp4', http_request)
+    try:
+        from codai.api.archive import archive_manager
+        asyncio.get_event_loop().create_task(asyncio.to_thread(
+            archive_manager.save_generation,
+            "video", "/v1/video/generations", request.model, request.prompt or "",
+            {"mode": request.mode, "workflow": data.get("workflow"),
+             "num_frames": data.get("num_frames"), "fps": data.get("fps"),
+             "width": data.get("width"), "height": data.get("height"),
+             "num_inference_steps": payload["num_inference_steps"],
+             "seed": request.seed},
+            [(mp4_bytes, "mp4")]))
+    except Exception:
+        pass
+
+    resp = VideoGenerationResponse(created=int(time.time()), data=[result])
+    if warnings:
+        resp.warnings = warnings
+    return resp
 
 
 async def _maybe_build_identity_keyframe(request: VideoGenerationRequest,
@@ -3733,6 +3881,20 @@ async def video_generations(request: VideoGenerationRequest,
     # Always define _model_cfg — it's used later regardless of whether we load now.
     _model_cfg = model_info.get('config') or {}
 
+    # ── MiniMax-H3 ───────────────────────────────────────────────────────────
+    # H3 is Modular-Diffusers-only and needs diffusers >= 0.40, so it runs in an
+    # isolated venv behind codai/api/h3_worker.py rather than through the loader
+    # below. Branch before any of it: the worker owns the weights, the VRAM
+    # reservation and the eviction hook.
+    try:
+        from codai.api import h3_worker
+        _is_h3 = h3_worker.is_h3_model(model_name, _model_cfg)
+    except Exception:
+        _is_h3 = False
+    if _is_h3:
+        return await _generate_h3(request, model_name, _model_cfg, http_request,
+                                  _identity_note)
+
     # Refuse to load onto a poisoned CUDA context — it would just re-assert.
     if getattr(multi_model_manager, 'cuda_context_poisoned', False):
         raise HTTPException(status_code=503, detail=(

codai/models/capabilities.py

@@ -116,6 +116,15 @@ def detect_model_capabilities(model_name: str) -> ModelCapabilities:
         return caps
 
     # ── Video generation ─────────────────────────────────────────────────────
+    # MiniMax-H3 generates video with a natively-synchronised soundtrack (one DiT
+    # denoises both), so it carries the audio capability too. Served through the
+    # isolated worker in codai/api/h3_worker.py, not the diffusers loader.
+    if 'minimax-h3' in n or 'minimax_h3' in n:
+        caps.video_generation = True
+        caps.image_to_video = True
+        caps.audio_generation = True
+        return caps
+
     if any(x in n for x in ['cogvideox', 'cogvideo', 'ltx-video', 'ltxvideo',
                               'hunyuan-video', 'mochi-1', 'dynamicrafter',
                               'animatediff', 'text2video', 'modelscope-t2v',

docs/minimax-h3.md

+# MiniMax-H3 (video + native audio)
+
+H3 is a 33B flow-matching DiT that denoises **video and audio in one packed
+sequence** — every generation comes back as an mp4 that already has a synchronised
+soundtrack, not a dub. It is served through an isolated worker rather than the
+normal diffusers loader, for one blunt reason: diffusers ships H3 as Modular
+Diffusers blocks only (`MiniMaxH3Blocks` / `MiniMaxH3ModularPipeline`, 0.40+),
+there is no `DiffusionPipeline` half, and the main venv is on diffusers 0.38.
+
+```
+/v1/video/generations ──► codai/api/video.py::_generate_h3
+                            └─► codai/api/h3_worker.py   (venv + process + eviction)
+                                  └─► tools/h3_service.py  (isolated venv, HTTP)
+                                        └─► ModularPipeline.from_pretrained(..., workflow=…)
+```
+
+Same shape as the pyannote, NeMo-Canary and vLLM workers.
+
+## Configure a model
+
+Add an entry to `models.json`. Detection is by name (anything containing
+`minimax-h3` / `minimax_h3`) or by an explicit backend:
+
+```json
+{
+  "minimax-h3": {
+    "type": "video",
+    "backend": "h3",
+    "model_path": "MiniMaxAI/MiniMax-H3",
+    "offload_strategy": "group",
+    "dtype": "bfloat16",
+    "used_vram_gb": 22.0
+  }
+}
+```
+
+| key | meaning |
+|---|---|
+| `offload_strategy` | `group` (default), `leaf` (smaller groups, slower), or `""` for straight-to-GPU. The diffusers CPU-offload strategies have no modular equivalent and are normalised to `group`. |
+| `device_map` | per-component device map (`auto`, `balanced`) instead of group offload |
+| `used_vram_gb` | reserved before start, and what eviction frees. Default 22. |
+| `h3_venv` | override the isolated venv location |
+| `gpu_device` | pins `CUDA_VISIBLE_DEVICES` for the worker |
+
+The venv builds itself on first use from `requirements-h3.txt` (several GB) under
+`/opt/coderai/h3_venv`, the `/cache` mount, or `~/.coderai/h3_venv`.
+
+## What the request translation does
+
+H3 has hard checkpoint contracts, so `_generate_h3` adapts the request instead of
+forwarding it, and reports every adaptation in the response `warnings`:
+
+- **Guidance-distilled** — no negative prompt, no guidance scale. Both are dropped.
+- **24 fps, fixed.** `num_frames` is snapped **up** to the next `17n + 5` the video
+  VAE can decode, clamped to 5–15 s. `fps` on the request is ignored.
+- **Canvas** axes are rounded down to a multiple of 32 (checkpoint short edge 768,
+  area cap 768×1344).
+- **Three workflows**, each loading its own ~61.7 GB transformer partition:
+  `t2va` (prompt), `fl2va` (first and/or last keyframe, `transformer/`), `ref2va`
+  (up to 12 references, `transformer_ref/`). The worker picks one per request and
+  **reloads** when it changes — loading with no workflow pulls *both* partitions.
+- **Identity is native**: `character_profiles` go in as `ref2va` image references
+  (max 9), so H3 skips the identity-keyframe bridge other video models need. An
+  explicit keyframe wins over references, because fl2va and ref2va are different
+  checkpoints and a request cannot be both.
+- `v2v` and `interp` are rejected — H3 has no such workflow.
+
+Post-processing (upscale, interpolation, dialogs, lip sync, subtitles) runs on top
+of the returned mp4 exactly as for any other video model, so a cloned voice line
+still lands over H3's own soundtrack.
+
+## LoRAs
+
+`loras` on the request are forwarded to the worker and loaded with
+`MiniMaxH3LoraLoaderMixin`, with `set_adapters` weights. Public recipes for
+training one: ~32 face crops at 512px, rank/alpha 16, LR 1e-4, ~1000 steps with
+the audio loss zeroed (stills carry no audio); or 50–200 clips of 3–15 s at
+*exactly* 24.000 fps with the audio kept, rank 16, 1500–5000 steps.
+
+## Operational notes
+
+- **Weights are ~62 GB per partition.** Budget disk and first-run download time.
+- The worker is registered as an evictable, VRAM-tracked model; eviction stops the
+  whole subprocess, which is the only way the transformer's pages actually go back.
+- Switching workflow within one model reloads the other partition — batch requests
+  by workflow if you care about wall-clock.
+- The service logs under `[h3]`; failures surface with the worker's last output.

requirements-h3.txt

+# Isolated venv for MiniMax-H3 video+audio generation.
+#
+# H3 exists ONLY as Modular Diffusers blocks (MiniMaxH3Blocks /
+# MiniMaxH3ModularPipeline), which landed in diffusers 0.40.0. The main coderai
+# venv is on diffusers 0.38.x, and moving the whole stack to 0.40 would shift the
+# floor under every Wan / LTX / Z-Image / SDXL pipeline at once — so H3 gets its
+# own venv, the same way pyannote, NeMo-Canary and vLLM do.
+# codai/api/h3_worker.py builds it (on demand under ~/.coderai/h3_venv or the
+# /cache mount) and runs tools/h3_service.py inside it.
+# Do NOT merge these into requirements.txt.
+
+# The transformer is 33B and the text encoder is Qwen3-VL, so this venv wants a
+# recent torch. Left unpinned deliberately: unlike pyannote there is no known-bad
+# version to dodge, and the H3 blocks track diffusers main.
+--extra-index-url https://download.pytorch.org/whl/cu130
+torch
+torchvision
+torchaudio
+
+# The floor for MiniMax-H3 support.
+diffusers>=0.40.0
+# Qwen3VLForConditionalGeneration (H3's text encoder) needs transformers 5.x.
+transformers>=5.0
+accelerate>=1.0
+safetensors>=0.4
+huggingface_hub>=0.34
+
+# Media: export_to_video writes the frames, soundfile writes the jointly-generated
+# soundtrack, and ffmpeg (system binary, already on PATH in the image) muxes them.
+imageio>=2.34
+imageio-ffmpeg>=0.5
+soundfile>=0.12
+Pillow>=10.0
+numpy
+
+# peft carries the LoRA adapters MiniMaxH3LoraLoaderMixin loads.
+peft>=0.14