C3: keep the soundtrack LTX generates, and stop reporting every file as silent

LTX-2.x is the one video family here with an audio VAE and a vocoder, so its
output is LTX2PipelineOutput(frames=..., audio=...) -- audio generated jointly
with the frames and therefore actually in step with them. Only `frames` was
read, so every LTX render came out silent and the only way to get sound was to
generate some separately with another model and mux that on top, which does not
match the picture. The generated track is now muxed in BEFORE the other
post-processing steps, because every one of them re-encodes the frames and the
track belongs to these exact ones.

`add_audio` becomes tri-state so "say nothing" and "say no" differ: unset keeps
what the model generated, false gives a silent video, true runs the older
separate-generation path. It defaulted to False, which cannot be told from an
explicit false -- so a straight default would have dropped the soundtrack. None
is falsy everywhere False was, so existing callers are unaffected.

Found while testing it: _has_audio tried `import av` under a bare
`except: return False`, and PyAV is in no main requirements file -- only the
isolated LongCat venv's. On an install without it EVERY file answered "no
audio", and because it gates _mux_original_audio, a clip's own sound was
silently dropped from every upscale and interpolate. The audio was there,
nothing could see it, and nothing said so. ffprobe cannot be missing the way
PyAV can, since ffmpeg is what does the muxing, so it is the fallback, and a
failure of both is logged instead of reported as silence.

Also: one of my own tests asserted a minimum number of ffmpeg progress reports.
`-progress` emits on a time interval, so a small encode can finish having
emitted a single block -- it passed alone and failed under the full suite. It
now asserts what is guaranteed: frames counted, counted upward.
Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
parent 96a84b69
This diff is collapsed.
......@@ -155,7 +155,22 @@ class VideoGenerationRequest(BaseModel):
loras: Optional[List[VideoLoraConfig]] = Field(None, description="Per-request LoRA adapters (e.g. trained per-character identity LoRAs) fused into the pipeline.")
# ── Audio generation / manipulation ──────────────────────────────────
add_audio: Optional[bool] = Field(False, description="Add an audio track to the generated video.")
#: Tri-state on purpose, so "say nothing" and "say no" are different.
#: None (default) - keep whatever the MODEL generated with the picture.
#: LTX-2.x has an audio VAE and returns a soundtrack in
#: step with its own frames; dropping it by default
#: would throw away the only audio that actually
#: matches the video.
#: False - a silent video, even when one was generated.
#: True - generate a track separately with an audio model and
#: mux that instead (the older post-processing path).
#: `False` was the default, and None is falsy in the same places, so every
#: existing caller behaves exactly as before.
add_audio: Optional[bool] = Field(
None, description="Audio track: leave unset to keep a soundtrack the "
"model generated with the video (LTX-2.x), false "
"for a silent video, true to generate one "
"separately with an audio model.")
audio_type: Optional[str] = Field(None, description="Audio to generate/add: 'music', 'speech', 'sfx' or 'ambient'.")
audio_prompt: Optional[str] = Field(None, description="Prompt for generated music/sfx.")
audio_file: Optional[str] = Field(None, description="Existing audio to add (base64/URL).")
......
......@@ -122,7 +122,12 @@ def test_run_reports_frames_written(tmp_path):
on_frame=seen.append, timeout=120)
assert seen, "ffmpeg wrote frames but none were reported"
assert seen == sorted(seen), f"the frame count went backwards: {seen}"
assert seen[-1] >= 10, f"only reached {seen[-1]} frames of ~20"
# Only that frames were counted, and counted upward. NOT how many reports
# arrive or how far they get: `-progress` emits on a time interval, so a
# small encode can finish having emitted a single block, and asserting a
# minimum made this pass alone and fail under the full suite -- a flake in
# the test, not in the hook.
assert seen[-1] >= 1, f"the reported frame count never advanced: {seen}"
@needs_ffmpeg
......
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment