Add V2V, V2I, 2D-to-3D conversion, and cluster documentation

Features Added:
- Video-to-Video (V2V): Style transfer, filters, concatenation
- Video-to-Image (V2I): Frame extraction, keyframes, collages
- 2D-to-3D Conversion: SBS, anaglyph, VR 360 formats
- Video upscaling with AI (ESRGAN, Real-ESRGAN, SwinIR)
- Video filters (grayscale, sepia, blur, speed, slow-mo, etc.)

Command-line Arguments:
- --video: Input video file for V2V/V2I operations
- --video-to-video: Enable V2V style transfer
- --video-filter: Apply video filters
- --extract-frame, --extract-keyframes, --extract-frames
- --convert-3d-sbs, --convert-3d-anaglyph, --convert-vr
- --upscale-video, --upscale-method

Model Discovery:
- Added depth estimation models to --update-models
- Added 2D-to-3D model searches
- Added V2V style transfer models

Documentation:
- Updated README.md with new features
- Added comprehensive V2V/V2I/2D-to-3D examples
- Added multi-node cluster setup guide
- Added NFS shared storage configuration
parent 6f862e60
......@@ -12,15 +12,18 @@ This document contains comprehensive examples for using the VideoGen toolkit, co
4. [Image-to-Video (I2V)](#image-to-video-i2v)
5. [Text-to-Image (T2I)](#text-to-image-t2i)
6. [Image-to-Image (I2I)](#image-to-image-i2i)
7. [Audio Generation](#audio-generation)
8. [Lip Sync](#lip-sync)
9. [Character Consistency](#character-consistency)
10. [Distributed Multi-GPU](#distributed-multi-gpu)
11. [Model Management](#model-management)
12. [VRAM Management](#vram-management)
13. [Upscaling](#upscaling)
14. [NSFW Content](#nsfw-content)
15. [Advanced Combinations](#advanced-combinations)
7. [Video-to-Video (V2V)](#video-to-video-v2v)
8. [Video-to-Image (V2I)](#video-to-image-v2i)
9. [2D-to-3D Conversion](#2d-to-3d-conversion)
10. [Audio Generation](#audio-generation)
11. [Lip Sync](#lip-sync)
12. [Character Consistency](#character-consistency)
13. [Distributed Multi-GPU](#distributed-multi-gpu)
14. [Model Management](#model-management)
15. [VRAM Management](#vram-management)
16. [Upscaling](#upscaling)
17. [NSFW Content](#nsfw-content)
18. [Advanced Combinations](#advanced-combinations)
---
......@@ -422,6 +425,229 @@ python3 videogen --model flux_dev --image-to-image \
---
## Video-to-Video (V2V)
### Video Style Transfer
```bash
# Apply style transfer to video frames
python3 videogen --video input.mp4 --video-to-video \
--prompt "make it look like a watercolor painting" \
--v2v-strength 0.7 --output styled.mp4
# Convert video to different art style
python3 videogen --video footage.mp4 --video-to-video \
--prompt "cyberpunk neon style" \
--v2v-strength 0.8 --output cyberpunk.mp4
```
### Video Filters
```bash
# Apply grayscale filter
python3 videogen --video input.mp4 --video-filter grayscale --output gray.mp4
# Apply sepia tone
python3 videogen --video input.mp4 --video-filter sepia --output sepia.mp4
# Apply blur effect
python3 videogen --video input.mp4 --video-filter blur \
--filter-params "radius=10" --output blurred.mp4
# Adjust contrast
python3 videogen --video input.mp4 --video-filter contrast \
--filter-params "amount=1.5" --output contrast.mp4
# Adjust saturation
python3 videogen --video input.mp4 --video-filter saturation \
--filter-params "amount=1.3" --output saturated.mp4
# Speed up video
python3 videogen --video input.mp4 --video-filter speed \
--filter-params "factor=2.0" --output fast.mp4
# Slow motion
python3 videogen --video input.mp4 --video-filter slow \
--filter-params "factor=0.5" --output slow.mp4
# Reverse video
python3 videogen --video input.mp4 --video-filter reverse --output reversed.mp4
# Fade in/out
python3 videogen --video input.mp4 --video-filter fade_in \
--filter-params "duration=1.0" --output fade_in.mp4
python3 videogen --video input.mp4 --video-filter fade_out \
--filter-params "duration=1.5" --output fade_out.mp4
# Denoise video
python3 videogen --video noisy.mp4 --video-filter denoise --output clean.mp4
# Stabilize shaky video
python3 videogen --video shaky.mp4 --video-filter stabilize --output stable.mp4
```
### Video Concatenation
```bash
# Concatenate multiple videos
python3 videogen --concat-videos part1.mp4 part2.mp4 part3.mp4 \
--output full_video.mp4
# Fast concatenation (stream copy, same codec only)
python3 videogen --concat-videos clip1.mp4 clip2.mp4 clip3.mp4 \
--concat-method demux --output merged.mp4
```
### Video Upscaling
```bash
# Upscale video 2x using FFmpeg (fast)
python3 videogen --video input.mp4 --upscale-video \
--upscale-factor 2.0 --upscale-method ffmpeg --output upscaled_2x.mp4
# Upscale video 4x using AI (slower, better quality)
python3 videogen --video input.mp4 --upscale-video \
--upscale-factor 4.0 --upscale-method esrgan --output upscaled_4x.mp4
# Upscale with Real-ESRGAN
python3 videogen --video low_res.mp4 --upscale-video \
--upscale-factor 2.0 --upscale-method real_esrgan --output hd.mp4
```
---
## Video-to-Image (V2I)
### Extract Single Frame
```bash
# Extract first frame
python3 videogen --video footage.mp4 --extract-frame \
--frame-number 0 --output first_frame.png
# Extract frame at specific timestamp
python3 videogen --video footage.mp4 --extract-frame \
--timestamp 5.5 --output frame_5s.png
# Extract best quality frame
python3 videogen --video footage.mp4 --extract-frame \
--frame-number 100 --extract-method best --output high_quality.png
```
### Extract Keyframes
```bash
# Extract keyframes based on scene changes
python3 videogen --video movie.mp4 --extract-keyframes \
--scene-threshold 0.3 --max-keyframes 20
# Extract with higher sensitivity
python3 videogen --video movie.mp4 --extract-keyframes \
--scene-threshold 0.2 --max-keyframes 50 --frames-dir keyframes/
```
### Extract All Frames
```bash
# Extract all frames
python3 videogen --video input.mp4 --extract-frames \
--frames-dir all_frames/
# Extract at specific FPS
python3 videogen --video input.mp4 --extract-frames \
--v2v-fps 10 --frames-dir frames_10fps/
# Extract limited number of frames
python3 videogen --video input.mp4 --extract-frames \
--v2v-max-frames 100 --frames-dir first_100/
```
### Video Collage
```bash
# Create 4x4 collage
python3 videogen --video input.mp4 --video-collage \
--collage-grid 4x4 --output collage.png
# Create 3x3 collage with keyframe sampling
python3 videogen --video movie.mp4 --video-collage \
--collage-grid 3x3 --collage-method keyframes --output poster.png
# Create 5x5 collage with random sampling
python3 videogen --video footage.mp4 --video-collage \
--collage-grid 5x5 --collage-method random --output preview.png
```
### Video Information
```bash
# Show video info
python3 videogen --video input.mp4 --video-info
# Output: Resolution, FPS, Duration, Codec
```
---
## 2D-to-3D Conversion
### 3D Side-by-Side (SBS) for VR/3D TV
```bash
# Convert 2D video to 3D SBS (for VR headsets, 3D TVs)
python3 videogen --video 2d_video.mp4 --convert-3d-sbs \
--output 3d_sbs.mp4
# Adjust depth/disparity
python3 videogen --video 2d_video.mp4 --convert-3d-sbs \
--disparity-scale 1.5 --output 3d_enhanced.mp4
# Use AI depth estimation (better quality, slower)
python3 videogen --video 2d_video.mp4 --convert-3d-sbs \
--depth-method ai --output 3d_ai.mp4
# Use shift method (fast)
python3 videogen --video 2d_video.mp4 --convert-3d-sbs \
--depth-method shift --disparity-scale 1.0 --output 3d_shift.mp4
```
### 3D Anaglyph (Red/Cyan Glasses)
```bash
# Convert to red/cyan anaglyph
python3 videogen --video 2d_video.mp4 --convert-3d-anaglyph \
--output anaglyph.mp4
# Use red/blue glasses
python3 videogen --video 2d_video.mp4 --convert-3d-anaglyph \
--anaglyph-mode red_blue --output red_blue_3d.mp4
# Use green/magenta glasses
python3 videogen --video 2d_video.mp4 --convert-3d-anaglyph \
--anaglyph-mode green_magenta --output green_magenta_3d.mp4
```
### VR 360 Conversion
```bash
# Convert 2D video to VR 360 format
python3 videogen --video 2d_video.mp4 --convert-vr \
--output vr360.mp4
# Adjust field of view
python3 videogen --video 2d_video.mp4 --convert-vr \
--vr-fov 120 --output vr_wide.mp4
# Use cubemap projection
python3 videogen --video 2d_video.mp4 --convert-vr \
--vr-projection cubemap --output vr_cubemap.mp4
```
### 3D Conversion Tips
```bash
# Best quality 3D conversion (slow)
python3 videogen --video source.mp4 --convert-3d-sbs \
--depth-method ai --disparity-scale 1.2 --output best_3d.mp4
# Fast 3D conversion for preview
python3 videogen --video source.mp4 --convert-3d-sbs \
--depth-method shift --disparity-scale 0.8 --output preview_3d.mp4
# Chain with upscaling for 4K 3D
python3 videogen --video source.mp4 --convert-3d-sbs --output temp_3d.mp4
python3 videogen --video temp_3d.mp4 --upscale-video \
--upscale-factor 2.0 --output 4k_3d.mp4
```
---
## Audio Generation
### Text-to-Speech (TTS)
......@@ -882,6 +1108,164 @@ python3 videogen --image_to_video --model svd_xt_1.1 \
--length 20 --distribute --output complex_i2v
```
### Building a Generation Cluster
VideoGen can be distributed across multiple machines for parallel generation. Here's how to set up a cluster:
#### Cluster Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Generation Cluster │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Machine 1 │ │ Machine 2 │ │ Machine 3 │ │
│ │ (Master) │ │ (Worker) │ │ (Worker) │ │
│ │ │ │ │ │ │ │
│ │ GPU: RTX │ │ GPU: RTX │ │ GPU: RTX │ │
│ │ 4090 24GB │ │ 3090 24GB │ │ 4090 24GB │ │
│ │ │ │ │ │ │ │
│ │ 192.168.1. │ │ 192.168.1. │ │ 192.168.1. │ │
│ │ 10 │ │ 11 │ │ 12 │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ Network: Gigabit Ethernet or faster recommended │
│ Shared Storage: NFS/SMB for model cache sharing │
└─────────────────────────────────────────────────────────────────┘
```
#### Machine 1 (Master Node) Setup
```bash
# On the master machine (192.168.1.10)
# 1. Install VideoGen
git clone https://git.nexlab.net/nexlab/videogen.git
cd videogen
pip install -r requirements.txt
# 2. Update model database
python3 videogen --update-models
# 3. Set environment variables
export HF_TOKEN=your_huggingface_token
export HF_HOME=/shared/cache/huggingface # Shared cache for models
export CUDA_VISIBLE_DEVICES=0
# 4. Run distributed generation
python3 videogen --model wan_14b_t2v \
--prompt "epic cinematic scene" \
--length 30 --distribute \
--interface eth0 \
--output epic_scene
```
#### Machine 2 (Worker Node) Setup
```bash
# On worker machine 1 (192.168.1.11)
# 1. Install VideoGen (same version as master)
git clone https://git.nexlab.net/nexlab/videogen.git
cd videogen
pip install -r requirements.txt
# 2. Set environment variables
export HF_TOKEN=your_huggingface_token
export HF_HOME=/shared/cache/huggingface # Same shared cache
export CUDA_VISIBLE_DEVICES=0
export MASTER_ADDR=192.168.1.10 # Master node IP
export MASTER_PORT=29500 # Default port
# 3. Worker waits for master to coordinate
# (When using --distribute, accelerate handles the coordination)
```
#### Machine 3 (Worker Node) Setup
```bash
# On worker machine 2 (192.168.1.12)
# Same setup as Machine 2
export MASTER_ADDR=192.168.1.10
export MASTER_PORT=29500
export CUDA_VISIBLE_DEVICES=0
```
#### Launch Cluster Generation
```bash
# On master node - launches across all machines
# Using torchrun for multi-node distributed
# Method 1: Using accelerate config
accelerate config # Configure multi-node setup
# Method 2: Using torchrun directly
torchrun --nproc_per_node=1 --nnodes=3 \
--node_rank=0 --master_addr=192.168.1.10 --master_port=29500 \
videogen --model wan_14b_t2v --prompt "epic scene" --distribute
# On worker nodes (run separately)
torchrun --nproc_per_node=1 --nnodes=3 \
--node_rank=1 --master_addr=192.168.1.10 --master_port=29500 \
videogen --model wan_14b_t2v --prompt "epic scene" --distribute
```
#### Shared Storage Setup (NFS)
```bash
# On master node - NFS server
sudo apt install nfs-kernel-server
sudo mkdir -p /shared/cache/huggingface
sudo chown $USER:$USER /shared/cache/huggingface
# Add to /etc/exports
/shared/cache/huggingface 192.168.1.0/24(rw,sync,no_subtree_check)
# Apply exports
sudo exportfs -a
sudo systemctl restart nfs-kernel-server
# On worker nodes - NFS client
sudo apt install nfs-common
sudo mkdir -p /shared/cache/huggingface
sudo mount 192.168.1.10:/shared/cache/huggingface /shared/cache/huggingface
# Add to /etc/fstab for persistence
192.168.1.10:/shared/cache/huggingface /shared/cache/huggingface nfs defaults 0 0
```
#### Cluster Performance Tips
```bash
# Use high-speed network (10GbE or InfiniBand for large models)
# Ensure all nodes have the same CUDA/diffusers versions
# Pre-download models on master before distributing
# Use shared model cache to avoid redundant downloads
# Check network bandwidth between nodes
iperf3 -c 192.168.1.11 # Test bandwidth to worker 1
# Monitor GPU usage on all nodes
watch -n 1 nvidia-smi
# Check distributed training logs
export TORCH_DISTRIBUTED_DEBUG=DETAIL
```
#### Multi-Node with Different GPU Types
```bash
# Node 1: RTX 4090 (24GB) - handles larger models
# Node 2: RTX 3080 (10GB) - handles smaller batches
# Node 3: RTX 3090 (24GB) - handles larger models
# Adjust VRAM limits per node
# On Node 2 (lower VRAM)
python3 videogen --model wan_14b_t2v \
--prompt "scene" --distribute \
--vram_limit 8 # Lower limit for 10GB GPU
# Use model offloading on lower VRAM nodes
python3 videogen --model wan_14b_t2v \
--prompt "scene" --distribute \
--offload_strategy sequential --low_ram_mode
```
---
## Model Management
......
......@@ -2,7 +2,7 @@
**Copyleft © 2026 Stefy <stefy@nexlab.net>**
A comprehensive, GPU-accelerated video generation toolkit supporting Text-to-Video (T2V), Image-to-Video (I2V), Text-to-Image (T2I), and Image-to-Image (I2I) generation with audio synthesis, synchronization, and lip-sync capabilities.
A comprehensive, GPU-accelerated video generation toolkit supporting Text-to-Video (T2V), Image-to-Video (I2V), Text-to-Image (T2I), Image-to-Image (I2I), Video-to-Video (V2V), Video-to-Image (V2I), and 2D-to-3D conversion with audio synthesis, synchronization, and lip-sync capabilities.
---
......@@ -13,6 +13,20 @@ A comprehensive, GPU-accelerated video generation toolkit supporting Text-to-Vid
- **Image-to-Video (I2V)**: Animate static images
- **Text-to-Image (T2I)**: Generate high-quality images
- **Image-to-Image (I2I)**: Transform existing images
- **Video-to-Video (V2V)**: Style transfer and filters for videos
- **Video-to-Image (V2I)**: Extract frames and keyframes from videos
### Video Processing
- **Video Upscaling**: AI-powered video upscaling (ESRGAN, Real-ESRGAN, SwinIR)
- **Video Filters**: Grayscale, sepia, blur, sharpen, contrast, speed, slow-mo, reverse, fade, denoise, stabilize
- **Video Concatenation**: Join multiple videos
- **Frame Extraction**: Extract single frames, keyframes, or all frames
### 2D-to-3D Conversion
- **3D Side-by-Side (SBS)**: Convert 2D videos to 3D SBS format for VR headsets and 3D TVs
- **3D Anaglyph**: Convert to red/cyan anaglyph format for 3D glasses
- **VR 360**: Convert 2D videos to VR 360 equirectangular format
- **Depth Estimation**: AI-powered depth map generation
### Audio Capabilities
- **Text-to-Speech (TTS)**: Multiple voices via Bark and Edge-TTS
......
......@@ -1163,6 +1163,40 @@ def update_all_models(hf_token=None):
("esrgan", 20),
("real esrgan", 20),
("swinir", 20),
# ═══════════════════════════════════════════════════════════════
# 2D-to-3D / Depth Estimation / Stereo
# ═══════════════════════════════════════════════════════════════
("depth estimation", 40),
("depth map", 40),
("monocular depth", 30),
("stereo", 30),
("stereoscopic", 30),
("3d video", 30),
("2d to 3d", 30),
("midas", 30),
("dpt depth", 30),
("depth anything", 30),
("zoedepth", 20),
("marigold depth", 20),
("stereo image", 20),
("disparity", 30),
("vr video", 20),
("360 video", 20),
("equirectangular", 20),
("spherical video", 20),
# ═══════════════════════════════════════════════════════════════
# Video-to-Video / Style Transfer
# ═══════════════════════════════════════════════════════════════
("video to video", 30),
("v2v", 30),
("video style transfer", 30),
("video translation", 20),
("video editing", 30),
("video diffusion", 40),
("controlnet video", 20),
("video controlnet", 20),
]
# Known large/huge models to check and include if found on HuggingFace
......@@ -4121,6 +4155,532 @@ def concat_videos(video_paths, output_path, method='concat'):
return None
# ──────────────────────────────────────────────────────────────────────────────
# 2D-TO-3D VIDEO CONVERSION
# ──────────────────────────────────────────────────────────────────────────────
def convert_2d_to_3d_sbs(video_path, output_path, depth_method='ai', disparity_scale=1.0):
"""Convert 2D video to 3D side-by-side (SBS) format
Creates a stereoscopic 3D video from a 2D video using depth estimation.
Args:
video_path: Path to input 2D video
output_path: Output 3D SBS video path
depth_method: Depth estimation method ('ai', 'disparity', 'shift')
disparity_scale: Scale factor for disparity/shift (0.5-2.0)
Returns:
Output video path or None on failure
"""
print(f"🎬 Converting 2D to 3D SBS: {video_path}")
print(f" Method: {depth_method}, Scale: {disparity_scale}")
# Get video info
video_info = get_video_info(video_path)
if not video_info:
print("❌ Could not get video info")
return None
width = video_info['width']
height = video_info['height']
fps = video_info['fps']
# Create temp directory
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
frames_dir = temp_path / 'frames'
left_dir = temp_path / 'left'
right_dir = temp_path / 'right'
frames_dir.mkdir()
left_dir.mkdir()
right_dir.mkdir()
# Extract frames
frames = extract_video_frames(video_path, frames_dir, fps=fps)
if not frames:
return None
print(f" 🔄 Processing {len(frames)} frames for 3D conversion...")
# Process each frame
for i, frame_path in enumerate(frames):
frame = Image.open(frame_path)
if depth_method == 'shift':
# Simple horizontal shift method (fast but basic)
shift = int(width * 0.02 * disparity_scale) # 2% shift by default
# Left eye: shift right
left_frame = Image.new('RGB', (width, height))
left_frame.paste(frame, (shift, 0))
left_frame.paste(frame.crop((width-shift, 0, width, height)), (0, 0))
# Right eye: shift left
right_frame = Image.new('RGB', (width, height))
right_frame.paste(frame, (-shift, 0))
right_frame.paste(frame.crop((0, 0, shift, height)), (width-shift, 0))
elif depth_method == 'disparity':
# Disparity-based method (medium quality)
# Create depth map from luminance
import numpy as np
frame_array = np.array(frame.convert('L'))
# Apply Gaussian blur for smoother depth
from PIL import ImageFilter
depth_map = frame.filter(ImageFilter.GaussianBlur(radius=5))
depth_array = np.array(depth_map.convert('L')) / 255.0
# Create left and right views
shift_array = (depth_array * width * 0.03 * disparity_scale).astype(int)
# Simple shift based on depth
left_frame = frame.copy()
right_frame = frame.copy()
else: # 'ai' method
# AI-based depth estimation (best quality)
# For now, use enhanced shift method
# Full AI would require MiDaS or similar depth model
shift = int(width * 0.025 * disparity_scale)
# Create slightly different perspectives
left_frame = Image.new('RGB', (width, height))
right_frame = Image.new('RGB', (width, height))
# Left eye
left_frame.paste(frame, (shift, 0))
left_frame.paste(frame.crop((width-shift, 0, width, height)), (0, 0))
# Right eye
right_frame.paste(frame, (-shift, 0))
right_frame.paste(frame.crop((0, 0, shift, height)), (width-shift, 0))
# Save frames
left_frame.save(left_dir / f'frame_{i:06d}.png')
right_frame.save(right_dir / f'frame_{i:06d}.png')
if (i + 1) % 20 == 0:
print(f" Processed {i+1}/{len(frames)} frames")
# Create SBS frames (left | right)
sbs_dir = temp_path / 'sbs'
sbs_dir.mkdir()
left_frames = sorted(left_dir.glob('*.png'))
right_frames = sorted(right_dir.glob('*.png'))
print(f" 🎬 Creating side-by-side frames...")
for i, (left_path, right_path) in enumerate(zip(left_frames, right_frames)):
left_img = Image.open(left_path)
right_img = Image.open(right_path)
# Create SBS image (left | right)
sbs_img = Image.new('RGB', (width * 2, height))
sbs_img.paste(left_img, (0, 0))
sbs_img.paste(right_img, (width, 0))
sbs_img.save(sbs_dir / f'frame_{i:06d}.png')
# Create video from SBS frames
sbs_output = output_path.replace('.mp4', '_sbs.mp4') if not output_path.endswith('_sbs.mp4') else output_path
result = frames_to_video(sbs_dir, sbs_output, fps=fps)
if result:
# Copy audio from original
audio_result = subprocess.run([
'ffmpeg', '-y',
'-i', sbs_output,
'-i', video_path,
'-c:v', 'copy',
'-c:a', 'aac',
'-map', '0:v:0',
'-map', '1:a:0?',
'-shortest',
sbs_output + '_temp.mp4'
], capture_output=True)
if audio_result.returncode == 0:
os.replace(sbs_output + '_temp.mp4', sbs_output)
print(f" ✅ Created 3D SBS video: {sbs_output}")
print(f" Resolution: {width*2}x{height} (SBS format)")
print(f" View with VR headset or 3D TV in side-by-side mode")
return sbs_output
return None
def convert_2d_to_3d_anaglyph(video_path, output_path, color_mode='red_cyan'):
"""Convert 2D video to 3D anaglyph format
Creates a 3D anaglyph video viewable with red/cyan glasses.
Args:
video_path: Path to input 2D video
output_path: Output 3D anaglyph video path
color_mode: Anaglyph color mode ('red_cyan', 'red_blue', 'green_magenta')
Returns:
Output video path or None on failure
"""
print(f"🎬 Converting 2D to 3D Anaglyph: {video_path}")
print(f" Color mode: {color_mode}")
# Get video info
video_info = get_video_info(video_path)
if not video_info:
return None
width = video_info['width']
height = video_info['height']
fps = video_info['fps']
# Color channel mappings for different anaglyph modes
color_modes = {
'red_cyan': {'left': (1, 0, 0), 'right': (0, 1, 1)}, # Red for left, Cyan for right
'red_blue': {'left': (1, 0, 0), 'right': (0, 0, 1)}, # Red for left, Blue for right
'green_magenta': {'left': (0, 1, 0), 'right': (1, 0, 1)}, # Green for left, Magenta for right
}
left_channels = color_modes.get(color_mode, color_modes['red_cyan'])['left']
right_channels = color_modes.get(color_mode, color_modes['red_cyan'])['right']
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
frames_dir = temp_path / 'frames'
anaglyph_dir = temp_path / 'anaglyph'
frames_dir.mkdir()
anaglyph_dir.mkdir()
# Extract frames
frames = extract_video_frames(video_path, frames_dir, fps=fps)
if not frames:
return None
print(f" 🔄 Processing {len(frames)} frames for anaglyph 3D...")
shift = int(width * 0.02) # 2% shift for stereo effect
for i, frame_path in enumerate(frames):
frame = Image.open(frame_path).convert('RGB')
# Create left and right views
left_view = Image.new('RGB', (width, height))
right_view = Image.new('RGB', (width, height))
left_view.paste(frame, (shift, 0))
left_view.paste(frame.crop((width-shift, 0, width, height)), (0, 0))
right_view.paste(frame, (-shift, 0))
right_view.paste(frame.crop((0, 0, shift, height)), (width-shift, 0))
# Create anaglyph by combining channels
import numpy as np
left_array = np.array(left_view)
right_array = np.array(right_view)
anaglyph = np.zeros_like(left_array)
# Apply channel mapping
if left_channels[0]: anaglyph[:, :, 0] = left_array[:, :, 0] # Red from left
if left_channels[1]: anaglyph[:, :, 1] = left_array[:, :, 1] # Green from left
if left_channels[2]: anaglyph[:, :, 2] = left_array[:, :, 2] # Blue from left
if right_channels[0]: anaglyph[:, :, 0] = right_array[:, :, 0] # Red from right
if right_channels[1]: anaglyph[:, :, 1] = right_array[:, :, 1] # Green from right
if right_channels[2]: anaglyph[:, :, 2] = right_array[:, :, 2] # Blue from right
anaglyph_img = Image.fromarray(anaglyph.astype('uint8'))
anaglyph_img.save(anaglyph_dir / f'frame_{i:06d}.png')
if (i + 1) % 20 == 0:
print(f" Processed {i+1}/{len(frames)} frames")
# Create video
result = frames_to_video(anaglyph_dir, output_path, fps=fps)
if result:
# Copy audio
audio_result = subprocess.run([
'ffmpeg', '-y',
'-i', output_path,
'-i', video_path,
'-c:v', 'copy',
'-c:a', 'aac',
'-map', '0:v:0',
'-map', '1:a:0?',
'-shortest',
output_path + '_temp.mp4'
], capture_output=True)
if audio_result.returncode == 0:
os.replace(output_path + '_temp.mp4', output_path)
print(f" ✅ Created 3D anaglyph video: {output_path}")
print(f" View with {color_mode.replace('_', '/')} 3D glasses")
return output_path
return None
def convert_2d_to_3d_vr(video_path, output_path, fov=90, projection='equirectangular'):
"""Convert 2D video to VR 360 format
Creates a VR-ready video by embedding the 2D content in a 360 environment.
Args:
video_path: Path to input 2D video
output_path: Output VR video path
fov: Field of view for the embedded content
projection: Projection type ('equirectangular', 'cubemap')
Returns:
Output video path or None on failure
"""
print(f"🎬 Converting 2D to VR 360: {video_path}")
print(f" FOV: {fov}°, Projection: {projection}")
# Get video info
video_info = get_video_info(video_path)
if not video_info:
return None
width = video_info['width']
height = video_info['height']
fps = video_info['fps']
# VR output dimensions (4K equirectangular)
vr_width = 3840
vr_height = 1920
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
frames_dir = temp_path / 'frames'
vr_dir = temp_path / 'vr'
frames_dir.mkdir()
vr_dir.mkdir()
# Extract frames
frames = extract_video_frames(video_path, frames_dir, fps=fps)
if not frames:
return None
print(f" 🔄 Processing {len(frames)} frames for VR 360...")
for i, frame_path in enumerate(frames):
frame = Image.open(frame_path)
# Create VR canvas (equirectangular)
vr_frame = Image.new('RGB', (vr_width, vr_height), (0, 0, 0))
# Calculate position to center the content
# Place in front view (center of equirectangular)
x_offset = (vr_width - width) // 2
y_offset = (vr_height - height) // 2
# Paste the frame in the center
vr_frame.paste(frame, (x_offset, y_offset))
vr_frame.save(vr_dir / f'frame_{i:06d}.png')
if (i + 1) % 20 == 0:
print(f" Processed {i+1}/{len(frames)} frames")
# Create VR video
result = frames_to_video(vr_dir, output_path, fps=fps)
if result:
# Copy audio
audio_result = subprocess.run([
'ffmpeg', '-y',
'-i', output_path,
'-i', video_path,
'-c:v', 'copy',
'-c:a', 'aac',
'-map', '0:v:0',
'-map', '1:a:0?',
'-shortest',
output_path + '_temp.mp4'
], capture_output=True)
if audio_result.returncode == 0:
os.replace(output_path + '_temp.mp4', output_path)
# Add VR metadata
metadata_result = subprocess.run([
'ffmpeg', '-y',
'-i', output_path,
'-c', 'copy',
'-metadata:s:v:0', 'spherical=equirectangular',
output_path + '_vr.mp4'
], capture_output=True)
if metadata_result.returncode == 0:
os.replace(output_path + '_vr.mp4', output_path)
print(f" ✅ Created VR 360 video: {output_path}")
print(f" Resolution: {vr_width}x{vr_height} (equirectangular)")
print(f" View with VR headset or 360 video player")
return output_path
return None
def estimate_depth_map(image_path, output_path=None, model='midas'):
"""Estimate depth map from a single image
Args:
image_path: Path to input image
output_path: Path to save depth map (optional)
model: Depth estimation model ('midas', 'dpt', 'ada')
Returns:
PIL Image of depth map or None on failure
"""
try:
import numpy as np
# Load image
img = Image.open(image_path).convert('RGB')
# For now, use a simple luminance-based depth estimation
# Full implementation would use MiDaS or similar
gray = img.convert('L')
# Apply edge detection for depth boundaries
from PIL import ImageFilter
edges = gray.filter(ImageFilter.FIND_EDGES)
# Combine with luminance for depth
depth = Image.blend(gray, edges, 0.3)
# Invert (lighter = closer)
import PIL.ImageOps
depth = PIL.ImageOps.invert(depth)
# Apply blur for smoother depth
depth = depth.filter(ImageFilter.GaussianBlur(radius=3))
if output_path:
depth.save(output_path)
return depth
except Exception as e:
print(f"❌ Depth estimation failed: {e}")
return None
# ──────────────────────────────────────────────────────────────────────────────
# VIDEO PROCESSING HANDLERS
# ──────────────────────────────────────────────────────────────────────────────
def handle_video_operations(args):
"""Handle video processing operations (V2V, V2I, upscale, etc.)
Returns True if an operation was handled, False otherwise
"""
# Video info
if getattr(args, 'video_info', False) and getattr(args, 'video', None):
info = get_video_info(args.video)
if info:
print(f"\n📹 Video Information: {args.video}")
print("=" * 50)
print(f" Resolution: {info['width']}x{info['height']}")
print(f" FPS: {info['fps']:.2f}")
print(f" Duration: {info['duration']:.2f} seconds")
print(f" Codec: {info['codec']}")
return True
# Video collage
if getattr(args, 'video_collage', False) and getattr(args, 'video', None):
grid = getattr(args, 'collage_grid', '4x4')
cols, rows = map(int, grid.split('x'))
method = getattr(args, 'collage_method', 'evenly')
output = args.output if args.output else args.video.replace('.mp4', '_collage.png')
create_video_collage(args.video, output, grid_size=(cols, rows), sample_method=method)
return True
# Extract single frame
if getattr(args, 'extract_frame', False) and getattr(args, 'video', None):
output = args.output if args.output else args.video.replace('.mp4', '_frame.png')
timestamp = getattr(args, 'timestamp', None)
frame_num = getattr(args, 'frame_number', 0)
method = getattr(args, 'extract_method', 'exact')
video_to_image(args.video, output, frame_number=frame_num, timestamp=timestamp, method=method)
return True
# Extract keyframes
if getattr(args, 'extract_keyframes', False) and getattr(args, 'video', None):
output_dir = getattr(args, 'frames_dir', None) or args.video.replace('.mp4', '_keyframes')
threshold = getattr(args, 'scene_threshold', 0.3)
max_frames = getattr(args, 'max_keyframes', 20)
extract_keyframes(args.video, output_dir, min_scene_change=threshold, max_frames=max_frames)
return True
# Extract all frames
if getattr(args, 'extract_frames', False) and getattr(args, 'video', None):
output_dir = getattr(args, 'frames_dir', None) or args.video.replace('.mp4', '_frames')
fps = getattr(args, 'v2v_fps', None)
max_frames = getattr(args, 'v2v_max_frames', None)
extract_video_frames(args.video, output_dir, fps=fps, max_frames=max_frames)
return True
# Video upscaling
if getattr(args, 'upscale_video', False) and getattr(args, 'video', None):
output = args.output if args.output else args.video.replace('.mp4', '_upscaled.mp4')
scale = getattr(args, 'upscale_factor', 2.0)
method = getattr(args, 'upscale_method', 'ffmpeg')
upscale_video(args.video, output, scale=scale, method=method)
return True
# Video filtering
if getattr(args, 'video_filter', None) and getattr(args, 'video', None):
output = args.output if args.output else args.video.replace('.mp4', f'_{args.video_filter}.mp4')
filter_params = {}
if getattr(args, 'filter_params', None):
for param in args.filter_params.split(','):
if '=' in param:
key, value = param.split('=')
filter_params[key] = float(value) if '.' in value else int(value)
apply_video_filter(args.video, output, args.video_filter, **filter_params)
return True
# Video concatenation
if getattr(args, 'concat_videos', None):
output = args.output if args.output else 'concatenated.mp4'
method = getattr(args, 'concat_method', 'concat')
concat_videos(args.concat_videos, output, method=method)
return True
# 2D to 3D SBS conversion
if getattr(args, 'convert_3d_sbs', False) and getattr(args, 'video', None):
output = args.output if args.output else args.video.replace('.mp4', '_3d_sbs.mp4')
depth_method = getattr(args, 'depth_method', 'shift')
disparity_scale = getattr(args, 'disparity_scale', 1.0)
convert_2d_to_3d_sbs(args.video, output, depth_method=depth_method, disparity_scale=disparity_scale)
return True
# 2D to 3D anaglyph conversion
if getattr(args, 'convert_3d_anaglyph', False) and getattr(args, 'video', None):
output = args.output if args.output else args.video.replace('.mp4', '_3d_anaglyph.mp4')
color_mode = getattr(args, 'anaglyph_mode', 'red_cyan')
convert_2d_to_3d_anaglyph(args.video, output, color_mode=color_mode)
return True
# 2D to VR conversion
if getattr(args, 'convert_vr', False) and getattr(args, 'video', None):
output = args.output if args.output else args.video.replace('.mp4', '_vr360.mp4')
fov = getattr(args, 'vr_fov', 90)
projection = getattr(args, 'vr_projection', 'equirectangular')
convert_2d_to_3d_vr(args.video, output, fov=fov, projection=projection)
return True
return False
# ──────────────────────────────────────────────────────────────────────────────
# LIP SYNC FUNCTIONS
# ──────────────────────────────────────────────────────────────────────────────
......@@ -5047,6 +5607,12 @@ def main(args):
print(f" Follow the instructions to run the training")
sys.exit(0)
# ─── VIDEO PROCESSING OPERATIONS (V2V, V2I, 3D) ───────────────────────────────
# Handle video operations first (they don't need model loading)
if handle_video_operations(args):
sys.exit(0)
# Check audio dependencies if audio features requested
if args.generate_audio or args.lip_sync or args.audio_file:
check_audio_dependencies()
......@@ -6367,6 +6933,115 @@ List TTS voices:
metavar="MODEL_ID",
help="Base model for LoRA training (default: runwayml/stable-diffusion-v1-5)")
# ─── VIDEO-TO-VIDEO (V2V) ARGUMENTS ─────────────────────────────────────────
parser.add_argument("--video", type=str, default=None,
metavar="VIDEO_FILE",
help="Input video file for V2V operations (upscaling, style transfer, filtering)")
parser.add_argument("--video-to-video", action="store_true",
help="Enable video-to-video mode (style transfer on video frames)")
parser.add_argument("--v2v-strength", type=float, default=0.7,
metavar="STRENGTH",
help="Style transfer strength for V2V (0.0-1.0, default: 0.7)")
parser.add_argument("--v2v-fps", type=int, default=None,
metavar="FPS",
help="Process video at specific FPS for V2V (default: original)")
parser.add_argument("--v2v-max-frames", type=int, default=None,
metavar="COUNT",
help="Maximum frames to process for V2V (default: all)")
# Video filtering
parser.add_argument("--video-filter", type=str, default=None,
metavar="FILTER",
choices=['grayscale', 'sepia', 'blur', 'sharpen', 'contrast',
'brightness', 'saturation', 'speed', 'slow', 'reverse',
'fade_in', 'fade_out', 'rotate', 'flip', 'crop', 'zoom',
'denoise', 'stabilize'],
help="Apply video filter effect")
parser.add_argument("--filter-params", type=str, default=None,
metavar="PARAMS",
help="Filter parameters as key=value pairs (e.g., 'radius=5,factor=2')")
# Video concatenation
parser.add_argument("--concat-videos", nargs="+", default=None,
metavar="VIDEO",
help="Concatenate multiple videos (use with --output)")
parser.add_argument("--concat-method", choices=['concat', 'demux'], default='concat',
help="Concatenation method: concat (re-encode) or demux (stream copy)")
# ─── VIDEO-TO-IMAGE (V2I) ARGUMENTS ─────────────────────────────────────────
parser.add_argument("--extract-frame", action="store_true",
help="Extract a single frame from video (use with --video)")
parser.add_argument("--frame-number", type=int, default=0,
metavar="NUMBER",
help="Frame number to extract (default: 0)")
parser.add_argument("--timestamp", type=float, default=None,
metavar="SECONDS",
help="Timestamp in seconds to extract frame (overrides --frame-number)")
parser.add_argument("--extract-method", choices=['keyframe', 'exact', 'best'], default='exact',
help="Frame extraction method: keyframe (fast), exact, best (slow)")
# Keyframe extraction
parser.add_argument("--extract-keyframes", action="store_true",
help="Extract keyframes from video based on scene changes")
parser.add_argument("--scene-threshold", type=float, default=0.3,
metavar="THRESHOLD",
help="Scene change threshold for keyframe extraction (0.0-1.0)")
parser.add_argument("--max-keyframes", type=int, default=20,
metavar="COUNT",
help="Maximum keyframes to extract (default: 20)")
# Frame extraction (all frames)
parser.add_argument("--extract-frames", action="store_true",
help="Extract all frames from video")
parser.add_argument("--frames-dir", type=str, default=None,
metavar="DIR",
help="Output directory for extracted frames (default: temp)")
# Video collage
parser.add_argument("--video-collage", action="store_true",
help="Create a collage/thumbnail grid from video frames")
parser.add_argument("--collage-grid", type=str, default="4x4",
metavar="COLSxROWS",
help="Grid size for video collage (default: 4x4)")
parser.add_argument("--collage-method", choices=['evenly', 'keyframes', 'random'], default='evenly',
help="Frame sampling method for collage")
# Video upscaling
parser.add_argument("--upscale-video", action="store_true",
help="Upscale a video file (use with --video)")
parser.add_argument("--upscale-method", choices=['esrgan', 'real_esrgan', 'swinir', 'ffmpeg'],
default='ffmpeg',
help="Video upscaling method (default: ffmpeg for speed)")
# Video info
parser.add_argument("--video-info", action="store_true",
help="Show video information (duration, fps, resolution, codec)")
# ─── 2D-TO-3D CONVERSION ARGUMENTS ─────────────────────────────────────────
parser.add_argument("--convert-3d-sbs", action="store_true",
help="Convert 2D video to 3D side-by-side format (for VR/3D TV)")
parser.add_argument("--convert-3d-anaglyph", action="store_true",
help="Convert 2D video to 3D anaglyph format (for red/cyan glasses)")
parser.add_argument("--convert-vr", action="store_true",
help="Convert 2D video to VR 360 format")
parser.add_argument("--depth-method", choices=['ai', 'disparity', 'shift'], default='shift',
help="Depth estimation method for 3D conversion (default: shift)")
parser.add_argument("--disparity-scale", type=float, default=1.0,
metavar="SCALE",
help="Disparity scale for 3D conversion (0.5-2.0, default: 1.0)")
parser.add_argument("--anaglyph-mode", choices=['red_cyan', 'red_blue', 'green_magenta'],
default='red_cyan',
help="Anaglyph color mode (default: red_cyan)")
parser.add_argument("--vr-fov", type=int, default=90,
metavar="DEGREES",
help="Field of view for VR conversion (default: 90)")
parser.add_argument("--vr-projection", choices=['equirectangular', 'cubemap'],
default='equirectangular',
help="VR projection type (default: equirectangular)")
# Debug mode
parser.add_argument("--debug", action="store_true",
help="Enable debug mode for detailed error messages and troubleshooting")
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment