• Stefy Lanza (nextime / spora )'s avatar
    longcat: weight variants and the audio-driven avatar tasks (v0.2.35) · 000c2dcf
    Stefy Lanza (nextime / spora ) authored
    Step 6. Read from upstream's avatar demo rather than assumed, because almost nothing
    carries over from the base model:
    
    - avatar is a DIFFERENT pipeline class, LongCatVideoAvatarPipeline, with its own methods
      generate_at2v / generate_ai2v / generate_avc;
    - it takes TWO guidance scales — text and audio — where the base pipeline takes one;
    - its distilled pass is dmd_lora at 8 steps with lora_network_dim=128/alpha=64 and
      dit.enable_loras(["dmd"]), NOT cfg_step_lora at 16;
    - use_int8 is a separate load_quantized_dit(subfolder="base_model_int8"), not a dtype;
    - both INT8 and the DMD LoRA exist only for avatar-1.5, which also REQUIRES distilled
      sampling;
    - the audio is librosa.load(sr=16000) then pipe.get_audio_embedding(...), and the encoder
      differs per family (chinese-wav2vec2-base for Avatar, whisper-large-v3 for Avatar-1.5).
    
    Unsupported combinations are refused BEFORE loading, because the alternative is quietly
    serving something that is not what was asked for: INT8 or DMD on anything but
    avatar-1.5, avatar-1.5 without distilled sampling, a base-model mode on avatar weights or
    the reverse, and GGUF, which needs ComfyUI's gguf loader and says so.
    
    FP8 loads the DiT as float8_e4m3fn. Community layouts differ between publishers (Kijai's
    scaled format is not a plain e4m3 dump), so a repo that needs its own loader fails with
    that reason rather than producing noise.
    
    Also: librosa is a real new dependency and went into requirements-longcat.txt, not only
    into the code that imports it. And the avatar audio comes from the request's `audio_file`
    field — `add_audio` is a BOOLEAN for the post-processing path, and using it would have
    passed True into the base64 encoder.
    
    35 new tests, 188 across the six LongCat suites.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    000c2dcf
Name
Last commit
Last update
..
__init__.py Loading commit data...
_film_net.py Loading commit data...
_rife_ifnet.py Loading commit data...
app.py Loading commit data...
archive.py Loading commit data...
audio_backends.py Loading commit data...
audio_clean.py Loading commit data...
audio_gen.py Loading commit data...
audio_stems.py Loading commit data...
audiocraft_worker.py Loading commit data...
canary_worker.py Loading commit data...
characters.py Loading commit data...
colibri_worker.py Loading commit data...
compose.py Loading commit data...
crisperwhisper_worker.py Loading commit data...
custom_pipelines.py Loading commit data...
diarization.py Loading commit data...
ds4_kv_janitor.py Loading commit data...
ds4_worker.py Loading commit data...
embeddings.py Loading commit data...
environments.py Loading commit data...
faceswap.py Loading commit data...
fastlog.py Loading commit data...
h3_worker.py Loading commit data...
host_worker.py Loading commit data...
hub_compat.py Loading commit data...
images.py Loading commit data...
k3_worker.py Loading commit data...
kt_worker.py Loading commit data...
lipsync.py Loading commit data...
log.py Loading commit data...
longcat_worker.py Loading commit data...
lora_archs.py Loading commit data...
loras.py Loading commit data...
melotts_worker.py Loading commit data...
model_test.py Loading commit data...
models_transfer.py Loading commit data...
mux_remote.py Loading commit data...
ocr.py Loading commit data...
parler_worker.py Loading commit data...
pipelines.py Loading commit data...
pod_http.py Loading commit data...
pod_tls.py Loading commit data...
prompt_cache.py Loading commit data...
pyannote_worker.py Loading commit data...
ratelimit.py Loading commit data...
remote_gateway.py Loading commit data...
rerank.py Loading commit data...
runpod_client.py Loading commit data...
runpod_ledger.py Loading commit data...
runpod_worker.py Loading commit data...
spatial.py Loading commit data...
speaker_embeddings.py Loading commit data...
speaker_registry.py Loading commit data...
state.py Loading commit data...
stt_backends.py Loading commit data...
text.py Loading commit data...
transcriptions.py Loading commit data...
tts.py Loading commit data...
tts_backends.py Loading commit data...
uploads.py Loading commit data...
urlutils.py Loading commit data...
video.py Loading commit data...
vllm_worker.py Loading commit data...
voice_clone.py Loading commit data...
voice_convert.py Loading commit data...
vpr_salad_server.py Loading commit data...
xtts_worker.py Loading commit data...