• Stefy Lanza (nextime / spora )'s avatar
    longcat: weight variants and the audio-driven avatar tasks (v0.2.35) · 000c2dcf
    Stefy Lanza (nextime / spora ) authored
    Step 6. Read from upstream's avatar demo rather than assumed, because almost nothing
    carries over from the base model:
    
    - avatar is a DIFFERENT pipeline class, LongCatVideoAvatarPipeline, with its own methods
      generate_at2v / generate_ai2v / generate_avc;
    - it takes TWO guidance scales — text and audio — where the base pipeline takes one;
    - its distilled pass is dmd_lora at 8 steps with lora_network_dim=128/alpha=64 and
      dit.enable_loras(["dmd"]), NOT cfg_step_lora at 16;
    - use_int8 is a separate load_quantized_dit(subfolder="base_model_int8"), not a dtype;
    - both INT8 and the DMD LoRA exist only for avatar-1.5, which also REQUIRES distilled
      sampling;
    - the audio is librosa.load(sr=16000) then pipe.get_audio_embedding(...), and the encoder
      differs per family (chinese-wav2vec2-base for Avatar, whisper-large-v3 for Avatar-1.5).
    
    Unsupported combinations are refused BEFORE loading, because the alternative is quietly
    serving something that is not what was asked for: INT8 or DMD on anything but
    avatar-1.5, avatar-1.5 without distilled sampling, a base-model mode on avatar weights or
    the reverse, and GGUF, which needs ComfyUI's gguf loader and says so.
    
    FP8 loads the DiT as float8_e4m3fn. Community layouts differ between publishers (Kijai's
    scaled format is not a plain e4m3 dump), so a repo that needs its own loader fails with
    that reason rather than producing noise.
    
    Also: librosa is a real new dependency and went into requirements-longcat.txt, not only
    into the code that imports it. And the avatar audio comes from the request's `audio_file`
    field — `add_audio` is a BOOLEAN for the post-processing path, and using it would have
    passed True into the base64 encoder.
    
    35 new tests, 188 across the six LongCat suites.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    000c2dcf
Name
Last commit
Last update
..
audiogenrequest.py Loading commit data...
embedrequest.py Loading commit data...
imagerequest.py Loading commit data...
textrequest.py Loading commit data...
transcriptionrequest.py Loading commit data...
videorequest.py Loading commit data...