• Stefy Lanza (nextime / spora )'s avatar
    longcat: weight variants and the audio-driven avatar tasks (v0.2.35) · 000c2dcf
    Stefy Lanza (nextime / spora ) authored
    Step 6. Read from upstream's avatar demo rather than assumed, because almost nothing
    carries over from the base model:
    
    - avatar is a DIFFERENT pipeline class, LongCatVideoAvatarPipeline, with its own methods
      generate_at2v / generate_ai2v / generate_avc;
    - it takes TWO guidance scales — text and audio — where the base pipeline takes one;
    - its distilled pass is dmd_lora at 8 steps with lora_network_dim=128/alpha=64 and
      dit.enable_loras(["dmd"]), NOT cfg_step_lora at 16;
    - use_int8 is a separate load_quantized_dit(subfolder="base_model_int8"), not a dtype;
    - both INT8 and the DMD LoRA exist only for avatar-1.5, which also REQUIRES distilled
      sampling;
    - the audio is librosa.load(sr=16000) then pipe.get_audio_embedding(...), and the encoder
      differs per family (chinese-wav2vec2-base for Avatar, whisper-large-v3 for Avatar-1.5).
    
    Unsupported combinations are refused BEFORE loading, because the alternative is quietly
    serving something that is not what was asked for: INT8 or DMD on anything but
    avatar-1.5, avatar-1.5 without distilled sampling, a base-model mode on avatar weights or
    the reverse, and GGUF, which needs ComfyUI's gguf loader and says so.
    
    FP8 loads the DiT as float8_e4m3fn. Community layouts differ between publishers (Kijai's
    scaled format is not a plain e4m3 dump), so a repo that needs its own loader fails with
    that reason rather than producing noise.
    
    Also: librosa is a real new dependency and went into requirements-longcat.txt, not only
    into the code that imports it. And the avatar audio comes from the request's `audio_file`
    field — `add_audio` is a BOOLEAN for the post-processing path, and using it would have
    passed True into the base64 encoder.
    
    35 new tests, 188 across the six LongCat suites.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    000c2dcf
requirements-longcat.txt 3.23 KB