• Stefy Lanza (nextime / spora )'s avatar
    loras: train Z-Image LoRA via 4-bit QLoRA (fast, uses the cached turbo build) · 26bdb59e
    Stefy Lanza (nextime / spora ) authored
    Training on the full bf16 Tongyi-MAI/Z-Image-Turbo was extremely slow — a ~10-min
    download plus a heavy bf16 model that doesn't fit cleanly on 24 GB. Switch _train_dit
    to QLoRA: load the transformer in 4-bit (frozen, ~4 GB, no CPU offload) and train the
    LoRA on top. This trains directly on the already-cached 4-bit (e.g. unsloth) build —
    no redirect to the full model, no download.
    
    - Load transformer with diffusers BitsAndBytesConfig (nf4); an already-4-bit
      checkpoint (embedded quant config, e.g. unsloth) is loaded as-is via fallback.
    - Enable gradient checkpointing and force the input-embedding (all_x_embedder)
      output to require grad so QLoRA grads reach the attention LoRA layers; hooks
      removed at job end.
    - Drop the quantized-base -> full-model redirect added earlier.
    
    LoRA still applies to the quantized model at inference (identical architecture).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    26bdb59e
loras.py 100 KB