• Stefy Lanza (nextime / spora )'s avatar
    images: quantized models must not load in f32 (FlashAttention needs fp16/bf16) · b3014a3d
    Stefy Lanza (nextime / spora ) authored
    Z-Image-Turbo-unsloth-bnb-4bit (and any pre-quantized bnb/fp8/nf4/gptq/awq
    checkpoint, or a runtime-quantized model) dequantizes to a HALF compute dtype
    and its transformer uses FlashAttention, which only supports fp16/bf16. The
    per-model image loader defaults precision to f32, so such a model loaded in
    float32 and crashed with "FlashAttention only support fp16 and bf16 data type"
    (image/character generation → 400/500), besides wasting VRAM.
    
    When precision is left at the f32 default AND the model is quantized (name
    contains bnb/4bit/8bit/fp8/nf4/gptq/awq, or config sets load_in_4bit/8bit/
    component_quantization), load in bf16 instead. Non-quantized models keep the f32
    default. --no-ram already forced fp16, so it's unaffected.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    b3014a3d
images.py 108 KB