• Stefy Lanza (nextime / spora )'s avatar
    images: quantized models must not load in f32 (FlashAttention needs fp16/bf16) · b3014a3d
    Stefy Lanza (nextime / spora ) authored
    Z-Image-Turbo-unsloth-bnb-4bit (and any pre-quantized bnb/fp8/nf4/gptq/awq
    checkpoint, or a runtime-quantized model) dequantizes to a HALF compute dtype
    and its transformer uses FlashAttention, which only supports fp16/bf16. The
    per-model image loader defaults precision to f32, so such a model loaded in
    float32 and crashed with "FlashAttention only support fp16 and bf16 data type"
    (image/character generation → 400/500), besides wasting VRAM.
    
    When precision is left at the f32 default AND the model is quantized (name
    contains bnb/4bit/8bit/fp8/nf4/gptq/awq, or config sets load_in_4bit/8bit/
    component_quantization), load in bf16 instead. Non-quantized models keep the f32
    default. --no-ram already forced fp16, so it's unaffected.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    b3014a3d
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...