• Stefy Lanza (nextime / spora )'s avatar
    images: quantized models must not load in f32 (FlashAttention needs fp16/bf16) · b3014a3d
    Stefy Lanza (nextime / spora ) authored
    Z-Image-Turbo-unsloth-bnb-4bit (and any pre-quantized bnb/fp8/nf4/gptq/awq
    checkpoint, or a runtime-quantized model) dequantizes to a HALF compute dtype
    and its transformer uses FlashAttention, which only supports fp16/bf16. The
    per-model image loader defaults precision to f32, so such a model loaded in
    float32 and crashed with "FlashAttention only support fp16 and bf16 data type"
    (image/character generation → 400/500), besides wasting VRAM.
    
    When precision is left at the f32 default AND the model is quantized (name
    contains bnb/4bit/8bit/fp8/nf4/gptq/awq, or config sets load_in_4bit/8bit/
    component_quantization), load in bf16 instead. Non-quantized models keep the f32
    default. --no-ram already forced fp16, so it's unaffected.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    b3014a3d
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...