• Stefy Lanza (nextime / spora )'s avatar
    an f32 default filled the card, and a decode OOM was called a load failure · 5afc48a2
    Stefy Lanza (nextime / spora ) authored
    Two image models "failed to load" on a 24 GB card mid-run. Neither had failed
    to load: both loaded, and both completed every denoising step before dying on
    the VAE decode.
    
    pornmaster-pro-pony (SDXL, no precision on its entry) loaded at f32 —
    "Using precision: f32 (torch.float32)" — finished 30/30 steps, then died on
    "Tried to allocate 1008.00 MiB ... 704.69 MiB is free", with 22.12 GiB of a
    23.56 GiB card held by its own weights. The same architecture at half width
    measures 7.49 GB. Krea-2-Edit, correctly quantized (GGUF Q4_K_M transformer +
    4-bit text encoder, bf16), finished 8/8 steps and died the same way on 190 MiB.
    
    The bf16-for-an-undeclared-precision default was unreachable: ImageConfig
    supplied precision "f32" into every per-model config, so by the time the loader
    asked, nothing was ever undeclared. The default is now "auto" and resolves from
    the card — a half precision at 32 GB or less, f32 above it, since a card big
    enough to hold both loses nothing by upcasting. bf16 rather than fp16 for the
    half type: same two bytes, but fp16 tops out near 65504 and a flow-matching DiT
    runs past it into NaN and a black image (Z-Image-Turbo, 2026-10-25). An
    explicit precision, on the card or the model, is still honoured.
    
    The reporting was the costlier half. The generate call sat inside the load's
    `except Exception`, so a decode OOM was filed as a load error, fell through to
    the sd.cpp backend, and answered "Failed to load image model 'X'. Errors:
    diffusers: CUDA out of memory". Two separate readers concluded from that wording
    that VRAM eviction was leaking and that the models never loaded; neither was
    true, and the real causes went unexamined for it. The same wording had already
    mislabelled a seeded-request failure once — see models.utils.generator_device.
    Generation now runs outside that handler and reports itself: 503 and "the model
    loaded and ran, so this is not a load failure" for an OOM, naming the decode.
    
    Also honour per-model `vae_tiling` on the image path. Slicing splits a BATCH
    and so does nothing for n=1 — which is the case that OOMed; tiling is what
    bounds the peak of one large decode.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    5afc48a2
config.py 174 KB