an f32 default filled the card, and a decode OOM was called a load failure

Two image models "failed to load" on a 24 GB card mid-run. Neither had failed
to load: both loaded, and both completed every denoising step before dying on
the VAE decode.

pornmaster-pro-pony (SDXL, no precision on its entry) loaded at f32 —
"Using precision: f32 (torch.float32)" — finished 30/30 steps, then died on
"Tried to allocate 1008.00 MiB ... 704.69 MiB is free", with 22.12 GiB of a
23.56 GiB card held by its own weights. The same architecture at half width
measures 7.49 GB. Krea-2-Edit, correctly quantized (GGUF Q4_K_M transformer +
4-bit text encoder, bf16), finished 8/8 steps and died the same way on 190 MiB.

The bf16-for-an-undeclared-precision default was unreachable: ImageConfig
supplied precision "f32" into every per-model config, so by the time the loader
asked, nothing was ever undeclared. The default is now "auto" and resolves from
the card — a half precision at 32 GB or less, f32 above it, since a card big
enough to hold both loses nothing by upcasting. bf16 rather than fp16 for the
half type: same two bytes, but fp16 tops out near 65504 and a flow-matching DiT
runs past it into NaN and a black image (Z-Image-Turbo, 2026-10-25). An
explicit precision, on the card or the model, is still honoured.

The reporting was the costlier half. The generate call sat inside the load's
`except Exception`, so a decode OOM was filed as a load error, fell through to
the sd.cpp backend, and answered "Failed to load image model 'X'. Errors:
diffusers: CUDA out of memory". Two separate readers concluded from that wording
that VRAM eviction was leaking and that the models never loaded; neither was
true, and the real causes went unexamined for it. The same wording had already
mislabelled a seeded-request failure once — see models.utils.generator_device.
Generation now runs outside that handler and reports itself: 503 and "the model
loaded and ran, so this is not a load failure" for an OOM, naming the decode.

Also honour per-model `vae_tiling` on the image path. Slicing splits a BATCH
and so does nothing for n=1 — which is the case that OOMed; tiling is what
bounds the peak of one large decode.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/api/images.py

@@ -655,6 +655,82 @@ def _finish_image_load(pipeline, model_config):
     return pipeline
 
 
+def _is_cuda_oom(exc) -> bool:
+    """Whether this exception is the GPU running out of memory.
+
+    Matched by type where torch offers one and by text otherwise, because the
+    same condition arrives as ``torch.OutOfMemoryError``, as a bare
+    ``RuntimeError`` from older call paths, and as a diffusers error that only
+    wraps the message.
+    """
+    try:
+        import torch
+        if isinstance(exc, getattr(torch, "OutOfMemoryError", ())):
+            return True
+    except Exception:
+        pass
+    return "out of memory" in str(exc).lower()
+
+
+#: VRAM, in GB, at or below which an undeclared precision becomes a half one.
+#: A 24 GB card cannot hold an SDXL checkpoint at f32 and still decode its VAE;
+#: an 80 GB card can, and upcasting there costs nothing anyone will notice.
+AUTO_HALF_PRECISION_VRAM_GB = 32.0
+
+
+def _largest_visible_vram_gb() -> float:
+    """Total VRAM (GB) of the biggest card this engine may use, 0.0 if unknown.
+
+    Deliberately the LARGEST and not the sum: a pipeline loads onto one card
+    unless it is explicitly sharded, so two 24 GB cards do not make a 48 GB one.
+    """
+    try:
+        from codai.models.gpu_query import visible_gpu_memory
+        best = 0.0
+        for d in (visible_gpu_memory() or []):
+            best = max(best, float(d.get("total") or 0) / 1e9)
+        return best
+    except Exception:
+        return 0.0
+
+
+def _auto_precision() -> tuple:
+    """Resolve precision "auto" to (precision, why) from the card's size.
+
+    The policy, and the reason it is not simply "always half":
+
+    A checkpoint published at fp16/bf16 loaded at f32 takes twice the VRAM for
+    weights that were never computed at that width. On a 24 GB card that is the
+    difference between fitting and not: measured 2026-10-31 on zeiss,
+    pornmaster-pro-pony (SDXL, no declared precision) loaded at f32, completed
+    all 30 denoising steps, and then died on the VAE decode -- "Tried to
+    allocate 1008.00 MiB ... 704.69 MiB is free" -- with 22.12 GiB of a 23.56
+    GiB card held by its own weights. The same architecture at half width
+    measures 7.49 GB and leaves the decode room to breathe.
+
+    On a card big enough to hold both, f32 is harmless and is left alone: this
+    is a rule about not overflowing small cards, not a claim that half precision
+    is better. An unknown card size is treated as small, because that is the
+    case where guessing wrong is survivable.
+
+    bf16 rather than fp16 for the half type. They are both two bytes, so the
+    VRAM saving is identical, but fp16 tops out near 65504 and a flow-matching
+    DiT runs past it: the latents go NaN and the VAE decodes NaN to a uniform
+    black image, answered 200 OK with a plausible PNG and nothing in the log
+    (measured on Z-Image-Turbo, 2026-10-25). A model that wants fp16 can still
+    ask for it by name.
+    """
+    vram = _largest_visible_vram_gb()
+    if vram <= 0:
+        return 'bf16', "card size unknown — treating it as small"
+    if vram <= AUTO_HALF_PRECISION_VRAM_GB:
+        return 'bf16', (f"{vram:.0f} GB card at or under "
+                        f"{AUTO_HALF_PRECISION_VRAM_GB:.0f} GB — f32 would double "
+                        f"the weights of a checkpoint published at half width")
+    return 'f32', (f"{vram:.0f} GB card is over "
+                   f"{AUTO_HALF_PRECISION_VRAM_GB:.0f} GB — f32 fits")
+
+
 def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict = None):
     """
     Try to load a model using the diffusers library.
@@ -722,11 +798,12 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
     #
     # An explicit 'f32' on the card is still honoured: this changes the default,
     # not the operator's choice.
-    _precision_cfg = _cfg('precision')
-    precision = str(_precision_cfg or '').strip() or 'bf16'
-    if not _precision_cfg:
-        print(f"Using precision: {precision} (card states none; f32 would "
-              f"double the weights of a checkpoint published at half width)")
+    _precision_cfg = str(_cfg('precision') or '').strip()
+    _auto_why = ''
+    if not _precision_cfg or _precision_cfg.lower() == 'auto':
+        precision, _auto_why = _auto_precision()
+    else:
+        precision = _precision_cfg
     precision_map = {
         'bf16': torch.bfloat16,
         'f32': torch.float32,
@@ -753,7 +830,8 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
             print("--no-ram mode: precision was f32 -> using fp16 "
                   "(half the VRAM, and FlashAttention needs a half type)")
         else:
-            print(f"--no-ram mode: keeping the configured precision "
+            _src = f"auto ({_auto_why})" if _auto_why else "configured"
+            print(f"--no-ram mode: keeping the {_src} precision "
                   f"{precision} (fp16 would be the same size and a narrower "
                   f"range)")
     else:
@@ -776,7 +854,8 @@ def _load_diffusers_pipeline(model_name: str, global_args, model_config: dict =
                       "(f32 breaks FlashAttention and defeats quantization)")
                 dtype = torch.bfloat16
                 precision = 'bf16'
-        print(f"Using precision: {precision} ({dtype})")
+        _suffix = f" (auto: {_auto_why})" if _auto_why else ""
+        print(f"Using precision: {precision} ({dtype}){_suffix}")
     
     # CPU offload comes from the per-model configuration: an explicit
     # cpu_offload flag, or an offload_strategy that implies CPU offloading.
@@ -1627,6 +1706,17 @@ async def _generate_with_diffusers(pipeline, request, global_args, http_request=
     width, height = _parse_size(request.size, 1024, 1024, global_args)
 
     # Enable memory optimizations
+    #
+    # Slicing decodes the batch one image at a time; TILING decodes one image in
+    # overlapping tiles, and that is the one that bounds the peak of a large
+    # single image. Slicing alone does nothing for n=1, which is why a 768x1344
+    # decode could still ask for a contiguous 1008 MiB on a card its own weights
+    # had just filled (zeiss, 2026-10-31: pornmaster-pro-pony and Krea-2 both
+    # finished every denoising step and then died in the decode).
+    #
+    # Tiling is off unless the model asks for it, because it trades a little
+    # seam fidelity for the headroom and that is the operator's call -- the same
+    # `vae_tiling` key the video path reads, where it defaults ON instead.
     try:
         if hasattr(pipeline, 'enable_attention_slicing'):
             pipeline.enable_attention_slicing(slice_size="auto")
@@ -1634,6 +1724,16 @@ async def _generate_with_diffusers(pipeline, request, global_args, http_request=
             pipeline.vae.enable_slicing()
         elif hasattr(pipeline, 'enable_vae_slicing'):
             pipeline.enable_vae_slicing()
+        if (model_config or {}).get('vae_tiling'):
+            _tiled = False
+            if hasattr(pipeline, 'vae') and hasattr(pipeline.vae, 'enable_tiling'):
+                pipeline.vae.enable_tiling()
+                _tiled = True
+            elif hasattr(pipeline, 'enable_vae_tiling'):
+                pipeline.enable_vae_tiling()
+                _tiled = True
+            _state = 'enabled' if _tiled else 'requested, but this pipeline offers none'
+            print(f"  [image] VAE tiling {_state} (vae_tiling=true)")
     except Exception as e:
         print(f"Warning: Could not enable memory optimizations: {e}")
 
@@ -2575,6 +2675,9 @@ async def create_image_generation(request: ImageGenerationRequest, http_request:
                                cancellable=False, pausable=False)
 
         # Try diffusers first (for non-GGUF models)
+        # Set once the pipeline is in hand, so GENERATION runs outside the load's
+        # exception handler. See the note where it is consumed below.
+        _loaded_diffusers = None
         if not is_gguf:
             try:
                 print(f"Loading diffusers model: {model_name}")
@@ -2611,9 +2714,8 @@ async def create_image_generation(request: ImageGenerationRequest, http_request:
                     print(f"Loaded diffusers model: {model_name}")
 
                     _treg.finish(_ltid, "done")
-                    return await _generate_with_diffusers(pipeline, request, global_args,
-                                                          http_request, model_config=_diff_cfg)
-                    
+                    _loaded_diffusers = pipeline
+
             except HTTPException:
                 # The model loaded and the request itself was rejected (a bad
                 # size, an embedding that does not exist). Falling through to
@@ -2628,7 +2730,52 @@ async def create_image_generation(request: ImageGenerationRequest, http_request:
                 diffusers_error = str(e)
                 print(f"diffusers error: {diffusers_error}")
                 print(f"Traceback: {traceback.format_exc()}")
-        
+
+            # Generation runs HERE, outside the load's try, because a failure
+            # during generation is not a failure to load and must not be
+            # reported as one.
+            #
+            # It used to run inside that try. A model would load, denoise every
+            # step, and die on the final VAE decode -- and the handler above
+            # filed the decode's OOM as `diffusers_error`, fell through to the
+            # sd.cpp branch, and answered "Failed to load image model 'X'.
+            # Errors: diffusers: CUDA out of memory". Measured on zeiss
+            # 2026-10-31: pornmaster-pro-pony finished 30/30 steps and Krea-2
+            # finished 8/8, then both failed on a 1008 MiB and a 190 MiB
+            # allocation respectively -- the VAE decode of a 768x1344 image, not
+            # a weight load. Two separate readers concluded from that wording
+            # that eviction was leaking VRAM and that the models never loaded;
+            # neither was true, and the real causes (an f32 default and no
+            # decode headroom) went unexamined. The same wording had already
+            # mislabelled a seeded-request failure once before -- see
+            # :func:`codai.models.utils.generator_device`.
+            #
+            # A loaded model also means the sd.cpp fallback is wrong: the
+            # backend worked, so retrying the request on another one would
+            # answer a confused report about backends that never failed.
+            if _loaded_diffusers is not None:
+                try:
+                    return await _generate_with_diffusers(
+                        _loaded_diffusers, request, global_args,
+                        http_request, model_config=_diff_cfg)
+                except HTTPException:
+                    raise
+                except Exception as e:
+                    import traceback
+                    print(f"image generation error (model '{model_name}' IS "
+                          f"loaded): {e}")
+                    print(f"Traceback: {traceback.format_exc()}")
+                    _oom = _is_cuda_oom(e)
+                    _treg.finish(_ltid, "error", str(e))
+                    raise HTTPException(
+                        status_code=503 if _oom else 500,
+                        detail=(f"Image generation failed for '{model_name}' — "
+                                f"the model loaded and ran, so this is not a "
+                                f"load failure. "
+                                + (f"The GPU ran out of memory while finishing "
+                                   f"the image (typically the VAE decode): {e}"
+                                   if _oom else str(e))))
+
         # Try stable-diffusion-cpp-python (for GGUF models or as fallback)
         try:
             # For GGUF models or URLs, resolve the model path through the cache

codai/config.py

@@ -358,7 +358,10 @@ class ImageConfig:
     width: int = 512
     height: int = 512
     cfg_scale: float = 1.0
-    precision: str = "f32"
+    # "auto" = decide from the card (see codai.api.images._auto_precision):
+    # a card of 32 GB or less gets a half precision, a bigger one keeps f32.
+    # An explicit "f32"/"bf16"/"f16" here, or on a model, is always honoured.
+    precision: str = "auto"
     cpu_offload: bool = False
     seed: Optional[int] = None
     vae_tiling: bool = False

tests/test_full_pass_regressions.py

@@ -43,16 +43,23 @@ def test_an_image_card_without_a_precision_does_not_get_f32():
     down -- so a model's width in bytes came down to an unrelated flag."""
     src = _src("codai/api/images.py")
     assert "_cfg('precision', 'f32')" not in src
-    assert "precision = str(_precision_cfg or '').strip() or 'bf16'" in src
+    # The default is no longer a constant: it resolves from the card, because a
+    # card big enough to hold an f32 checkpoint loses nothing by upcasting
+    # while a 24 GB one cannot hold the weights AND decode the VAE. See
+    # tests/test_image_precision_auto_and_decode_errors.py for the policy and
+    # the measurement behind it.
+    assert "_auto_precision()" in src
+    assert "AUTO_HALF_PRECISION_VRAM_GB" in src
 
 
 def test_an_explicit_f32_is_still_honoured():
     """Changing a default must not overrule an operator who chose."""
     src = _src("codai/api/images.py")
-    head = src.index("_precision_cfg = _cfg('precision')")
+    head = src.index("_precision_cfg = str(_cfg('precision')")
     assert "_cfg('precision')" in src[head:head + 200]
-    # The default is applied only when the card states nothing.
-    assert "if not _precision_cfg:" in src
+    # The default is applied only when the card states nothing, or says "auto".
+    assert "if not _precision_cfg or _precision_cfg.lower() == 'auto':" in src
+    assert "precision = _precision_cfg" in src
 
 
 # ---------------------------------------------------------------------------

tests/test_image_precision_auto_and_decode_errors.py

+"""An f32 default filled a 24 GB card, and a decode OOM was called a load failure.
+
+Measured on zeiss, 2026-10-31, mid test run:
+
+* ``pornmaster-pro-pony`` (SDXL, no ``precision`` on its entry) loaded at
+  **f32**, completed 30/30 denoising steps, and then died on the VAE decode --
+  "Tried to allocate 1008.00 MiB ... 704.69 MiB is free", with 22.12 GiB of a
+  23.56 GiB card held by its own weights. The same architecture at half width
+  measures 7.49 GB.
+* ``Krea-2-Edit`` (GGUF Q4_K_M transformer + 4-bit text encoder, bf16) loaded
+  correctly, completed 8/8 steps, and died the same way on 190 MiB.
+
+Two things were wrong.
+
+1. The bf16 default added for an undeclared precision was unreachable: the
+   config layer always supplied ``image.precision = "f32"``, so there was no
+   such thing as "undeclared" by the time the loader looked. The default is now
+   ``"auto"`` and resolves from the card -- half on a card of 32 GB or less,
+   f32 above it, since a card big enough for both loses nothing by upcasting.
+
+2. Both failures were reported as ``Failed to load image model 'X'. Errors:
+   diffusers: CUDA out of memory`` -- because the generate call sat inside the
+   load's ``except Exception``, which also fell through to the sd.cpp backend.
+   The models had in fact loaded and run. Two separate readers concluded from
+   that wording that VRAM eviction was leaking; it was not, and the real causes
+   went unexamined for it.
+"""
+import ast
+
+import pytest
+
+from codai.api import images as images_mod
+from codai.config import ImageConfig
+
+
+SRC = open("codai/api/images.py").read()
+
+
+# --------------------------------------------------------------------------- #
+#  the default is a decision, not a constant
+# --------------------------------------------------------------------------- #
+
+def test_the_global_image_precision_default_is_auto_not_f32():
+    # Not cosmetic: while this said "f32" the loader's own "card states none"
+    # branch could never run, because the config layer filled the key in.
+    assert ImageConfig().precision == "auto"
+
+
+@pytest.mark.parametrize("vram_gb,expected", [
+    (8.0, "bf16"),      # RX 580 / small
+    (23.56, "bf16"),    # the 3090 this was measured on
+    (32.0, "bf16"),     # boundary: "32 or less" includes 32
+    (32.1, "f32"),      # just over
+    (48.0, "f32"),      # A6000
+    (80.0, "f32"),      # A100
+])
+def test_auto_precision_follows_the_card_size(monkeypatch, vram_gb, expected):
+    monkeypatch.setattr(images_mod, "_largest_visible_vram_gb", lambda: vram_gb)
+    precision, why = images_mod._auto_precision()
+    assert precision == expected
+    assert why, "the choice has to explain itself in the log"
+
+
+def test_an_unknown_card_is_treated_as_small():
+    # Guessing half on a big card wastes nothing; guessing f32 on a small one
+    # is the failure this whole module is about.
+    import unittest.mock as mock
+    with mock.patch.object(images_mod, "_largest_visible_vram_gb", lambda: 0.0):
+        assert images_mod._auto_precision()[0] == "bf16"
+
+
+def test_the_half_type_is_bf16_because_fp16_blacked_out_z_image():
+    import torch
+    import unittest.mock as mock
+    with mock.patch.object(images_mod, "_largest_visible_vram_gb", lambda: 24.0):
+        assert images_mod._auto_precision()[0] == "bf16"
+    # Same size, four orders of magnitude more range. See
+    # tests/test_black_image_and_precision.py for what fp16 cost.
+    assert torch.finfo(torch.bfloat16).bits == torch.finfo(torch.float16).bits
+    assert torch.finfo(torch.bfloat16).max > torch.finfo(torch.float16).max * 1000
+
+
+def test_largest_visible_vram_is_the_biggest_card_not_the_sum(monkeypatch):
+    # A pipeline loads onto ONE card unless explicitly sharded, so two 24 GB
+    # cards must not read as a 48 GB one and tip the policy over to f32.
+    monkeypatch.setattr(
+        "codai.models.gpu_query.visible_gpu_memory",
+        lambda: [{"total": 24e9}, {"total": 24e9}])
+    assert images_mod._largest_visible_vram_gb() == pytest.approx(24.0, abs=0.1)
+
+
+def test_an_explicit_precision_is_still_honoured():
+    # The policy changes the DEFAULT. An operator who wrote f32 gets f32.
+    body = SRC[SRC.index("_precision_cfg = str(_cfg('precision')"):][:400]
+    assert "else:" in body and "precision = _precision_cfg" in body
+
+
+# --------------------------------------------------------------------------- #
+#  a decode failure is not a load failure
+# --------------------------------------------------------------------------- #
+
+def test_generation_runs_outside_the_load_exception_handler():
+    # The structural fix. If the generate call is inside the try that catches
+    # load errors, a decode OOM is filed as a load error again.
+    tree = ast.parse(SRC)
+    fn = next(n for n in ast.walk(tree)
+              if isinstance(n, (ast.AsyncFunctionDef, ast.FunctionDef))
+              and n.name == "create_image_generation")
+
+    def generate_calls(node):
+        return [c for c in ast.walk(node)
+                if isinstance(c, ast.Call)
+                and getattr(c.func, "id", None) == "_generate_with_diffusers"]
+
+    for handler_owner in ast.walk(fn):
+        if not isinstance(handler_owner, ast.Try):
+            continue
+        names = {getattr(h.type, "id", None) for h in handler_owner.handlers}
+        if "ImportError" not in names:
+            continue        # not the load try
+        for call in generate_calls(handler_owner):
+            pytest.fail("_generate_with_diffusers is back inside the load try; "
+                        "a VAE-decode OOM will be reported as "
+                        "'Failed to load image model' again")
+
+
+def test_a_generation_failure_says_the_model_loaded():
+    assert "the model loaded and ran, so this is not a " in SRC
+    assert "load failure" in SRC
+
+
+def test_a_decode_oom_is_503_and_names_the_decode():
+    body = SRC[SRC.index("if _loaded_diffusers is not None:"):][:2000]
+    assert "503 if _oom else 500" in body
+    assert "VAE decode" in body
+
+
+@pytest.mark.parametrize("exc,expected", [
+    (RuntimeError("CUDA out of memory. Tried to allocate 1008.00 MiB"), True),
+    (RuntimeError("Out Of Memory"), True),
+    (ValueError("size must be a multiple of 8"), False),
+    (RuntimeError("CUDA error: device-side assert triggered"), False),
+])
+def test_is_cuda_oom_matches_the_wordings_this_actually_arrives_as(exc, expected):
+    assert images_mod._is_cuda_oom(exc) is expected
+
+
+def test_is_cuda_oom_matches_the_torch_type_too():
+    torch = pytest.importorskip("torch")
+    oom = getattr(torch, "OutOfMemoryError", None)
+    if oom is None:
+        pytest.skip("this torch has no OutOfMemoryError type")
+    assert images_mod._is_cuda_oom(oom("ran out")) is True
+
+
+# --------------------------------------------------------------------------- #
+#  tiling is what bounds a single large decode
+# --------------------------------------------------------------------------- #
+
+def test_vae_tiling_is_honoured_from_the_model_config():
+    body = SRC[SRC.index("# Enable memory optimizations"):][:2200]
+    assert "(model_config or {}).get('vae_tiling')" in body, \
+        "tiling has to be readable from the model's own config"
+    assert "enable_tiling" in body and "enable_vae_tiling" in body, \
+        "both the vae-level and pipeline-level diffusers APIs"
+
+
+def test_slicing_alone_is_not_enough_and_the_code_says_why():
+    body = SRC[SRC.index("# Enable memory optimizations"):][:2200]
+    # Slicing splits a BATCH; it does nothing for n=1, which is the case that
+    # OOMed. Whoever next trims these comments should know that.
+    assert "n=1" in body

tests/test_image_size_and_embeddings.py

@@ -412,8 +412,10 @@ def test_a_rejected_request_is_not_mistaken_for_a_failed_backend():
     head = src.index("Step 4: Model not loaded")
     tail = src.index("Step 5: Both backends failed")
     section = src[head:tail]
-    # one pass-through per backend attempt, each above its catch-all
-    assert section.count("except HTTPException:") == 2
+    # One pass-through per backend attempt, each above its catch-all -- plus a
+    # third for the generate call, which now runs outside the load's handler so
+    # a VAE-decode OOM is not reported as "Failed to load image model".
+    assert section.count("except HTTPException:") == 3
     for idx in (section.index("diffusers_error = str(e)\n                print(f\"diffusers error"),
                 section.index("sdcpp_error = str(e)\n            print(f\"sd.cpp error")):
         assert "except HTTPException:" in section[max(0, idx - 700):idx]

tests/test_prompt_prefix_and_ocr_upload_size.py

@@ -141,7 +141,7 @@ def test_the_prefix_reaches_the_encode_and_the_cache_key_together():
     prefixes would share one entry and the second would get the first's text."""
     src = open("codai/api/images.py").read()
     head = src.index("async def _generate_with_diffusers")
-    body = src[head:head + 9000]
+    body = src[head:head + 12000]   # the function grew; the two calls are still paired
     assert "prompt, neg_prompt = _effective_prompts(request, model_config)" in body
     assert "_embed_cache.get(prompt, neg_prompt, model_id)" in body
     assert "_embed_cache.get(request.prompt" not in body