an f32 default filled the card, and a decode OOM was called a load failure

Two image models "failed to load" on a 24 GB card mid-run. Neither had failed
to load: both loaded, and both completed every denoising step before dying on
the VAE decode.

pornmaster-pro-pony (SDXL, no precision on its entry) loaded at f32 —
"Using precision: f32 (torch.float32)" — finished 30/30 steps, then died on
"Tried to allocate 1008.00 MiB ... 704.69 MiB is free", with 22.12 GiB of a
23.56 GiB card held by its own weights. The same architecture at half width
measures 7.49 GB. Krea-2-Edit, correctly quantized (GGUF Q4_K_M transformer +
4-bit text encoder, bf16), finished 8/8 steps and died the same way on 190 MiB.

The bf16-for-an-undeclared-precision default was unreachable: ImageConfig
supplied precision "f32" into every per-model config, so by the time the loader
asked, nothing was ever undeclared. The default is now "auto" and resolves from
the card — a half precision at 32 GB or less, f32 above it, since a card big
enough to hold both loses nothing by upcasting. bf16 rather than fp16 for the
half type: same two bytes, but fp16 tops out near 65504 and a flow-matching DiT
runs past it into NaN and a black image (Z-Image-Turbo, 2026-10-25). An
explicit precision, on the card or the model, is still honoured.

The reporting was the costlier half. The generate call sat inside the load's
`except Exception`, so a decode OOM was filed as a load error, fell through to
the sd.cpp backend, and answered "Failed to load image model 'X'. Errors:
diffusers: CUDA out of memory". Two separate readers concluded from that wording
that VRAM eviction was leaking and that the models never loaded; neither was
true, and the real causes went unexamined for it. The same wording had already
mislabelled a seeded-request failure once — see models.utils.generator_device.
Generation now runs outside that handler and reports itself: 503 and "the model
loaded and ran, so this is not a load failure" for an OOM, naming the decode.

Also honour per-model `vae_tiling` on the image path. Slicing splits a BATCH
and so does nothing for n=1 — which is the case that OOMed; tiling is what
bounds the peak of one large decode.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/config.py

@@ -358,7 +358,10 @@ class ImageConfig:
     width: int = 512
     height: int = 512
     cfg_scale: float = 1.0
-    precision: str = "f32"
+    # "auto" = decide from the card (see codai.api.images._auto_precision):
+    # a card of 32 GB or less gets a half precision, a bigger one keeps f32.
+    # An explicit "f32"/"bf16"/"f16" here, or on a model, is always honoured.
+    precision: str = "auto"
     cpu_offload: bool = False
     seed: Optional[int] = None
     vae_tiling: bool = False

tests/test_full_pass_regressions.py

@@ -43,16 +43,23 @@ def test_an_image_card_without_a_precision_does_not_get_f32():
     down -- so a model's width in bytes came down to an unrelated flag."""
     src = _src("codai/api/images.py")
     assert "_cfg('precision', 'f32')" not in src
-    assert "precision = str(_precision_cfg or '').strip() or 'bf16'" in src
+    # The default is no longer a constant: it resolves from the card, because a
+    # card big enough to hold an f32 checkpoint loses nothing by upcasting
+    # while a 24 GB one cannot hold the weights AND decode the VAE. See
+    # tests/test_image_precision_auto_and_decode_errors.py for the policy and
+    # the measurement behind it.
+    assert "_auto_precision()" in src
+    assert "AUTO_HALF_PRECISION_VRAM_GB" in src
 
 
 def test_an_explicit_f32_is_still_honoured():
     """Changing a default must not overrule an operator who chose."""
     src = _src("codai/api/images.py")
-    head = src.index("_precision_cfg = _cfg('precision')")
+    head = src.index("_precision_cfg = str(_cfg('precision')")
     assert "_cfg('precision')" in src[head:head + 200]
-    # The default is applied only when the card states nothing.
-    assert "if not _precision_cfg:" in src
+    # The default is applied only when the card states nothing, or says "auto".
+    assert "if not _precision_cfg or _precision_cfg.lower() == 'auto':" in src
+    assert "precision = _precision_cfg" in src
 
 
 # ---------------------------------------------------------------------------

tests/test_image_precision_auto_and_decode_errors.py

+"""An f32 default filled a 24 GB card, and a decode OOM was called a load failure.
+
+Measured on zeiss, 2026-10-31, mid test run:
+
+* ``pornmaster-pro-pony`` (SDXL, no ``precision`` on its entry) loaded at
+  **f32**, completed 30/30 denoising steps, and then died on the VAE decode --
+  "Tried to allocate 1008.00 MiB ... 704.69 MiB is free", with 22.12 GiB of a
+  23.56 GiB card held by its own weights. The same architecture at half width
+  measures 7.49 GB.
+* ``Krea-2-Edit`` (GGUF Q4_K_M transformer + 4-bit text encoder, bf16) loaded
+  correctly, completed 8/8 steps, and died the same way on 190 MiB.
+
+Two things were wrong.
+
+1. The bf16 default added for an undeclared precision was unreachable: the
+   config layer always supplied ``image.precision = "f32"``, so there was no
+   such thing as "undeclared" by the time the loader looked. The default is now
+   ``"auto"`` and resolves from the card -- half on a card of 32 GB or less,
+   f32 above it, since a card big enough for both loses nothing by upcasting.
+
+2. Both failures were reported as ``Failed to load image model 'X'. Errors:
+   diffusers: CUDA out of memory`` -- because the generate call sat inside the
+   load's ``except Exception``, which also fell through to the sd.cpp backend.
+   The models had in fact loaded and run. Two separate readers concluded from
+   that wording that VRAM eviction was leaking; it was not, and the real causes
+   went unexamined for it.
+"""
+import ast
+
+import pytest
+
+from codai.api import images as images_mod
+from codai.config import ImageConfig
+
+
+SRC = open("codai/api/images.py").read()
+
+
+# --------------------------------------------------------------------------- #
+#  the default is a decision, not a constant
+# --------------------------------------------------------------------------- #
+
+def test_the_global_image_precision_default_is_auto_not_f32():
+    # Not cosmetic: while this said "f32" the loader's own "card states none"
+    # branch could never run, because the config layer filled the key in.
+    assert ImageConfig().precision == "auto"
+
+
+@pytest.mark.parametrize("vram_gb,expected", [
+    (8.0, "bf16"),      # RX 580 / small
+    (23.56, "bf16"),    # the 3090 this was measured on
+    (32.0, "bf16"),     # boundary: "32 or less" includes 32
+    (32.1, "f32"),      # just over
+    (48.0, "f32"),      # A6000
+    (80.0, "f32"),      # A100
+])
+def test_auto_precision_follows_the_card_size(monkeypatch, vram_gb, expected):
+    monkeypatch.setattr(images_mod, "_largest_visible_vram_gb", lambda: vram_gb)
+    precision, why = images_mod._auto_precision()
+    assert precision == expected
+    assert why, "the choice has to explain itself in the log"
+
+
+def test_an_unknown_card_is_treated_as_small():
+    # Guessing half on a big card wastes nothing; guessing f32 on a small one
+    # is the failure this whole module is about.
+    import unittest.mock as mock
+    with mock.patch.object(images_mod, "_largest_visible_vram_gb", lambda: 0.0):
+        assert images_mod._auto_precision()[0] == "bf16"
+
+
+def test_the_half_type_is_bf16_because_fp16_blacked_out_z_image():
+    import torch
+    import unittest.mock as mock
+    with mock.patch.object(images_mod, "_largest_visible_vram_gb", lambda: 24.0):
+        assert images_mod._auto_precision()[0] == "bf16"
+    # Same size, four orders of magnitude more range. See
+    # tests/test_black_image_and_precision.py for what fp16 cost.
+    assert torch.finfo(torch.bfloat16).bits == torch.finfo(torch.float16).bits
+    assert torch.finfo(torch.bfloat16).max > torch.finfo(torch.float16).max * 1000
+
+
+def test_largest_visible_vram_is_the_biggest_card_not_the_sum(monkeypatch):
+    # A pipeline loads onto ONE card unless explicitly sharded, so two 24 GB
+    # cards must not read as a 48 GB one and tip the policy over to f32.
+    monkeypatch.setattr(
+        "codai.models.gpu_query.visible_gpu_memory",
+        lambda: [{"total": 24e9}, {"total": 24e9}])
+    assert images_mod._largest_visible_vram_gb() == pytest.approx(24.0, abs=0.1)
+
+
+def test_an_explicit_precision_is_still_honoured():
+    # The policy changes the DEFAULT. An operator who wrote f32 gets f32.
+    body = SRC[SRC.index("_precision_cfg = str(_cfg('precision')"):][:400]
+    assert "else:" in body and "precision = _precision_cfg" in body
+
+
+# --------------------------------------------------------------------------- #
+#  a decode failure is not a load failure
+# --------------------------------------------------------------------------- #
+
+def test_generation_runs_outside_the_load_exception_handler():
+    # The structural fix. If the generate call is inside the try that catches
+    # load errors, a decode OOM is filed as a load error again.
+    tree = ast.parse(SRC)
+    fn = next(n for n in ast.walk(tree)
+              if isinstance(n, (ast.AsyncFunctionDef, ast.FunctionDef))
+              and n.name == "create_image_generation")
+
+    def generate_calls(node):
+        return [c for c in ast.walk(node)
+                if isinstance(c, ast.Call)
+                and getattr(c.func, "id", None) == "_generate_with_diffusers"]
+
+    for handler_owner in ast.walk(fn):
+        if not isinstance(handler_owner, ast.Try):
+            continue
+        names = {getattr(h.type, "id", None) for h in handler_owner.handlers}
+        if "ImportError" not in names:
+            continue        # not the load try
+        for call in generate_calls(handler_owner):
+            pytest.fail("_generate_with_diffusers is back inside the load try; "
+                        "a VAE-decode OOM will be reported as "
+                        "'Failed to load image model' again")
+
+
+def test_a_generation_failure_says_the_model_loaded():
+    assert "the model loaded and ran, so this is not a " in SRC
+    assert "load failure" in SRC
+
+
+def test_a_decode_oom_is_503_and_names_the_decode():
+    body = SRC[SRC.index("if _loaded_diffusers is not None:"):][:2000]
+    assert "503 if _oom else 500" in body
+    assert "VAE decode" in body
+
+
+@pytest.mark.parametrize("exc,expected", [
+    (RuntimeError("CUDA out of memory. Tried to allocate 1008.00 MiB"), True),
+    (RuntimeError("Out Of Memory"), True),
+    (ValueError("size must be a multiple of 8"), False),
+    (RuntimeError("CUDA error: device-side assert triggered"), False),
+])
+def test_is_cuda_oom_matches_the_wordings_this_actually_arrives_as(exc, expected):
+    assert images_mod._is_cuda_oom(exc) is expected
+
+
+def test_is_cuda_oom_matches_the_torch_type_too():
+    torch = pytest.importorskip("torch")
+    oom = getattr(torch, "OutOfMemoryError", None)
+    if oom is None:
+        pytest.skip("this torch has no OutOfMemoryError type")
+    assert images_mod._is_cuda_oom(oom("ran out")) is True
+
+
+# --------------------------------------------------------------------------- #
+#  tiling is what bounds a single large decode
+# --------------------------------------------------------------------------- #
+
+def test_vae_tiling_is_honoured_from_the_model_config():
+    body = SRC[SRC.index("# Enable memory optimizations"):][:2200]
+    assert "(model_config or {}).get('vae_tiling')" in body, \
+        "tiling has to be readable from the model's own config"
+    assert "enable_tiling" in body and "enable_vae_tiling" in body, \
+        "both the vae-level and pipeline-level diffusers APIs"
+
+
+def test_slicing_alone_is_not_enough_and_the_code_says_why():
+    body = SRC[SRC.index("# Enable memory optimizations"):][:2200]
+    # Slicing splits a BATCH; it does nothing for n=1, which is the case that
+    # OOMed. Whoever next trims these comments should know that.
+    assert "n=1" in body

tests/test_image_size_and_embeddings.py

@@ -412,8 +412,10 @@ def test_a_rejected_request_is_not_mistaken_for_a_failed_backend():
     head = src.index("Step 4: Model not loaded")
     tail = src.index("Step 5: Both backends failed")
     section = src[head:tail]
-    # one pass-through per backend attempt, each above its catch-all
-    assert section.count("except HTTPException:") == 2
+    # One pass-through per backend attempt, each above its catch-all -- plus a
+    # third for the generate call, which now runs outside the load's handler so
+    # a VAE-decode OOM is not reported as "Failed to load image model".
+    assert section.count("except HTTPException:") == 3
     for idx in (section.index("diffusers_error = str(e)\n                print(f\"diffusers error"),
                 section.index("sdcpp_error = str(e)\n            print(f\"sd.cpp error")):
         assert "except HTTPException:" in section[max(0, idx - 700):idx]

tests/test_prompt_prefix_and_ocr_upload_size.py

@@ -141,7 +141,7 @@ def test_the_prefix_reaches_the_encode_and_the_cache_key_together():
     prefixes would share one entry and the second would get the first's text."""
     src = open("codai/api/images.py").read()
     head = src.index("async def _generate_with_diffusers")
-    body = src[head:head + 9000]
+    body = src[head:head + 12000]   # the function grew; the two calls are still paired
     assert "prompt, neg_prompt = _effective_prompts(request, model_config)" in body
     assert "_embed_cache.get(prompt, neg_prompt, model_id)" in body
     assert "_embed_cache.get(request.prompt" not in body