loras: route Z-Image to _train_dit by name; train quantized base via full model

Env/fighter LoRA training for Z-Image misrouted to _train_sdxl (CLIPTokenizer
crash: "NoneType cannot be interpreted as an integer"). Cause: _resolve_base_model_path
returns the HF id verbatim (e.g. unsloth/Z-Image-Turbo-unsloth-bnb-4bit) when the
model has no local `path`, so the isdir(base_path/transformer) DiT check was False
and it fell through to the SDXL/SD15 trainer.

- Detect Z-Image by NAME (id contains z-image/zimage) in addition to the
  diffusers-config class name, so an HF-id base routes to _train_dit.
- _train_dit: when the base is a pre-quantized (bnb/nf4/4bit) build, train against
  the full Tongyi-MAI/Z-Image-Turbo instead (a 4-bit checkpoint isn't a clean
  full-precision LoRA base); the LoRA still applies to the quantized model at
  inference. Overridable via lora_train_base_model.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
parent e72e33eb
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API # Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here. # metadata and the admin web UI read from here.
__version__ = "0.1.17" __version__ = "0.1.18"
# Configure the CUDA caching allocator BEFORE torch is imported anywhere. # Configure the CUDA caching allocator BEFORE torch is imported anywhere.
# expandable_segments lets the allocator return freed pages to the driver even # expandable_segments lets the allocator return freed pages to the driver even
......
...@@ -1052,11 +1052,16 @@ def _train_lora_sync(req: LoraTrainRequest) -> dict: ...@@ -1052,11 +1052,16 @@ def _train_lora_sync(req: LoraTrainRequest) -> dict:
import os as _os, json as _json2 import os as _os, json as _json2
_has_unet = _os.path.isdir(_os.path.join(base_path, "unet")) _has_unet = _os.path.isdir(_os.path.join(base_path, "unet"))
_has_transformer = _os.path.isdir(_os.path.join(base_path, "transformer")) _has_transformer = _os.path.isdir(_os.path.join(base_path, "transformer"))
if _has_transformer and not _has_unet: # Z-Image is a DiT and must train via _train_dit (its LoRA then loads on the
# DiT architecture. Native training is implemented for Z-Image; route there # Z-Image pipeline). Detect it by NAME first: _resolve_base_model_path returns an
# so its LoRA loads on the Z-Image pipeline. Other DiTs (Flux/SD3) still need # HF id verbatim (e.g. 'unsloth/Z-Image-Turbo-...') for models with no local
# an SDXL `lora_train_base_model` override or an external trainer. # `path`, so the isdir(transformer/) check below is False and it would otherwise
_is_zimage = False # misroute to the SDXL/SD15 trainer (which crashes loading a CLIP tokenizer
# Z-Image doesn't have). Fall back to the diffusers-config class name for local
# dirs / other DiTs.
_bm_name = f"{base_path} {getattr(req, 'base_model', '') or ''}".lower()
_is_zimage = ("z-image" in _bm_name) or ("zimage" in _bm_name)
if not _is_zimage and _has_transformer and not _has_unet:
try: try:
for _p in (_os.path.join(base_path, "model_index.json"), for _p in (_os.path.join(base_path, "model_index.json"),
_os.path.join(base_path, "transformer", "config.json")): _os.path.join(base_path, "transformer", "config.json")):
...@@ -1066,9 +1071,10 @@ def _train_lora_sync(req: LoraTrainRequest) -> dict: ...@@ -1066,9 +1071,10 @@ def _train_lora_sync(req: LoraTrainRequest) -> dict:
break break
except Exception: except Exception:
pass pass
if _is_zimage: if _is_zimage:
return _train_dit(req, base_path, images, instance_prompt, return _train_dit(req, base_path, images, instance_prompt,
steps, rank, resolution, lr, seed, device) steps, rank, resolution, lr, seed, device)
if _has_transformer and not _has_unet:
raise ValueError( raise ValueError(
f"LoRA training base model '{base_path}' is a transformer/DiT " f"LoRA training base model '{base_path}' is a transformer/DiT "
f"architecture (has 'transformer/', no 'unet/'). Native DiT training is " f"architecture (has 'transformer/', no 'unet/'). Native DiT training is "
...@@ -1470,6 +1476,20 @@ def _train_dit(req, base_path, images, instance_prompt, ...@@ -1470,6 +1476,20 @@ def _train_dit(req, base_path, images, instance_prompt,
torch.manual_seed(seed) torch.manual_seed(seed)
compute_dtype = torch.bfloat16 compute_dtype = torch.bfloat16
# A pre-quantized (bnb/nf4/4-bit) build — e.g. the unsloth Z-Image used for fast
# inference — can't serve as a clean full-precision LoRA training base. Train
# against the full Z-Image instead; the resulting LoRA still applies to the
# quantized model at inference (identical architecture/keys). Overridable by a
# per-model `lora_train_base_model` pointing at a specific full-precision base.
_bl = str(base_path).lower()
if any(t in _bl for t in ("bnb", "nf4", "4bit", "int4")) and \
not getattr(req, "train_base_model", None):
_full = "Tongyi-MAI/Z-Image-Turbo"
print(f" [lora][zimage] base '{base_path}' is a quantized build; training "
f"against the full base '{_full}' (LoRA still applies to the quantized "
f"model at inference)", flush=True)
base_path = _full
def _calc_shift(seq_len, base=256, mx=4096, bs=0.5, ms=1.15): def _calc_shift(seq_len, base=256, mx=4096, bs=0.5, ms=1.15):
m = (ms - bs) / (mx - base) m = (ms - bs) / (mx - base)
return seq_len * m + (bs - m * base) return seq_len * m + (bs - m * base)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment