Synchronous data parallel through torch.distributed with a hand-rolled
all-reduce of the adapter gradients (codai/cluster/ddp.py): the engine that
takes the job is rank 0, sends the same job with the images inlined to each
node in 'nodes' (or the model's lora_train_nodes), all ranks start from rank
0's init, average gradients every step, rank 0 saves. Peers write to scratch
and are cancelled with the coordinator. All five in-process trainers (SD1.5,
SDXL, Z-Image, flow DiT, Wan incl. 4-bit) take the hooks; two-process test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKm7Z56iKjHgNkHc6VhWKz