-
Stefy Lanza (nextime / spora ) authored
Three fixes for the Wan2.2-VACE-Fun OOM cascade: 1. Offload retry death-spiral (the OOM driver). When from_pretrained OOMs MID-LOAD, `pipe` is never assigned, so the ~20 GB already placed on the GPU is pinned by the EXCEPTION'S TRACEBACK frames (their locals hold the partial model), not by `pipe`. _clear_mem() frees via `pipe` (None) so it reclaims nothing, and the next balanced step recomputes its GPU budget from a card still ~20 GB full (70%->17.5, 60%->3.4, 40%->2.3 GiB — all doomed). Drop `e.__traceback__` and run _clear_mem() OUTSIDE the except block (where sys.exc_info no longer pins the frames) so the stranded tensors are collectable before the next attempt. Applied to the balanced chain and the full-GPU and model-offload handlers. 2. Observability. The load-failure and generation-failure handlers raised HTTPException(500) with the cause only in the HTTP detail — debug.log showed a bare "Response status: 500". Log the full traceback in all three handlers. 3. Pipeline cache robustness. diffusers refuses save_pretrained on a CPU/seq offloaded pipeline, and a device_map/disk-offloaded one has meta tensors — either way the save half-writes a .building dir then a killed process leaves it orphaned (tens of GB). Add _unsavable_reason() to skip those cleanly (no junk), and sweep_stale() to delete orphaned *.building dirs. A full-GPU load stays savable and still caches the 4-bit pipeline. Co-Authored-By:
Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
e48b90e0