• Stefy Lanza (nextime / spora )'s avatar
    video: stop the offload retry death-spiral; log swallowed load/gen errors; robust pipeline cache · e48b90e0
    Stefy Lanza (nextime / spora ) authored
    Three fixes for the Wan2.2-VACE-Fun OOM cascade:
    
    1. Offload retry death-spiral (the OOM driver). When from_pretrained OOMs
       MID-LOAD, `pipe` is never assigned, so the ~20 GB already placed on the GPU is
       pinned by the EXCEPTION'S TRACEBACK frames (their locals hold the partial
       model), not by `pipe`. _clear_mem() frees via `pipe` (None) so it reclaims
       nothing, and the next balanced step recomputes its GPU budget from a card still
       ~20 GB full (70%->17.5, 60%->3.4, 40%->2.3 GiB — all doomed). Drop
       `e.__traceback__` and run _clear_mem() OUTSIDE the except block (where
       sys.exc_info no longer pins the frames) so the stranded tensors are collectable
       before the next attempt. Applied to the balanced chain and the full-GPU and
       model-offload handlers.
    
    2. Observability. The load-failure and generation-failure handlers raised
       HTTPException(500) with the cause only in the HTTP detail — debug.log showed a
       bare "Response status: 500". Log the full traceback in all three handlers.
    
    3. Pipeline cache robustness. diffusers refuses save_pretrained on a CPU/seq
       offloaded pipeline, and a device_map/disk-offloaded one has meta tensors —
       either way the save half-writes a .building dir then a killed process leaves it
       orphaned (tens of GB). Add _unsavable_reason() to skip those cleanly (no junk),
       and sweep_stale() to delete orphaned *.building dirs. A full-GPU load stays
       savable and still caches the 4-bit pipeline.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    e48b90e0
video.py 160 KB