• Stefy Lanza (nextime / spora )'s avatar
    video: stop the offload retry death-spiral; log swallowed load/gen errors; robust pipeline cache · e48b90e0
    Stefy Lanza (nextime / spora ) authored
    Three fixes for the Wan2.2-VACE-Fun OOM cascade:
    
    1. Offload retry death-spiral (the OOM driver). When from_pretrained OOMs
       MID-LOAD, `pipe` is never assigned, so the ~20 GB already placed on the GPU is
       pinned by the EXCEPTION'S TRACEBACK frames (their locals hold the partial
       model), not by `pipe`. _clear_mem() frees via `pipe` (None) so it reclaims
       nothing, and the next balanced step recomputes its GPU budget from a card still
       ~20 GB full (70%->17.5, 60%->3.4, 40%->2.3 GiB — all doomed). Drop
       `e.__traceback__` and run _clear_mem() OUTSIDE the except block (where
       sys.exc_info no longer pins the frames) so the stranded tensors are collectable
       before the next attempt. Applied to the balanced chain and the full-GPU and
       model-offload handlers.
    
    2. Observability. The load-failure and generation-failure handlers raised
       HTTPException(500) with the cause only in the HTTP detail — debug.log showed a
       bare "Response status: 500". Log the full traceback in all three handlers.
    
    3. Pipeline cache robustness. diffusers refuses save_pretrained on a CPU/seq
       offloaded pipeline, and a device_map/disk-offloaded one has meta tensors —
       either way the save half-writes a .building dir then a killed process leaves it
       orphaned (tens of GB). Add _unsavable_reason() to skip those cleanly (no junk),
       and sweep_stale() to delete orphaned *.building dirs. A full-GPU load stays
       savable and still caches the 4-bit pipeline.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    e48b90e0
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...