• Stefy Lanza (nextime / spora )'s avatar
    video: stop the offload retry death-spiral; log swallowed load/gen errors; robust pipeline cache · e48b90e0
    Stefy Lanza (nextime / spora ) authored
    Three fixes for the Wan2.2-VACE-Fun OOM cascade:
    
    1. Offload retry death-spiral (the OOM driver). When from_pretrained OOMs
       MID-LOAD, `pipe` is never assigned, so the ~20 GB already placed on the GPU is
       pinned by the EXCEPTION'S TRACEBACK frames (their locals hold the partial
       model), not by `pipe`. _clear_mem() frees via `pipe` (None) so it reclaims
       nothing, and the next balanced step recomputes its GPU budget from a card still
       ~20 GB full (70%->17.5, 60%->3.4, 40%->2.3 GiB — all doomed). Drop
       `e.__traceback__` and run _clear_mem() OUTSIDE the except block (where
       sys.exc_info no longer pins the frames) so the stranded tensors are collectable
       before the next attempt. Applied to the balanced chain and the full-GPU and
       model-offload handlers.
    
    2. Observability. The load-failure and generation-failure handlers raised
       HTTPException(500) with the cause only in the HTTP detail — debug.log showed a
       bare "Response status: 500". Log the full traceback in all three handlers.
    
    3. Pipeline cache robustness. diffusers refuses save_pretrained on a CPU/seq
       offloaded pipeline, and a device_map/disk-offloaded one has meta tensors —
       either way the save half-writes a .building dir then a killed process leaves it
       orphaned (tens of GB). Add _unsavable_reason() to skip those cleanly (no junk),
       and sweep_stale() to delete orphaned *.building dirs. A full-GPU load stays
       savable and still caches the 4-bit pipeline.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    e48b90e0
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...