• Stefy Lanza (nextime / spora )'s avatar
    pipeline-cache: reject corrupt-JSON caches (stop the empty-tokenizer death spiral) · 31793b05
    Stefy Lanza (nextime / spora ) authored
    A cache dir's completion marker only proves the save FINISHED, not that every
    file landed intact. A transient truncated write — repeatedly a 0-byte
    tokenizer/tokenizer_config.json — slipped into an otherwise "valid" cache and
    then threw JSONDecodeError on EVERY subsequent video load, knocking the pipeline
    off its fast path into the offload fallback ladder (balanced→sequential→disk),
    which churned for hours, leaked ~22 GB VRAM, and died on a meta-tensor error.
    
    Add _first_bad_json(dir): walk the (small) JSONs and flag the first that is
    empty or unparseable — the big weights are .safetensors and aren't scanned, so
    it's cheap. Wire it in on BOTH sides:
     - load: valid() and component_valid() now invalidate + return False when any
       cached JSON is corrupt, so a poisoned cache becomes a clean rebuild instead
       of a death spiral.
     - save: save()/save_component() verify the temp dir before committing, and
       mark_monolithic_complete() refuses to finalize a dir with a corrupt JSON —
       so a truncated write is never cached in the first place.
    Added invalidate_path(p) helper.
    
    Verified: _first_bad_json flags 0-byte and garbage JSONs, passes clean ones.
    The already-poisoned Wan2.2-VACE cache was deleted out-of-band to unblock.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    31793b05
pipeline_cache.py 16.1 KB