• Stefy Lanza (nextime / spora )'s avatar
    stop a VRAM measurement outliving the quantization it was taken under · 8303b38e
    Stefy Lanza (nextime / spora ) authored
    A measurement describes the model as it was BUILT. Change
    component_quantization and the stored number describes something that no
    longer exists -- and because force_vram_update makes the measurement win over
    the configured size, the stale figure goes on deciding placement.
    
    Wan2.2-TI2V-5B is the worked example, and the numbers are worth keeping. Its
    components are 18.63 GB transformer (~5B params at ~3.7 bytes each, i.e.
    fp32), 10.58 GB text encoder and 2.63 GB VAE. Its map named text_encoder and
    vae; vae is conv-only so bitsandbytes skips it, which left exactly ONE
    component quantized and an fp32 18.63 GB transformer loading whole. Hence
    measured_vram_gb = 40.0 for a model whose name says 5B.
    
    With the transformer actually quantized the same pipeline is roughly 10 GB,
    but 40.0 would have kept forcing CPU offload on a 25.3 GB card until some
    unlucky load happened to re-measure it.
    
    The measurement is now stamped with pipeline_cache._signature -- which already
    hashes model id, precision, load_in_4bit/8bit and component_quantization, the
    exact set that decides how big the built weights are -- and is ignored when
    that signature no longer matches. Offload strategy and used_vram_gb are not in
    it, so a runtime placement change does not throw away a good measurement.
    
    An entry with no recorded signature is still trusted, so upgrading does not
    invalidate every existing measurement at once.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    8303b38e
test_measurement_invalidated_by_quant_change.py 3.76 KB