• Stefy Lanza (nextime / spora )'s avatar
    quantize what load_in_4bit was asked to quantize, and say why a load spills · bb1b78e3
    Stefy Lanza (nextime / spora ) authored
    component_quantization replaced the global flag instead of refining it, so
    naming ONE component switched load_in_4bit off for every other one. Measured
    2026-10-29 on LTX-2.5 (load_in_4bit=True, map naming transformer and
    text_encoder): ten of its twelve components loaded at full precision, among
    them connectors at 11.82 GB and prompt_enhancer at 9.51 GB -- 21.3 GB of a
    25.3 GB card before the quantized transformer arrived, which is why the
    resident attempt OOMed on exactly those components.
    
    The candidate list was the second half of it. Written for unet/SD-era
    pipelines, it matched only transformer*/unet/text_encoder*, so connectors and
    prompt_enhancer could not be reached by the global flag either. A modern video
    pipeline carries several multi-GB Linear-heavy parts under other names.
    
    Both builders are fixed together -- build_pipeline_quant_config and the
    incremental component cache -- because they had the same shape, and a cached
    build that quantizes differently from a direct load is its own bug. An
    explicit map entry still wins, including "none"/"full", which is how a
    component is opted out.
    
    Wan2.2-TI2V-5B shows the size of this: load_in_4bit=True with a map naming
    only text_encoder and vae, so its transformer was never quantized at all --
    a 5B model measuring 40.0 GB on a 25.3 GB card.
    
    Separately, the message printed when eviction cannot make room claimed
    "Remaining models are busy or VRAM is held elsewhere" without having
    established either, and it was usually false. Over one production day, 10 of
    12 of these were a model asking for more than the whole card (25.4 to 40.0 GB
    against 25.3 GB), four of them printing "freed 0.0 GB (now 25.3 GB free)" --
    the card was already empty. "Skipping eviction of ... still busy" never
    appeared once: eviction was never blocked by a busy model.
    
    That wording sent an investigation at the scheduler when the causes were a
    model larger than the GPU and components that were never quantized. It now
    names which of the three cases applies, and for the common one points at the
    two things that actually help.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    bb1b78e3
test_quantization_map_refines.py 3.54 KB