bitsandbytes NF4 works — verified on the 3090, in the LongCat venv, loading the
real UMT5-XXL: 144 Linear4bit modules, is_loaded_in_4bit=True, finite embeddings
of the right shape. But it does not cost what v0.2.77 claimed.

bitsandbytes replaces Linear layers only. This encoder's 256k-token embedding
table and one linear per layer stay at the load dtype, leaving 6.13 GB that
cannot shrink. Measured on the card, against ~11.0 GB at bf16:

    nf4 + double quant   8.00 GB      (the bits-per-weight estimate said 2.8)
    int8                 9.76 GB      (it said 5.6)

So the estimate now comes from those measurements rather than the bit width,
because under-reserving by 3x is an OOM at load time. What the card must hold:

    bf16 encoder, offloaded   13.9 GB    9.7 GB headroom
    nf4  encoder, resident    22.1 GB    1.4 GB headroom
    int8 encoder, resident    24.0 GB    does not fit
    bf16 encoder, resident    25.2 GB    does not fit

Which inverts the recommendation this was built for: an NF4 encoder does fit a
24 GB card, but leaves less room for activations than a 29-frame clip already
used, so it would OOM on anything longer. Offload plus the per-request embedding
cache from v0.2.76 keeps 9.7 GB free and touches the encoder once, so it stays
the better configuration here. The setting remains, correctly sized, for cards
where it is the right trade.

The model page now states the measured figures too, since "nf4 — ~2.8 GB" would
have invited exactly the configuration that OOMs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185zEBiy3KWB37xNGSrxL7w