The UMT5-XXL text encoder is ~11 GB and turns a prompt into a few MB of
embeddings that are a pure function of the prompt and the encoder. v0.2.76 made
that run once per request instead of once per segment; this makes it once per
prompt, ever.

The saved GPU pass is not the point — it is seconds. The point is that a cached
prompt needs no encoder AT ALL, so with the cache on the encoder is built
lazily: ~11 GB of host RAM and about a minute of load that a repeated prompt
never spends. That is the answer to "can we keep everything in VRAM": not by
quantising the encoder (v0.2.79 measured int8 resident at 24.0 GB on a 24 GB
card, and NF4 at 22.1 GB with less headroom than a 29-frame clip already uses),
but by not having an encoder resident at all.

Off by default — embeddings are derived data, but they are also several MB each
under the user's cache directory, so growing that is opted into:

    "prompt_cache": "off",          # off | disk
    "prompt_cache_dir": "",         # blank = <coderai home>/prompt_cache/longcat
    "prompt_cache_max_gb": 0,       # 0 = 2 GB, least-recently-used evicted

per model or server-wide, validated at startup rather than after a seven-minute
load, settable from the models page, and in api_model_configure's whitelist so a
save from that page does not silently drop it.

An entry served to the wrong pipeline would be a silent quality change, which is
worse than a miss, so the key covers the checkpoint, the dtype, the encoder's
quantisation and the call's own arguments — an NF4 encoder cannot be served a
bf16 entry. `device` is deliberately excluded: it moves the embeddings without
changing them, and including it would split one entry into one per device.

Every failure is a miss, never an error. Entries are written to a .tmp and
renamed, read with weights_only=True (a cache directory is not a trusted source
for a pickle), deleted when unreadable, and a cache that cannot be written warns
once and generation carries on without it.

_LazyEncoder is deliberately not a transparent proxy: upstream guards every use
with `if self.text_encoder is not None`, so a proxy that looked like a module
would be moved to the card by pipe.to() and offloaded by _place_pipeline,
building the very weights it exists to avoid. pipe.text_encoder stays None and
the holder sits beside it in _state, which is why both paths already do the
right thing.

Found while wiring it: a lazily realised encoder can be quantised, and the
offload swap would have shuttled bitsandbytes weights to the host. The swap
guard now checks the model, not just the offload mode.

/health reports the cache and whether the encoder was ever built — laziness is
invisible otherwise — and GET /prompt-cache returns its stats.

No new dependencies. Unverified on the card: the round trip, eviction, corrupt
entries and the key are covered by 54 tests against real tensors, but no
generation has run through it yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185zEBiy3KWB37xNGSrxL7w