longcat: cache prompt embeddings, so the encoder need not load (v0.2.81)

The UMT5-XXL text encoder is ~11 GB and turns a prompt into a few MB of
embeddings that are a pure function of the prompt and the encoder. v0.2.76 made
that run once per request instead of once per segment; this makes it once per
prompt, ever.

The saved GPU pass is not the point — it is seconds. The point is that a cached
prompt needs no encoder AT ALL, so with the cache on the encoder is built
lazily: ~11 GB of host RAM and about a minute of load that a repeated prompt
never spends. That is the answer to "can we keep everything in VRAM": not by
quantising the encoder (v0.2.79 measured int8 resident at 24.0 GB on a 24 GB
card, and NF4 at 22.1 GB with less headroom than a 29-frame clip already uses),
but by not having an encoder resident at all.

Off by default — embeddings are derived data, but they are also several MB each
under the user's cache directory, so growing that is opted into:

    "prompt_cache": "off",          # off | disk
    "prompt_cache_dir": "",         # blank = <coderai home>/prompt_cache/longcat
    "prompt_cache_max_gb": 0,       # 0 = 2 GB, least-recently-used evicted

per model or server-wide, validated at startup rather than after a seven-minute
load, settable from the models page, and in api_model_configure's whitelist so a
save from that page does not silently drop it.

An entry served to the wrong pipeline would be a silent quality change, which is
worse than a miss, so the key covers the checkpoint, the dtype, the encoder's
quantisation and the call's own arguments — an NF4 encoder cannot be served a
bf16 entry. `device` is deliberately excluded: it moves the embeddings without
changing them, and including it would split one entry into one per device.

Every failure is a miss, never an error. Entries are written to a .tmp and
renamed, read with weights_only=True (a cache directory is not a trusted source
for a pickle), deleted when unreadable, and a cache that cannot be written warns
once and generation carries on without it.

_LazyEncoder is deliberately not a transparent proxy: upstream guards every use
with `if self.text_encoder is not None`, so a proxy that looked like a module
would be moved to the card by pipe.to() and offloaded by _place_pipeline,
building the very weights it exists to avoid. pipe.text_encoder stays None and
the holder sits beside it in _state, which is why both paths already do the
right thing.

Found while wiring it: a lazily realised encoder can be quantised, and the
offload swap would have shuttled bitsandbytes weights to the host. The swap
guard now checks the model, not just the offload mode.

/health reports the cache and whether the encoder was ever built — laziness is
invisible otherwise — and GET /prompt-cache returns its stats.

No new dependencies. Unverified on the card: the round trip, eviction, corrupt
entries and the key are covered by 54 tests against real tensors, but no
generation has run through it yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0185zEBiy3KWB37xNGSrxL7w

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.2.80"
+__version__ = "0.2.81"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -2945,7 +2945,10 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                 
                     # LongCat component quantisation, block-sparse attention, the base
                     # checkpoint an avatar family borrows from, and the CP tile.
-                    "text_encoder_quant", "bsa", "base_model", "cp_split_hw",):
+                    "text_encoder_quant", "bsa", "base_model", "cp_split_hw",
+                    # The prompt-embedding cache: an encode outlives the request that
+                    # paid for it, and the text encoder is then built only on a miss.
+                    "prompt_cache", "prompt_cache_dir", "prompt_cache_max_gb",):
         if key in data:
             entry[key] = data[key]
     # Work-parallel fan-out over engines and nodes (codai/cluster/fanout.py).

codai/admin/templates/models.html

@@ -719,6 +719,15 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
               <option value="auto">auto — on when triton is available</option>
             </select></div>
         </div>
+        <div style="display:grid;grid-template-columns:1fr 1fr 1fr;gap:.6rem;margin-top:.5rem">
+          <div class="form-row" style="margin:0"><label class="form-label">Prompt cache</label>
+            <select id="cfg-longcat-prompt-cache" class="form-input">
+              <option value="">off — encode every request</option>
+              <option value="disk">disk — encode once ever, encoder built only on a miss</option>
+            </select></div>
+          <div class="form-row" style="margin:0"><label class="form-label">Prompt cache dir <span class="muted">(blank = ~/.coderai)</span></label><input type="text" id="cfg-longcat-prompt-cache-dir" class="form-input" placeholder="~/.coderai/prompt_cache/longcat"></div>
+          <div class="form-row" style="margin:0"><label class="form-label">Prompt cache cap <span class="muted">(GB)</span></label><input type="number" id="cfg-longcat-prompt-cache-max" class="form-input" min="0" step="0.5" placeholder="2"></div>
+        </div>
         <div style="display:grid;grid-template-columns:1fr 1fr 1fr;gap:.6rem;margin-top:.5rem">
           <div class="form-row" style="margin:0"><label class="form-label">Segments <span class="muted">(blank = 1)</span></label><input type="number" id="cfg-longcat-segments" class="form-input" min="1" placeholder="1"></div>
           <div class="form-row" style="margin:0"><label class="form-label">Conditioning frames</label><input type="number" id="cfg-longcat-cond-frames" class="form-input" min="1" placeholder="13"></div>
@@ -4586,6 +4595,9 @@ function _populateCluster(m, s){
   set('cfg-longcat-offload', s.offload_strategy || '');
   set('cfg-longcat-te-quant', s.text_encoder_quant || '');
   set('cfg-longcat-bsa', s.bsa || '');
+  set('cfg-longcat-prompt-cache', s.prompt_cache || '');
+  set('cfg-longcat-prompt-cache-dir', s.prompt_cache_dir || '');
+  set('cfg-longcat-prompt-cache-max', s.prompt_cache_max_gb || '');
   set('cfg-longcat-base-model', s.base_model || '');
   set('cfg-longcat-cp-split', s.cp_split_hw || '');
   set('cfg-longcat-segments', s.num_segments || '');
@@ -4633,6 +4645,9 @@ function _collectCluster(){
     out.offload_strategy = val('cfg-longcat-offload') || '';
     out.text_encoder_quant = val('cfg-longcat-te-quant') || '';
     out.bsa = val('cfg-longcat-bsa') || '';
+    out.prompt_cache = val('cfg-longcat-prompt-cache') || '';
+    out.prompt_cache_dir = val('cfg-longcat-prompt-cache-dir') || '';
+    out.prompt_cache_max_gb = num('cfg-longcat-prompt-cache-max');
     out.base_model = val('cfg-longcat-base-model') || '';
     out.cp_split_hw = val('cfg-longcat-cp-split') || '';
     out.num_segments = num('cfg-longcat-segments');

codai/api/longcat_worker.py

@@ -47,6 +47,11 @@ Per-model config keys (models.json entry, all optional)::
     "bsa": "off",                  # block-sparse attention: on | off | auto
     "base_model": "",              # avatar families borrow tokenizer/text_encoder/vae
     "text_encoder_quant": "none",   # none | int8 | nf4 | fp4 (keeps it resident)
+    "prompt_cache": "off",         # off | disk — cache prompt embeddings between
+                                   #   requests; with it on the ~11 GB text encoder is
+                                   #   built only when a prompt misses
+    "prompt_cache_dir": "",        # blank = <coderai home>/prompt_cache/longcat
+    "prompt_cache_max_gb": 0,      # 0 = the service default (2 GB), LRU-evicted
     "cp_split_hw": "",             # context-parallel tile, e.g. "1x2"
     "used_vram_gb": 0,             # what to reserve before starting; 0 = measured
     "gpu_device": 0
@@ -508,6 +513,28 @@ def ensure_service(model_path: str, config: dict = None,
                    or "").strip()
         if te_q:
             cmd += ["--text-encoder-quant", te_q]
+        # The prompt cache makes an encode survive the request that paid for it, which
+        # is also what lets the encoder be built lazily. Per model, then server-wide:
+        # a checkpoint whose prompts never repeat gains nothing but the disk.
+        pc = str(config.get("prompt_cache") or
+                 (getattr(sec, "prompt_cache", "") if sec is not None else "")
+                 or "").strip()
+        if pc:
+            cmd += ["--prompt-cache", pc]
+        pc_dir = str(config.get("prompt_cache_dir") or
+                     (getattr(sec, "prompt_cache_dir", "") if sec is not None else "")
+                     or "").strip()
+        if pc_dir:
+            cmd += ["--prompt-cache-dir", pc_dir]
+        pc_max = (config.get("prompt_cache_max_gb")
+                  or (getattr(sec, "prompt_cache_max_gb", 0) if sec is not None else 0))
+        try:
+            if float(pc_max or 0) > 0:
+                cmd += ["--prompt-cache-max-gb", str(float(pc_max))]
+        except (TypeError, ValueError):
+            # A bad value is the service's to reject with a message naming the key,
+            # rather than this silently dropping it.
+            cmd += ["--prompt-cache-max-gb", str(pc_max)]
         if sec is not None and str(getattr(sec, "extra_args", "") or "").strip():
             import shlex
             cmd += shlex.split(str(sec.extra_args))

tests/test_longcat_prompt_cache.py

+"""Caching prompt embeddings on disk, so the text encoder need not run — or load.
+
+The UMT5-XXL encoder is ~11 GB and runs ONCE per request to produce a few MB of
+embeddings that are a pure function of the prompt and the encoder. v0.2.76 made that
+once per request instead of once per segment. This makes it once per prompt, ever.
+
+The payoff is not the saved GPU pass, which is seconds. It is that a cached prompt
+needs no encoder at all, so with the cache on the encoder is built LAZILY: ~11 GB of
+host RAM and about a minute of load that a repeated prompt never spends. The whole
+feature is therefore only worth having if the key is exact — an entry served to the
+wrong pipeline is a silent quality change, which is worse than a miss.
+"""
+import importlib.util
+import pathlib
+import re
+
+import pytest
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+COMMON = ROOT / "tools" / "longcat_common.py"
+SERVICE = ROOT / "tools" / "longcat_service.py"
+WORKER = ROOT / "codai" / "api" / "longcat_worker.py"
+ROUTES = ROOT / "codai" / "admin" / "routes.py"
+TEMPLATE = ROOT / "codai" / "admin" / "templates" / "models.html"
+
+
+def _load(name, path):
+    spec = importlib.util.spec_from_file_location(name, path)
+    mod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(mod)
+    return mod
+
+
+@pytest.fixture
+def LC():
+    return _load("lc_pc", COMMON)
+
+
+@pytest.fixture
+def SVC():
+    return _load("lsvc_pc", SERVICE)
+
+
+# ------------------------------------------------------------------ the contract
+
+def test_it_is_off_unless_configured(LC):
+    """Embeddings are derived data, but they are also MBs under the user's cache dir."""
+    assert LC.PROMPT_CACHE_MODES[0] == "off"
+    assert LC.prompt_cache_problems("off") == []
+    assert LC.prompt_cache_problems("") == []
+
+
+def test_an_unknown_mode_is_refused_by_name(LC):
+    problems = LC.prompt_cache_problems("ram")
+    assert problems and "ram" in problems[0] and "prompt_cache" in problems[0]
+
+
+@pytest.mark.parametrize("bad", ["0", "-1", "nonsense"])
+def test_a_cap_that_cannot_hold_anything_is_refused(LC, bad):
+    assert LC.prompt_cache_problems("disk", bad)
+
+
+def test_a_blank_cap_means_the_default(LC):
+    assert LC.prompt_cache_problems("disk", "") == []
+    assert LC.prompt_cache_problems("disk", None) == []
+    assert LC.DEFAULT_PROMPT_CACHE_MAX_GB > 0
+
+
+def test_the_default_dir_is_under_the_coderai_home(LC):
+    assert LC.prompt_cache_dir().endswith("prompt_cache/longcat")
+    assert LC.prompt_cache_dir("~/elsewhere").endswith("elsewhere")
+
+
+# ------------------------------------------------------------------ the key
+
+def _key(LC, **over):
+    args = dict(checkpoint="/AI/hfcache/LongCat-Video", dtype="bfloat16",
+                text_encoder_quant="none", args=(), kwargs={"prompt": "a cat"})
+    args.update(over)
+    return LC.prompt_cache_key(**args)
+
+
+def test_the_same_call_keys_the_same_way(LC):
+    assert _key(LC) == _key(LC)
+
+
+def test_device_is_excluded_because_it_moves_the_answer_not_changes_it(LC):
+    """Including it would split one entry into one per device, for identical numbers."""
+    on_cuda = _key(LC, kwargs={"prompt": "a cat", "device": "cuda"})
+    on_cpu = _key(LC, kwargs={"prompt": "a cat", "device": "cpu"})
+    assert on_cuda == on_cpu == _key(LC)
+
+
+def test_a_different_prompt_is_a_different_entry(LC):
+    assert _key(LC) != _key(LC, kwargs={"prompt": "a dog"})
+
+
+def test_the_negative_prompt_counts_too(LC):
+    assert _key(LC) != _key(LC, kwargs={"prompt": "a cat", "negative_prompt": "blurry"})
+
+
+def test_quantisation_cannot_be_served_from_a_bf16_entry(LC):
+    """An NF4 encoder does not produce the bf16 encoder's numbers. Reusing one for the
+    other is a silent quality change — the thing a cache must never do."""
+    assert _key(LC) != _key(LC, text_encoder_quant="nf4")
+    assert _key(LC, text_encoder_quant="int8") != _key(LC, text_encoder_quant="nf4")
+
+
+def test_the_dtype_counts(LC):
+    assert _key(LC) != _key(LC, dtype="float16")
+
+
+def test_the_sequence_length_counts(LC):
+    assert _key(LC) != _key(LC, kwargs={"prompt": "a cat", "max_sequence_length": 256})
+
+
+def test_the_same_checkpoint_through_a_symlink_is_the_same_encoder(LC):
+    """hfcache is reached by different paths on different machines and in the pod."""
+    assert _key(LC, checkpoint="/other/mount/LongCat-Video") == _key(LC)
+
+
+def test_a_different_checkpoint_is_not(LC):
+    assert _key(LC, checkpoint="/AI/hfcache/LongCat-Video-Avatar-1.5") != _key(LC)
+
+
+def test_the_format_version_invalidates_an_old_cache(LC):
+    """A stored tuple whose shape changed must be ignored, not misread."""
+    assert isinstance(LC.PROMPT_CACHE_FORMAT, int)
+    assert "format" in __import__("inspect").getsource(LC.prompt_cache_key)
+
+
+def test_an_unhashable_argument_still_keys(LC):
+    """The in-memory cache gives up on these; the disk key must not, since a prompt
+    list is both unhashable and the normal case for a batch."""
+    assert _key(LC, kwargs={"prompt": ["a cat", "a dog"]}) \
+        != _key(LC, kwargs={"prompt": ["a dog", "a cat"]})
+
+
+# ------------------------------------------------------------------ the cache itself
+
+torch = pytest.importorskip("torch", reason="the cache stores tensors")
+
+
+@pytest.fixture
+def cache(SVC, tmp_path):
+    return SVC._PromptCache(str(tmp_path / "pc"), 1.0)
+
+
+def _embeds():
+    """The 4-tuple upstream's encode_prompt returns: embeds, mask, and the negatives."""
+    return (torch.randn(1, 8, 16, dtype=torch.bfloat16),
+            torch.ones(1, 8, dtype=torch.long), None, None)
+
+
+def test_a_cold_cache_is_a_miss_not_an_error(cache):
+    assert cache.get("nothing-here") is None
+    assert cache.stats["misses"] == 1
+
+
+def test_what_goes_in_comes_back_out(cache):
+    value = _embeds()
+    cache.put("k", value)
+    got = cache.get("k", device="cpu", dtype=torch.bfloat16)
+    assert got is not None
+    assert torch.equal(got[0], value[0])
+
+
+def test_the_attention_mask_is_not_cast_to_the_compute_dtype(cache):
+    """It is an integer mask. Casting it to bf16 with the embeddings would corrupt it."""
+    cache.put("k", _embeds())
+    got = cache.get("k", device="cpu", dtype=torch.bfloat16)
+    assert got[1].dtype == torch.long
+
+
+def test_absent_negatives_stay_absent(cache):
+    """do_classifier_free_guidance=False returns None for both; None is not a tensor."""
+    cache.put("k", _embeds())
+    got = cache.get("k", device="cpu", dtype=torch.bfloat16)
+    assert got[2] is None and got[3] is None
+
+
+def test_a_hit_is_counted_and_a_miss_is_not(cache):
+    cache.put("k", _embeds())
+    cache.get("k", device="cpu")
+    assert cache.stats["hits"] == 1 and cache.stats["misses"] == 0
+
+
+def test_an_unreadable_entry_is_a_miss_and_deletes_itself(cache, tmp_path):
+    """Truncated by a crash mid-write, or written by an incompatible torch. A cache
+    that cannot be read must not be able to break generation."""
+    cache.put("k", _embeds())
+    with open(cache._path("k"), "wb") as fh:
+        fh.write(b"not a tensor")
+    assert cache.get("k", device="cpu") is None
+    assert not pathlib.Path(cache._path("k")).exists()
+    assert cache.stats["errors"] == 1
+
+
+def test_a_half_written_entry_is_never_visible(cache):
+    """Written to a .tmp and renamed, so a reader sees all of it or none of it."""
+    src = __import__("inspect").getsource(type(cache).put)
+    assert "os.replace" in src and ".tmp" in src
+
+
+def test_it_will_not_accept_a_pickle_from_its_own_directory(cache):
+    """A cache dir is not a trusted source: weights_only, so an entry cannot execute."""
+    src = __import__("inspect").getsource(type(cache).get)
+    assert "weights_only=True" in src
+
+
+def test_it_stays_under_its_cap_by_dropping_the_oldest(SVC, tmp_path):
+    """A prompt list is the case that runs away, so the cap is on bytes, not count."""
+    tiny = SVC._PromptCache(str(tmp_path / "tiny"), 0.000001)   # ~1 KB
+    tiny.put("k", _embeds())
+    assert tiny.stats["evicted"] >= 1
+    assert tiny.get("k") is None
+
+
+def test_a_hit_counts_as_a_use_for_eviction(cache):
+    """LRU by mtime: a prompt used every day must not be evicted before a stale one."""
+    src = __import__("inspect").getsource(type(cache).get)
+    assert "os.utime" in src
+
+
+def test_it_reports_what_is_actually_on_disk(cache):
+    cache.put("k", _embeds())
+    report = cache.report()
+    assert report["entries"] == 1 and report["writes"] == 1
+    assert report["size_mb"] >= 0 and report["dir"].endswith("pc")
+
+
+def test_an_unwritable_cache_warns_once_and_keeps_generating(SVC, tmp_path):
+    """A cache we cannot write is not a reason to fail a request."""
+    c = SVC._PromptCache(str(tmp_path / "ro"), 1.0)
+    c.dir = "/proc/nonexistent-cannot-write"      # a real write failure
+    c.put("k", _embeds())
+    c.put("k2", _embeds())
+    assert c.stats["writes"] == 0 and c._warned is True
+
+
+# ------------------------------------------------------------------ laziness
+
+def test_the_encoder_is_not_built_until_something_misses(SVC):
+    built = []
+
+    class _Pipe:
+        text_encoder = None
+
+    lazy = SVC._LazyEncoder(lambda: built.append(1) or "encoder")
+    assert built == [], "constructing the holder must not build 11 GB of weights"
+    lazy.realise(_Pipe())
+    assert built == [1]
+
+
+def test_realising_twice_builds_once(SVC):
+    built = []
+
+    class _Pipe:
+        text_encoder = None
+
+    lazy = SVC._LazyEncoder(lambda: built.append(1) or "encoder")
+    pipe = _Pipe()
+    lazy.realise(pipe)
+    lazy.realise(pipe)
+    assert built == [1] and pipe.text_encoder == "encoder"
+
+
+def test_it_is_not_a_transparent_proxy(SVC):
+    """Upstream guards every use with `if self.text_encoder is not None`. A proxy that
+    looked like a module would be moved to the card by pipe.to() and offloaded by
+    _place_pipeline — building the very weights this exists to avoid. So the pipeline's
+    text_encoder stays None and the holder sits beside it."""
+    src = SERVICE.read_text(encoding="utf-8")
+    assert "_state[\"lazy_encoder\"] = _LazyEncoder(" in src
+    assert re.search(r"lazy_encoder.*\n.*text_encoder = None", src) or \
+        "            text_encoder = None" in src
+
+
+def test_laziness_follows_the_cache_and_nothing_else(SVC):
+    """Off by default means the encoder loads eagerly, exactly as before."""
+    src = SERVICE.read_text(encoding="utf-8")
+    assert 'if _state.get("prompt_cache") is not None:' in src
+
+
+def test_the_pipeline_is_dropped_with_its_deferred_encoder(SVC):
+    """The holder closes over the load arguments; the pipeline it would attach to is
+    gone. The DISK cache survives, which is the point of it being on disk."""
+    src = SERVICE.read_text(encoding="utf-8")
+    unload = src[src.index("def unload("):]
+    assert '_state["lazy_encoder"] = None' in unload[:unload.index("\ndef ")]
+
+
+def test_a_lazily_realised_quantised_encoder_is_never_swapped(SVC):
+    """bitsandbytes quantises as weights land on CUDA and is not built to shuttle them
+    back. A quantised encoder realised on a miss must not then be offloaded."""
+    src = SERVICE.read_text(encoding="utf-8")
+    wrap = src[src.index("def _wrap_encode_prompt("):]
+    wrap = wrap[:wrap.index("\n@contextlib")]
+    assert "_encoder_is_quantised(pipe)" in wrap
+
+
+def test_the_disk_cache_is_consulted_before_the_encoder_is_touched(SVC):
+    """Order matters: in-memory, then disk, then encode. Realising the encoder before
+    the disk lookup would spend the 11 GB this feature exists to save."""
+    wrap = SERVICE.read_text(encoding="utf-8")
+    wrap = wrap[wrap.index("def _wrap_encode_prompt("):]
+    assert wrap.index("cache.get(") < wrap.index("lazy.realise(")
+
+
+# ------------------------------------------------------------------ orchestration
+
+def test_the_service_validates_before_the_slow_load(SVC):
+    """A bad value should not surface after a multi-minute load."""
+    src = SERVICE.read_text(encoding="utf-8")
+    assert "LC.prompt_cache_problems(" in src
+    assert src.index("LC.prompt_cache_problems(") < src.index("load_pipeline(_state[\"model\"]")
+
+
+@pytest.mark.parametrize("flag", ["--prompt-cache", "--prompt-cache-dir",
+                                  "--prompt-cache-max-gb"])
+def test_the_service_takes_the_flag(SVC, flag):
+    assert flag in SERVICE.read_text(encoding="utf-8")
+
+
+@pytest.mark.parametrize("flag", ["--prompt-cache", "--prompt-cache-dir",
+                                  "--prompt-cache-max-gb"])
+def test_the_worker_passes_it_through(flag):
+    assert flag in WORKER.read_text(encoding="utf-8")
+
+
+@pytest.mark.parametrize("key", ["prompt_cache", "prompt_cache_dir",
+                                 "prompt_cache_max_gb"])
+def test_the_key_is_documented_where_the_others_are(key):
+    """The worker's header block is what anyone configuring this reads."""
+    head = WORKER.read_text(encoding="utf-8")
+    head = head[:head.index("import collections")]
+    assert key in head
+
+
+@pytest.mark.parametrize("key", ["prompt_cache", "prompt_cache_dir",
+                                 "prompt_cache_max_gb"])
+def test_the_key_survives_a_save_from_the_models_page(key):
+    """api_model_configure rebuilds the entry from scratch, so a key missing from its
+    whitelist is silently dropped on the next save — which is how H3's documented keys
+    came to never survive."""
+    src = ROUTES.read_text(encoding="utf-8")
+    block = src[src.index("\"text_encoder_quant\", \"bsa\""):]
+    assert f'"{key}"' in block[:block.index("):")]
+
+
+@pytest.mark.parametrize("element", ["cfg-longcat-prompt-cache",
+                                     "cfg-longcat-prompt-cache-dir",
+                                     "cfg-longcat-prompt-cache-max"])
+def test_the_models_page_can_set_it(element):
+    """Three places, or the control exists and does nothing: the markup, the loader
+    that fills it from the entry, and the serialiser that reads it back."""
+    html = TEMPLATE.read_text(encoding="utf-8")
+    assert html.count(f'id="{element}"') == 1, "markup"
+    assert f"set('{element}'" in html, "not populated from the saved entry"
+    assert f"'{element}')" in html.split("if (backend === 'longcat')")[1], "not saved"
+
+
+def test_the_service_reports_the_cache_and_whether_the_encoder_loaded(SVC):
+    """Laziness is invisible otherwise: /health must say if the 11 GB was spent."""
+    src = SERVICE.read_text(encoding="utf-8")
+    assert "text_encoder_loaded" in src
+    assert "/prompt-cache" in src

tools/longcat_common.py

@@ -374,6 +374,95 @@ def component_quant_bpe(value: str):
     return COMPONENT_QUANTS.get(str(value or "none").strip().lower())
 
 
+# ── prompt-embedding cache ────────────────────────────────────────────────────
+#
+# The UMT5-XXL text encoder is ~11 GB and runs ONCE per request, to turn a prompt into
+# a few MB of embeddings. v0.2.76 stopped re-running it per segment; caching the result
+# on DISK stops re-running it per request, and — because the entry is keyed on
+# everything that can change the answer — lets the encoder be loaded LAZILY: a request
+# whose prompt is already cached never builds it at all, which is ~11 GB of host RAM
+# and a minute of load time that nothing has to spend.
+#
+# Off by default. Embeddings are derived data, but they are also several MB each and
+# live under the user's cache directory, so growing that is something to opt into.
+PROMPT_CACHE_MODES = ("off", "disk")
+
+# Each entry is prompt_embeds + mask + the negative pair, 512x4096 at bf16 ≈ 4 MB per
+# side. A cap rather than a count, because a long prompt list is the case that runs away.
+DEFAULT_PROMPT_CACHE_MAX_GB = 2.0
+
+# Bumped when the stored tuple's shape or the key's inputs change, so a cache written by
+# an older build is ignored rather than misread.
+PROMPT_CACHE_FORMAT = 1
+
+
+def prompt_cache_problems(mode: str, max_gb=None) -> list:
+    """Reasons this prompt-cache configuration cannot be used, or []."""
+    problems = []
+    m = str(mode or "off").strip().lower()
+    if m not in PROMPT_CACHE_MODES:
+        problems.append(f"prompt_cache must be one of {sorted(PROMPT_CACHE_MODES)}; "
+                        f"got {mode!r}")
+    if max_gb is not None and str(max_gb).strip() != "":
+        try:
+            if float(max_gb) <= 0:
+                problems.append(f"prompt_cache_max_gb must be positive; got {max_gb!r}")
+        except (TypeError, ValueError):
+            problems.append(f"prompt_cache_max_gb must be a number; got {max_gb!r}")
+    return problems
+
+
+def prompt_cache_key(*, checkpoint: str, dtype: str, text_encoder_quant: str,
+                     args: tuple = (), kwargs: dict = None) -> str:
+    """A stable filename-safe key for one encode_prompt call.
+
+    Keyed on everything that changes the EMBEDDINGS and nothing that does not. The
+    checkpoint and the encoder's quantisation are in here because an NF4 encoder does
+    not produce the bf16 encoder's numbers -- reusing one for the other would be a
+    silent quality change, which is worse than a cache miss. ``device`` is deliberately
+    excluded: it moves the result, it does not alter it.
+    """
+    import hashlib
+    import json as _json
+
+    def _plain(v):
+        """Reduce an argument to something JSON can hold, deterministically."""
+        if isinstance(v, (str, int, float, bool)) or v is None:
+            return v
+        if isinstance(v, (list, tuple)):
+            return [_plain(x) for x in v]
+        if isinstance(v, set):
+            return sorted(_plain(x) for x in v)
+        if isinstance(v, dict):
+            return {str(k): _plain(v[k]) for k in sorted(v, key=str)}
+        # A torch.device, a torch.dtype, anything else: its repr is stable within a
+        # build, and `device` is dropped before we get here.
+        return repr(v)
+
+    kwargs = dict(kwargs or {})
+    kwargs.pop("device", None)
+    payload = {
+        "format": PROMPT_CACHE_FORMAT,
+        # basename, not the full path: the same checkpoint reached through a symlink or
+        # a differently-mounted hfcache is the same encoder.
+        "checkpoint": os.path.basename(os.path.normpath(str(checkpoint or ""))),
+        "dtype": str(dtype or ""),
+        "text_encoder_quant": str(text_encoder_quant or "none").strip().lower(),
+        "args": _plain(list(args)),
+        "kwargs": _plain(kwargs),
+    }
+    blob = _json.dumps(payload, sort_keys=True, separators=(",", ":"))
+    return hashlib.sha256(blob.encode("utf-8")).hexdigest()
+
+
+def prompt_cache_dir(configured: str = "") -> str:
+    """Where the prompt cache lives. Under the coderai cache root unless overridden."""
+    if str(configured or "").strip():
+        return os.path.expanduser(str(configured).strip())
+    root = os.environ.get("CODERAI_HOME") or os.path.join("~", ".coderai")
+    return os.path.expanduser(os.path.join(root, "prompt_cache", "longcat"))
+
+
 def dit_subdir(family: str = "") -> str:
     """Which subfolder holds the bf16 transformer for this family."""
     return AVATAR_DIT_SUBDIR if family else BASE_DIT_SUBDIR

tools/longcat_service.py

@@ -31,6 +31,7 @@ Endpoints:
     GET  /progress  → {"active", "stage", "segment", "segments", "step", "steps"}
     POST /generate  → {"mp4_b64", "num_frames", "fps", "stages", "segments"}
     POST /unload    → {"ok"}  (drops the pipeline, keeps the process)
+    GET  /prompt-cache → the disk prompt-embedding cache's stats, or {"enabled": false}
 """
 
 import argparse
@@ -63,6 +64,9 @@ _state = {
     "family": "", "variant": "bf16", "use_int8": False, "use_distill": False,
     "cp_split_hw": [1, 1], "audio_encoder": None, "base_model": "",
     "text_encoder_quant": "none",
+    # A _PromptCache when prompt_cache is configured, else None — which is also the
+    # switch that decides whether the text encoder is built lazily.
+    "prompt_cache": None, "lazy_encoder": None,
 }
 _progress = {"active": False, "stage": "", "segment": 0, "segments": 0,
              "step": 0, "steps": 0}
@@ -133,8 +137,21 @@ def load_pipeline(checkpoint_dir: str, source_dir: str, dtype: str = "bfloat16",
                 f"(~{11.0 * LC.component_quant_fraction(te_quant):.1f} GB instead of "
                 f"~11 GB at bf16 — the embedding table does not quantise) "
                 f"— resident, never offloaded")
-        text_encoder = UMT5EncoderModel.from_pretrained(
+        def _build_text_encoder():
+            return UMT5EncoderModel.from_pretrained(
                 shared, subfolder="text_encoder", **te_kw)
+
+        # With the prompt cache on, the encoder is not built until something misses.
+        # ~11 GB and about a minute, skipped entirely for a prompt already encoded —
+        # which is the whole return on having a cache on disk rather than in memory.
+        if _state.get("prompt_cache") is not None:
+            _state["lazy_encoder"] = _LazyEncoder(_build_text_encoder)
+            text_encoder = None
+            log("text encoder: deferred — the prompt cache is on, so it is built only "
+                "on a miss")
+        else:
+            _state["lazy_encoder"] = None
+            text_encoder = _build_text_encoder()
         vae = AutoencoderKLWan.from_pretrained(shared, subfolder="vae", torch_dtype=td)
         scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(
             root, subfolder="scheduler")
@@ -385,7 +402,204 @@ def _quant_config(kind: str, dtype):
                               bnb_4bit_use_double_quant=True)
 
 
-def _wrap_encode_prompt(pipe, swap: bool):
+class _PromptCache:
+    """A disk cache of prompt embeddings, so the text encoder need not run — or load.
+
+    An encode is ~11 GB of weights and a GPU pass to produce a few MB of embeddings
+    that are a pure function of the prompt and the encoder. v0.2.76 made that once per
+    request instead of once per segment; this makes it once per *prompt, ever*.
+
+    The reason it is worth a disk format rather than a bigger in-memory cache is
+    LAZINESS: with a hit, the encoder is never built, which is ~11 GB of host RAM and
+    about a minute of load time. See ``_LazyEncoder``.
+
+    Entries are keyed by :func:`LC.prompt_cache_key` — the checkpoint, the dtype, the
+    encoder's quantisation and the call's own arguments — so a cache cannot serve a
+    bf16 pipeline with NF4 numbers. Tensors are stored on the CPU and moved on read.
+
+    Every failure here is a MISS, never an error: a cache that cannot be read must not
+    be able to break generation. The only exception is a write failing for lack of
+    space, which is logged once so a silently useless cache does not look like a
+    working one.
+    """
+
+    def __init__(self, directory: str, max_gb: float):
+        self.dir = directory
+        self.max_bytes = int(float(max_gb) * (1024 ** 3))
+        self._lock = threading.Lock()
+        self.stats = {"hits": 0, "misses": 0, "writes": 0, "evicted": 0, "errors": 0}
+        self._warned = False
+        os.makedirs(self.dir, exist_ok=True)
+
+    def _path(self, key: str) -> str:
+        return os.path.join(self.dir, f"{key}.pt")
+
+    def get(self, key: str, device=None, dtype=None):
+        """The cached embeddings moved onto `device`, or None for a miss."""
+        import torch
+
+        path = self._path(key)
+        try:
+            # torch.load with weights_only=True: these files are plain tensors, and a
+            # cache directory is not a place to accept a pickle from.
+            blob = torch.load(path, map_location="cpu", weights_only=True)
+        except FileNotFoundError:
+            with self._lock:
+                self.stats["misses"] += 1
+            return None
+        except Exception as exc:
+            # Truncated by a crash mid-write, or written by an incompatible torch.
+            log(f"prompt cache: ignoring unreadable entry ({type(exc).__name__}: {exc})")
+            with self._lock:
+                self.stats["errors"] += 1
+                self.stats["misses"] += 1
+            try:
+                os.remove(path)
+            except OSError:
+                pass
+            return None
+
+        try:
+            out = tuple(None if t is None
+                        else t.to(device=device, dtype=dtype) if dtype is not None
+                        and t.is_floating_point() else t.to(device=device)
+                        for t in blob)
+        except Exception as exc:
+            log(f"prompt cache: entry could not be placed ({type(exc).__name__}: {exc})")
+            with self._lock:
+                self.stats["errors"] += 1
+                self.stats["misses"] += 1
+            return None
+
+        with self._lock:
+            self.stats["hits"] += 1
+        # Touch it: eviction is LRU by mtime, and a hit is a use.
+        try:
+            os.utime(path, None)
+        except OSError:
+            pass
+        return out
+
+    def put(self, key: str, value) -> None:
+        """Store one encode's output. Best effort; a failure is not an error."""
+        import torch
+
+        if not isinstance(value, (tuple, list)):
+            return
+        try:
+            on_cpu = [None if t is None else t.detach().to("cpu") for t in value]
+        except Exception:
+            return      # not a tuple of tensors: this pipeline returns something else
+
+        path = self._path(key)
+        tmp = f"{path}.{os.getpid()}.tmp"
+        try:
+            # Write then rename, so a reader never sees a half-written entry and a
+            # crash leaves a .tmp rather than a corrupt cache hit.
+            with open(tmp, "wb") as fh:
+                torch.save(on_cpu, fh)
+                fh.flush()
+                os.fsync(fh.fileno())
+            os.replace(tmp, path)
+        except Exception as exc:
+            try:
+                os.remove(tmp)
+            except OSError:
+                pass
+            if not self._warned:
+                self._warned = True
+                log(f"prompt cache: cannot write to {self.dir} "
+                    f"({type(exc).__name__}: {exc}) — running without it")
+            with self._lock:
+                self.stats["errors"] += 1
+            return
+        with self._lock:
+            self.stats["writes"] += 1
+        self._evict()
+
+    def _evict(self) -> None:
+        """Drop the least recently used entries until the cache is under its cap."""
+        try:
+            entries = []
+            for name in os.listdir(self.dir):
+                if not name.endswith(".pt"):
+                    continue
+                full = os.path.join(self.dir, name)
+                try:
+                    st = os.stat(full)
+                except OSError:
+                    continue
+                entries.append((st.st_mtime, st.st_size, full))
+            total = sum(e[1] for e in entries)
+            if total <= self.max_bytes:
+                return
+            for _mtime, size, full in sorted(entries):
+                try:
+                    os.remove(full)
+                except OSError:
+                    continue
+                total -= size
+                with self._lock:
+                    self.stats["evicted"] += 1
+                if total <= self.max_bytes:
+                    break
+        except Exception as exc:
+            log(f"prompt cache: eviction failed ({type(exc).__name__}: {exc})")
+
+    def report(self) -> dict:
+        """Stats plus what is actually on disk, for /health and /prompt-cache."""
+        size = count = 0
+        try:
+            for name in os.listdir(self.dir):
+                if name.endswith(".pt"):
+                    try:
+                        size += os.path.getsize(os.path.join(self.dir, name))
+                        count += 1
+                    except OSError:
+                        pass
+        except OSError:
+            pass
+        with self._lock:
+            stats = dict(self.stats)
+        stats.update(dir=self.dir, entries=count,
+                     size_mb=round(size / (1024 ** 2), 1),
+                     max_gb=round(self.max_bytes / (1024 ** 3), 2))
+        return stats
+
+
+class _LazyEncoder:
+    """Stands in for the text encoder until something actually needs it.
+
+    The point of the prompt cache is that a cached prompt costs no encoder at all, and
+    an encoder that is built anyway costs ~11 GB of host RAM and about a minute whether
+    or not it is used. So with the cache on, the encoder is not built at load time —
+    this holder is, and the first cache MISS realises it.
+
+    It is deliberately NOT a transparent proxy. Upstream guards every use with
+    ``if self.text_encoder is not None``, so a proxy that looked like a module would be
+    moved to the card by ``pipe.to()`` and offloaded by ``_place_pipeline`` — building
+    the very weights this exists to avoid. Instead ``pipe.text_encoder`` stays None and
+    this sits beside it in _state, which is why both of those paths already do the
+    right thing: they skip a None encoder.
+    """
+
+    def __init__(self, build):
+        self._build = build
+        self._lock = threading.Lock()
+        self.realised = False
+
+    def realise(self, pipe):
+        """Build the encoder and attach it to the pipeline. Idempotent."""
+        with self._lock:
+            if pipe.text_encoder is not None:
+                return pipe.text_encoder
+            log("prompt cache miss: building the text encoder now (deferred at load)")
+            pipe.text_encoder = self._build()
+            self.realised = True
+            return pipe.text_encoder
+
+
+def _wrap_encode_prompt(pipe, swap: bool, cache=None, lazy=None):
     """Encode a prompt at most once per request, and (when offloading) swap for it.
 
     _run_segments calls generate_t2v once PER SEGMENT, and every one of those calls
@@ -397,12 +611,16 @@ def _wrap_encode_prompt(pipe, swap: bool):
     The embeddings are a few MB, so they are cached on the arguments that produce them.
     Segment 2 onward then never touches the encoder at all, which means the card holds
     nothing but the DiT and VAE for the whole of a long generation.
+
+    With a `cache` (``prompt_cache: disk``) that memory lives across requests and across
+    restarts too, and `lazy` is the reason it is worth having: on a hit the encoder is
+    never built at all. The order is in-memory, then disk, then encode — cheapest first.
     """
     import gc
     import torch
 
     inner = pipe.encode_prompt      # the bound method, before it is shadowed below
-    cache = collections.OrderedDict()
+    mem = collections.OrderedDict()
 
     def _move_dit(device):
         """The DiT and the LoRA networks hanging off it, the way upstream's to() does."""
@@ -423,13 +641,44 @@ def _wrap_encode_prompt(pipe, swap: bool):
             key = (tuple(_hashable(a) for a in args),
                    tuple(sorted((k, _hashable(v)) for k, v in kwargs.items())))
             hash(key)
-            if key in cache:
-                cache.move_to_end(key)
-                return cache[key]
+            if key in mem:
+                mem.move_to_end(key)
+                return mem[key]
         except TypeError:
             key = None          # an unhashable argument: encode, do not cache
 
-        if swap:
+        # The disk cache next. Keyed by the shared helper rather than by the in-memory
+        # key, because that one includes `device` — which moves the embeddings without
+        # changing them, and would split one entry into one per device.
+        disk_key = None
+        if cache is not None:
+            try:
+                disk_key = LC.prompt_cache_key(
+                    checkpoint=_state.get("model") or "",
+                    dtype=_state.get("dtype") or "",
+                    text_encoder_quant=_state.get("text_encoder_quant") or "none",
+                    args=args, kwargs=kwargs)
+            except Exception as exc:
+                log(f"prompt cache: cannot key this call ({type(exc).__name__}: {exc})")
+                disk_key = None
+        if disk_key is not None:
+            hit = cache.get(disk_key, device=kwargs.get("device"),
+                            dtype=kwargs.get("dtype"))
+            if hit is not None:
+                log("prompt cache hit — the text encoder is not needed for this request")
+                if key is not None:
+                    mem[key] = hit
+                return hit
+
+        # A miss. If the encoder was deferred, this is the moment it has to exist.
+        if lazy is not None:
+            lazy.realise(pipe)
+
+        # Not just `swap`: a quantised encoder is resident by construction, and a
+        # lazily realised one can be quantised — bitsandbytes weights must not be
+        # shuttled to the host, so they never take turns even under offload.
+        if swap and pipe.text_encoder is not None \
+                and not _encoder_is_quantised(pipe):
             # They take TURNS. The INT8 DiT is 13.6 GB resident and the encoder is
             # ~11 GB: bringing the encoder over while the DiT stays is 24.8 GB on a
             # 24 GB card, which is the OOM that model offload exists to prevent.
@@ -437,10 +686,13 @@ def _wrap_encode_prompt(pipe, swap: bool):
             gc.collect()
             torch.cuda.empty_cache()
             pipe.text_encoder = pipe.text_encoder.to("cuda", non_blocking=True)
+            moved = True
+        else:
+            moved = False
         try:
             out = inner(*args, **kwargs)
         finally:
-            if swap:
+            if moved:
                 # finally, not just on success: leaving 11 GB stranded after a failed
                 # encode would make the retry the thing that OOMs, and a DiT left on
                 # the host would run every later segment at PCIe speed.
@@ -449,16 +701,16 @@ def _wrap_encode_prompt(pipe, swap: bool):
                 torch.cuda.empty_cache()
                 _move_dit("cuda")
         if key is not None:
-            cache[key] = out
-            while len(cache) > 4:       # a few MB each; bounded for a long-lived service
-                cache.popitem(last=False)
+            mem[key] = out
+            while len(mem) > 4:         # a few MB each; bounded for a long-lived service
+                mem.popitem(last=False)
+        if disk_key is not None:
+            cache.put(disk_key, out)
         return out
 
     # Set on the INSTANCE: all four generate_* methods call self.encode_prompt.
     pipe.encode_prompt = encode_prompt
     return pipe
-
-
 @contextlib.contextmanager
 def _bsa_for(pipe, ctx):
     """Run this pass with block-sparse attention only if the geometry allows it.
@@ -530,13 +782,15 @@ def _place_pipeline(pipe, offload):
         if pipe.vae is not None:
             pipe.vae = pipe.vae.to("cuda", non_blocking=True)
         pipe.device = "cuda"
-        _wrap_encode_prompt(pipe, swap=False)
+        _wrap_encode_prompt(pipe, swap=False, cache=_state.get("prompt_cache"),
+                            lazy=_state.get("lazy_encoder"))
         return pipe
 
     if offload not in ("model", "sequential"):
         pipe.to("cuda")
         # Still worth caching: 12 segments would otherwise run 12 identical encodes.
-        _wrap_encode_prompt(pipe, swap=False)
+        _wrap_encode_prompt(pipe, swap=False, cache=_state.get("prompt_cache"),
+                            lazy=_state.get("lazy_encoder"))
         log("the whole pipeline is resident on the card")
         return pipe
     if offload == "sequential":
@@ -558,7 +812,8 @@ def _place_pipeline(pipe, offload):
     if encoder is not None:
         pipe.text_encoder = encoder.to("cpu")
 
-    _wrap_encode_prompt(pipe, swap=True)
+    _wrap_encode_prompt(pipe, swap=True, cache=_state.get("prompt_cache"),
+                        lazy=_state.get("lazy_encoder"))
     log("offload: the DiT and VAE are resident; the text encoder (~11 GB) and the DiT "
         "swap places for encode_prompt so only one of them is ever on the card")
     return pipe
@@ -681,6 +936,10 @@ def unload(reason: str = ""):
             return
         _state["pipe"] = None
         _state["audio_encoder"] = None
+        # It holds a closure over the load arguments, and the pipeline it would have
+        # attached to is gone. The next load builds a fresh one; the DISK cache is
+        # untouched, which is the point of it being on disk.
+        _state["lazy_encoder"] = None
         try:
             import torch
             import gc
@@ -1136,7 +1395,16 @@ def make_handler():
                 self._json({"ok": True, "loaded": _state["pipe"] is not None,
                             "model": _state["model"], "stages": list(LC.STAGES),
                             "family": _state.get("family") or "",
-                            "variant": _state.get("variant") or "bf16"})
+                            "variant": _state.get("variant") or "bf16",
+                            "prompt_cache": (_state["prompt_cache"].report()
+                                             if _state.get("prompt_cache") else None),
+                            "text_encoder_loaded": (
+                                _state["pipe"] is not None
+                                and getattr(_state["pipe"], "text_encoder", None)
+                                is not None)})
+            elif self.path.startswith("/prompt-cache"):
+                cache = _state.get("prompt_cache")
+                self._json(cache.report() if cache else {"enabled": False})
             elif self.path.startswith("/progress"):
                 with _lock:
                     self._json(dict(_progress))
@@ -1251,6 +1519,15 @@ def main(argv=None):
     ap.add_argument("--text-encoder-quant", default="none",
                     help="none | int8 | nf4 | fp4 — quantise the UMT5-XXL text encoder "
                          "so it can stay resident instead of being offloaded")
+    ap.add_argument("--prompt-cache", default="off",
+                    help="off | disk — cache prompt embeddings on disk so a prompt is "
+                         "encoded once ever; with it on, the ~11 GB text encoder is "
+                         "built only when a prompt misses")
+    ap.add_argument("--prompt-cache-dir", default="",
+                    help="where to keep it; blank uses <coderai home>/prompt_cache/longcat")
+    ap.add_argument("--prompt-cache-max-gb", default="",
+                    help="size cap, least-recently-used evicted first "
+                         "(default 2 GB; entries are a few MB each)")
     ap.add_argument("--base-model", default="",
                     help="checkpoint holding tokenizer/text_encoder/vae for an avatar "
                          "family; blank uses the upstream base repo")
@@ -1281,6 +1558,22 @@ def main(argv=None):
     if _problems:
         raise SystemExit("; ".join(_problems))
     _state["text_encoder_quant"] = args.text_encoder_quant or "none"
+    _problems = LC.prompt_cache_problems(args.prompt_cache, args.prompt_cache_max_gb)
+    if _problems:
+        raise SystemExit("; ".join(_problems))
+    if str(args.prompt_cache or "off").strip().lower() == "disk":
+        _max_gb = (float(args.prompt_cache_max_gb)
+                   if str(args.prompt_cache_max_gb).strip()
+                   else LC.DEFAULT_PROMPT_CACHE_MAX_GB)
+        try:
+            _state["prompt_cache"] = _PromptCache(
+                LC.prompt_cache_dir(args.prompt_cache_dir), _max_gb)
+            log(f"prompt cache: {_state['prompt_cache'].dir} (cap {_max_gb:g} GB)")
+        except OSError as exc:
+            # A cache we cannot create is not a reason not to generate. Say so loudly
+            # once, then carry on with the encoder loaded the ordinary way.
+            log(f"WARNING: prompt cache unavailable ({exc}) — continuing without it")
+            _state["prompt_cache"] = None
     # Validate both before anything slow happens: a bad value should not surface after
     # a multi-minute load.
     _bsa_enabled()