decide LongCat's placement from the card, not from whatever else is loaded

LongCat had two settings that look independent and are not. A quantised text
encoder MUST be resident -- bitsandbytes quantises as weights land on CUDA and
cannot shuttle them back, and _place_pipeline returns early for a quantised
encoder without consulting the offload mode at all. So "quantise the encoder"
and "spill when VRAM runs short" are one decision: asking for both pins 22.1 GB
of a 24 GB card (13.6 DiT + 8.0 nf4 encoder + 0.5 VAE) and leaves under 2 GB for
the activations of a 13.6B video model, which OOMs at the first long clip rather
than at load and so reads as a flaky model.

offload_strategy now accepts "auto", which takes that decision once, at load,
and logs the arithmetic:

  * resident -- DiT, VAE and a quantised encoder all on the card, no swapping;
  * spill    -- DiT and VAE resident, encoder on the host, the two swapping for
                encode_prompt so only one is ever on the card.

The decision is made against the card's CAPACITY -- total VRAM less a context
reserve -- and deliberately not against the free VRAM at that instant. Free VRAM
makes the placement depend on whichever model happened to be resident when the
request arrived: a sibling holding 8 GB would park LongCat on the host, and it
would stay there for every later request, because nothing re-examines the
condition. It is also self-fulfilling -- the worker RESERVES the footprint and
the manager evicts what is in the way, so "no room right now" is not a fact
about the card but about a decision nobody has taken yet. A model that could be
resident is never left on the CPU merely because something else got there first.

What is reserved is what the chosen placement OCCUPIES, not what residency would
need: on a 24 GB card that is 14.1 GB (DiT + VAE), because reserving the 22.1 GB
residency figure would evict siblings to free 8 GB this placement never touches.
Both numbers are reported, so the one the decision was made on stays visible.

The worker and the service run the SAME pure arithmetic over the same inputs, so
they cannot disagree, and the spawn moved inside the acquire loader: it used to
run before the reservation, so the service loaded its weights onto a card the
manager had not cleared yet and the eviction meant to make room happened
afterwards. Fine for a 13.6 GB DiT, wrong for any placement that wants the card.

Every load now logs what it actually cost against what the plan predicted, so a
constant that drifts shows up as a wrong prediction instead of as an OOM at
generation time.

And it is selectable: the model page's Offload list had no "auto" entry, which
mattered more than it looks, because the page sets that select from the saved
value -- a card reading "auto" would have displayed as the first option and the
next save would have silently rewritten it to "". The text-encoder labels now
say which levels need a card bigger than 24 GB, since that is what decides
whether the quantisation is honoured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.35"
+__version__ = "0.5.36"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/templates/models.html

@@ -721,18 +721,19 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
             </select></div>
           <div class="form-row" style="margin:0"><label class="form-label">Offload</label>
             <select id="cfg-longcat-offload" class="form-input">
+              <option value="auto">auto — measure the card, spill only if needed</option>
               <option value="">none — all on the card</option>
-              <option value="model">model — per-module CPU offload</option>
+              <option value="model">model — encoder on the host, swaps for encode</option>
               <option value="sequential">sequential — lowest VRAM, slowest</option>
             </select></div>
         </div>
         <div style="display:grid;grid-template-columns:1fr 1fr;gap:.6rem;margin-top:.5rem">
           <div class="form-row" style="margin:0"><label class="form-label">Text encoder</label>
             <select id="cfg-longcat-te-quant" class="form-input">
-              <option value="">bf16 — ~11 GB, needs offload</option>
-              <option value="int8">int8 — ~9.8 GB, too big to stay resident</option>
-              <option value="nf4">nf4 — ~8.0 GB, resident but tight</option>
-              <option value="fp4">fp4 — ~8.0 GB, resident but tight</option>
+              <option value="">bf16 — ~11 GB, lives on the host</option>
+              <option value="int8">int8 — ~9.8 GB, resident (needs a card over 24 GB)</option>
+              <option value="nf4">nf4 — ~8.0 GB, resident (needs a card over 24 GB)</option>
+              <option value="fp4">fp4 — ~8.0 GB, resident (needs a card over 24 GB)</option>
             </select></div>
           <div class="form-row" style="margin:0"><label class="form-label">Block-sparse attention</label>
             <select id="cfg-longcat-bsa" class="form-input">

codai/api/longcat_worker.py

@@ -43,7 +43,11 @@ Per-model config keys (models.json entry, all optional)::
     "longcat_source": "",          # repo checkout; blank = bundled/config
     "variant": "bf16",             # bf16 | fp8 | int8
     "quality": "fast",             # draft | fast | best
-    "offload_strategy": "",        # '' | model | sequential
+    "offload_strategy": "",        # '' | auto | model | sequential
+                                   #   '' = everything on the card
+                                   #   auto = measure free VRAM, then go resident
+                                   #          or keep the encoder on the host
+                                   #   model = always keep the encoder on the host
     "bsa": "off",                  # block-sparse attention: on | off | auto
     "bsa_pad_cond": "off",         # on | off — let i2v/continuation passes use BSA by
                                    #   rounding the conditioning latents up to the
@@ -463,10 +467,16 @@ def ensure_service(model_path: str, config: dict = None,
                         env[k.strip()] = v
 
         offload = str(config.get("offload_strategy") or "").strip().lower()
-        if offload in ("auto", "balanced", "disk", "group", "leaf"):
+        if offload in ("balanced", "disk", "group", "leaf"):
             # The upstream pipeline exposes only the diffusers CPU-offload hooks; the
             # manager's other strategies have no equivalent here.
             offload = "model"
+        # "auto" is passed THROUGH, not collapsed to "model". It used to be
+        # collapsed, which made the one setting that means "use the card and spill
+        # if it is not enough" a synonym for "always keep the encoder on the host"
+        # -- so a bigger card could never be used, and a 24 GB one could never be
+        # told apart from it. The service decides at load time from the free VRAM
+        # it can actually see, and logs the arithmetic (see LC.placement_plan).
         # Context parallelism: upstream splits the DiT's spatial dims across GPUs under
         # torchrun, so cp_size > 1 means N processes, not N threads. Only rank 0 binds
         # the port; the others take their jobs over a broadcast (see the service).
@@ -926,17 +936,47 @@ def acquire(model_name: str, config: dict = None) -> LongcatHandle:
     key = service_key(model_path, cfg)
 
     register_releaser()                          # both directions, from the first use
-    url = ensure_service(model_path, cfg)        # idempotent; re-spawns on a new port
+
+    needed = float(cfg.get("used_vram_gb") or 0.0)
+    _plan = None
+    if not needed and str(cfg.get("offload_strategy") or "").strip().lower() == "auto":
+        # Size the reservation from the placement the service is about to choose,
+        # so the manager evicts enough room for it. Both sides run the SAME pure
+        # arithmetic over the card's capacity, so they cannot disagree -- and the
+        # reservation has to be the resident footprint, not a default, or the
+        # service would decide "resident" against a card the manager never
+        # cleared and OOM at the first long clip.
+        try:
+            LCc = common()
+            _cap = LCc.usable_capacity_gb(multi_model_manager._total_vram_gb())
+            _plan = LCc.placement_plan(_cap, cfg.get("variant") or "int8",
+                                       cfg.get("text_encoder_quant") or "none")
+            needed = float(_plan.get("needed_gb") or 0.0)
+            print(f"[longcat] placement: {_plan['why']}", flush=True)
+        except Exception as exc:
+            print(f"[longcat] could not plan the placement ({exc}); "
+                  f"reserving the default", flush=True)
+            _plan = None
+    needed = needed or DEFAULT_VRAM_GB
+
+    # Spawned INSIDE the loader, so the reservation above evicts first. Spawning
+    # before it meant the service loaded its weights onto a card the manager had
+    # not cleared yet, and the eviction that was supposed to make room for it
+    # happened afterwards -- which is fine for a 13.6 GB DiT and wrong for any
+    # placement that needs the whole card.
+    _url = {}
 
     def _loader():
-        return LongcatHandle(key, url)
+        _url["v"] = ensure_service(model_path, cfg)   # idempotent; may re-spawn
+        return LongcatHandle(key, _url["v"])
 
-    needed = float(cfg.get("used_vram_gb") or 0.0) or DEFAULT_VRAM_GB
     # The manager's generic "register an out-of-band backend as an evictable,
     # VRAM-tracked model" entry point (STT is merely where it was first needed).
     handle = multi_model_manager.acquire_stt_backend(key, needed, True, _loader,
                                                      keep_resident=False)
-    handle.url = url                             # a re-spawned worker may have moved
+    # A handle that was already loaded never ran the loader, so make sure the
+    # service is up and the url is current either way.
+    handle.url = _url.get("v") or ensure_service(model_path, cfg)
     return handle
 
 

tools/longcat_common.py

@@ -597,3 +597,147 @@ def source_problems(src_dir: str) -> list:
     needed = ("pipeline_longcat_video.py", os.path.join("modules", "longcat_video_dit.py"))
     return [f"missing longcat_video/{n}" for n in needed
             if not os.path.isfile(os.path.join(pkg, n))]
+
+
+# ── placement planning ────────────────────────────────────────────────────────
+#
+# Measured component sizes on a 3090, used only to DECIDE a placement before any
+# weights are read. Every number here is checked against the real footprint after
+# the load, and the decision is logged with both, so a wrong constant shows up as
+# a wrong prediction rather than as an OOM nobody can explain.
+TEXT_ENCODER_BF16_GB = 11.0     # UMT5-XXL, 5.6B stored fp32 and loaded bf16
+VAE_GB = 0.5
+DIT_GB = {
+    "int8": 13.6,               # upstream's base_model_int8/, 1 byte per weight
+    "fp8": 13.6,
+    "gguf": 13.6,
+    "bf16": 27.0,
+}
+#: Room left for latents, activations and compute buffers. A 13.6B video DiT at
+#: 480x832x93 frames was measured using 13-20 GB beyond its weights on the LTX
+#: cards of the same shape, but LongCat runs its own segment loop with the KV
+#: cache offloadable, so this is the floor below which full residency is not
+#: worth attempting -- not a prediction of the peak.
+PLACEMENT_HEADROOM_GB = 3.0
+
+#: Held back from the card's total for the CUDA context, the driver and the
+#: allocator's own overhead, none of which a model gets to use.
+CONTEXT_RESERVE_GB = 1.0
+
+
+def usable_capacity_gb(total_vram_gb, reserve_gb: float = CONTEXT_RESERVE_GB) -> float:
+    """What a model may occupy on a card of ``total_vram_gb``."""
+    try:
+        total = float(total_vram_gb or 0.0)
+    except (TypeError, ValueError):
+        total = 0.0
+    return max(0.0, total - max(0.0, float(reserve_gb or 0.0)))
+
+
+def placement_plan(capacity_gb, variant: str = "int8",
+                   text_encoder_quant: str = "none",
+                   headroom_gb: float = PLACEMENT_HEADROOM_GB) -> dict:
+    """Decide where LongCat's components go, from the card's CAPACITY.
+
+    Returns ``{"resident": bool, "text_encoder_quant": str, "needed_gb": float,
+    "capacity_gb": float, "why": str}``.
+
+    ``capacity_gb`` is what the card holds with this model as THE ONLY ONE on
+    it -- total VRAM less the context reserve -- deliberately not the free VRAM
+    at this instant. Deciding on free VRAM makes the placement depend on
+    whichever model happens to be resident when the request arrives: a sibling
+    occupying 8 GB would push LongCat onto the host permanently, and it would
+    stay there for every later request too, because the condition that caused it
+    is never re-examined. Worse, it is self-fulfilling -- the whole point of a
+    reservation is that the manager evicts what is in the way, so "there is no
+    room right now" is not a fact about the card, it is a fact about a decision
+    nobody has taken yet.
+
+    So the question asked here is the stable one: can this model be resident on
+    this card at all? The answer sizes the reservation, the manager evicts
+    whatever it must to honour it, and a model that could be resident is never
+    left on the CPU merely because something else got there first.
+
+    The question is narrow, because only one component can move. The DiT has to
+    be resident -- it runs every denoising step of every segment -- and the VAE
+    is half a gigabyte. The UMT5-XXL text encoder is 11 GB and runs ONCE per
+    request, to turn the prompt into embeddings, so it is the only candidate for
+    the host.
+
+    Two placements follow:
+
+    * **resident** -- everything on the card, no swapping. Needs the DiT, the
+      VAE and the encoder (quantised, if asked) to fit together with headroom.
+      A quantised encoder MUST be resident: bitsandbytes quantises as weights
+      land on CUDA and cannot shuttle them back, so quantising it is also a
+      decision to keep it there.
+    * **offload** -- DiT and VAE resident, encoder on the host, the two swapping
+      places for encode_prompt so only one is ever on the card. Slower per
+      request by one swap, and it always fits.
+
+    Which is why "quantise the encoder" and "spill if it does not fit" are the
+    same decision rather than two: asking for both at once pins 22.1 GB of a
+    24 GB card (13.6 DiT + 8.0 nf4 encoder + 0.5 VAE) and leaves under 2 GB for
+    the activations of a 13.6B video model. So when the quantised encoder does
+    not fit, this plan drops the quantisation rather than the spill -- an
+    unquantised encoder on the host costs host RAM, while a resident one that
+    does not fit costs the request.
+    """
+    try:
+        free = float(capacity_gb or 0.0)
+    except (TypeError, ValueError):
+        free = 0.0
+    head = max(0.0, float(headroom_gb or 0.0))
+    dit = DIT_GB.get(str(variant or "int8").strip().lower(), DIT_GB["bf16"])
+    want = str(text_encoder_quant or "none").strip().lower()
+    frac = component_quant_fraction(want)
+    te_resident = TEXT_ENCODER_BF16_GB * frac if frac is not None else None
+
+    base = dit + VAE_GB
+    if te_resident is None:
+        # No quantisation asked for. Full residency would mean an 11 GB bf16
+        # encoder parked on the card doing nothing between prompts, so it is
+        # only worth it when there is room to spare.
+        resident_need = base + TEXT_ENCODER_BF16_GB
+        if resident_need + head <= free:
+            return {"resident": True, "text_encoder_quant": "none",
+                    "needed_gb": round(resident_need, 2),
+                    "resident_needed_gb": round(resident_need, 2),
+                    "capacity_gb": round(free, 2),
+                    "why": (f"{resident_need:.1f} GB fits the card's {free:.1f} GB "
+                            f"usable with {head:.1f} GB to spare — everything stays "
+                            f"on the card, nothing swaps")}
+        return {"resident": False, "text_encoder_quant": "none",
+                # Reserve the SPILL placement, not residency: the encoder is on
+                # the host and swaps with the DiT, so only one of them is ever on
+                # the card. Reserving the residency figure would evict siblings to
+                # free 8 GB that this placement never touches.
+                "needed_gb": round(base, 2),
+                "resident_needed_gb": round(resident_need, 2),
+                "capacity_gb": round(free, 2),
+                "why": (f"{resident_need:.1f} GB does not fit the card's "
+                        f"{free:.1f} GB usable (+{head:.1f} GB headroom) — the DiT "
+                        f"and VAE stay resident ({base:.1f} GB) and the "
+                        f"{TEXT_ENCODER_BF16_GB:.0f} GB encoder swaps in for "
+                        f"encode_prompt")}
+
+    resident_need = base + te_resident
+    if resident_need + head <= free:
+        return {"resident": True, "text_encoder_quant": want,
+                "needed_gb": round(resident_need, 2),
+                "resident_needed_gb": round(resident_need, 2),
+                "capacity_gb": round(free, 2),
+                "why": (f"{dit:.1f} GB DiT + {te_resident:.1f} GB {want} encoder "
+                        f"+ {VAE_GB:.1f} GB VAE = {resident_need:.1f} GB fits the "
+                        f"card's {free:.1f} GB usable with {head:.1f} GB to spare")}
+    # The quantised encoder is what does not fit, and it cannot be offloaded, so
+    # the quantisation is what gives way.
+    return {"resident": False, "text_encoder_quant": "none",
+            "needed_gb": round(base, 2),
+            "resident_needed_gb": round(resident_need, 2),
+            "capacity_gb": round(free, 2),
+            "why": (f"a resident {want} encoder would need {resident_need:.1f} GB "
+                    f"of the card's {free:.1f} GB usable (+{head:.1f} GB headroom), "
+                    f"and a quantised encoder cannot be offloaded — loading it "
+                    f"unquantised on the host instead ({base:.1f} GB on the card), "
+                    f"swapping with the DiT for encode_prompt")}

tools/longcat_service.py

@@ -179,6 +179,55 @@ def load_pipeline(checkpoint_dir: str, source_dir: str, dtype: str = "bfloat16",
         if family and shared != root:
             log(f"family '{family}': tokenizer/text_encoder/vae from {shared}")
 
+        # offload "auto": decide the placement from what the card HAS FREE right
+        # now, instead of from a setting written when the model was configured.
+        #
+        # The two settings that look independent are not. A quantised text
+        # encoder must be resident (bitsandbytes quantises as the weights land on
+        # CUDA and cannot shuttle them back), so asking for a quantised encoder
+        # AND for spilling when VRAM runs short asks for two incompatible things:
+        # on a 24 GB card it pins 22.1 GB (13.6 DiT + 8.0 nf4 encoder + 0.5 VAE)
+        # and leaves under 2 GB for the activations of a 13.6B video model. Which
+        # is an OOM at the first long clip rather than at load, so it looks like a
+        # flaky model.
+        #
+        # So one decision is taken here, with the numbers logged either way: go
+        # fully resident when it fits with headroom, else keep the DiT and VAE on
+        # the card and let the encoder live on the host and swap in for the
+        # encode. The prediction is logged next to the free VRAM it was made
+        # against, so a wrong constant shows up as a wrong prediction rather than
+        # as an OOM nobody can place.
+        if str(offload or "").strip().lower() == "auto":
+            # Decided against the card's CAPACITY, not the free VRAM at this
+            # instant. Free VRAM makes the placement depend on whichever model
+            # happened to be resident when the request arrived -- a sibling
+            # holding 8 GB would park LongCat on the host, and it would stay
+            # there for every later request because nothing re-examines the
+            # condition. It is also self-fulfilling: the worker RESERVES this
+            # footprint and the manager evicts whatever is in the way, so "no
+            # room right now" is not a fact about the card.
+            _total_gb, _free_gb = 0.0, 0.0
+            try:
+                if torch.cuda.is_available():
+                    _free_b, _total_b = torch.cuda.mem_get_info()
+                    _free_gb, _total_gb = _free_b / 1e9, _total_b / 1e9
+            except Exception:
+                pass
+            _cap_gb = LC.usable_capacity_gb(_total_gb)
+            _plan = LC.placement_plan(_cap_gb,
+                                      _state.get("variant") or "int8",
+                                      _state.get("text_encoder_quant") or "none")
+            log(f"placement (auto): card holds {_total_gb:.1f} GB "
+                f"({_cap_gb:.1f} GB usable), {_free_gb:.1f} GB free right now")
+            log(f"placement (auto): {_plan['why']}")
+            if (_state.get("text_encoder_quant") or "none") != _plan["text_encoder_quant"]:
+                log(f"placement (auto): text_encoder_quant "
+                    f"{_state.get('text_encoder_quant')!r} -> "
+                    f"{_plan['text_encoder_quant']!r}")
+            _state["text_encoder_quant"] = _plan["text_encoder_quant"]
+            _state["placement_plan"] = _plan
+            offload = "" if _plan["resident"] else "model"
+
         tokenizer = AutoTokenizer.from_pretrained(shared, subfolder="tokenizer")
         te_quant = str(_state.get("text_encoder_quant") or "none").strip().lower()
         te_kw = {"torch_dtype": td}
@@ -294,6 +343,27 @@ def load_pipeline(checkpoint_dir: str, source_dir: str, dtype: str = "bfloat16",
                 f"passes keep BSA, at the cost of attending over latent frames the "
                 f"pipeline still marks as noise")
         _place_pipeline(pipe, offload)
+        # What the placement actually cost, against what it was predicted to cost.
+        # The plan is arithmetic over measured constants, and a constant that drifts
+        # (a new variant, a different encoder) would otherwise be invisible until it
+        # produced an OOM at generation time. Logging both makes the prediction
+        # falsifiable on every load.
+        try:
+            if torch.cuda.is_available():
+                _alloc = torch.cuda.memory_allocated() / 1e9
+                _free_now = torch.cuda.mem_get_info()[0] / 1e9
+                _p = _state.get("placement_plan") or {}
+                if _p:
+                    _delta = _alloc - float(_p.get("needed_gb") or 0.0)
+                    log(f"placement: {_alloc:.2f} GB on the card after loading "
+                        f"(predicted {_p.get('needed_gb')} GB, "
+                        f"{_delta:+.2f} GB vs the plan), {_free_now:.2f} GB still "
+                        f"free for activations")
+                else:
+                    log(f"placement: {_alloc:.2f} GB on the card after loading, "
+                        f"{_free_now:.2f} GB still free for activations")
+        except Exception:
+            pass
         _state.update(pipe=pipe, model=root, source=src, dtype=dtype)
         log("ready")
         return pipe
@@ -1752,7 +1822,9 @@ def main(argv=None):
     ap.add_argument("--host", default="127.0.0.1")
     ap.add_argument("--port", type=int, required=True)
     ap.add_argument("--dtype", default="bfloat16")
-    ap.add_argument("--offload", default="", help="'' | model | sequential")
+    ap.add_argument("--offload", default="",
+                    help="'' (all on the card) | auto (measure, then decide) | "
+                         "model | sequential")
     ap.add_argument("--attention", default="xformers",
                     help="xformers | flash | flash3 (falls back to what is installed)")
     ap.add_argument("--text-encoder-quant", default="none",