lora: a trained text adapter can reach a pod, and be picked per request (v0.3.36)

A QLoRA trained for a client sat on disk for ten days while a second service
rented its own A40 every morning, pushed the adapter over SSH and proxied to it.
The path that should have carried it was written but had never been run end to
end, and five things were in the way.

* `ensure_text_loras` DROPPED a PEFT directory — "publish it to HuggingFace" —
  which is the shape every trainer writes. It now travels as one deterministic
  tar (sorted names, fixed uid/gid/mtime, so the content id is stable) and is
  unpacked by the blob store, because vLLM loads an adapter from a directory:
  adapter_config.json carries the rank and target modules, and a bare
  .safetensors is not loadable.

* the adapter was re-read and re-hashed ON EVERY REQUEST. A Target is rebuilt per
  request, so `lora_entry` was set again each time; the bytes were never re-sent
  (the store is content-addressed) but 159 MB was read and hashed, and the pod
  asked, only to learn it already had it. Cached per (pod, path, size+mtime), so
  a retrained adapter at the same path is still sent again.

* `_lora_args` passed a CONTENT ID straight through, producing
  `--lora-modules name=sha256:…`, which vLLM cannot open. On a pod that is the
  only form the adapter has.

* an engine launched for a base model carried none of its adapters. vLLM
  registers them at launch and one vLLM serves one base, so the pod a shared pool
  rented for the base alone could never serve an adapter of it — the whole point
  of sharing the pool. Every model naming the same base now contributes its
  adapter under its own model id, which is the name requests ask for, so two
  versions can be A/B'd on one GPU with no reload.

* `--max-loras` was never set. At its default of 1, requests naming different
  adapters cannot share a batch and serialise instead: the cost lands on latency,
  not on an error.

Extracting a bundle is a write from network bytes, so member names are checked
for escapes before anything is unpacked, and tar's data filter is used as well.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.35"
+__version__ = "0.3.36"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/loras.py

@@ -268,25 +268,84 @@ def _lora_blob_dir() -> str:
     return d
 
 
-def _lora_blob_path(h: str) -> str:
-    """Local path for a blob given its hex sha256 (accepts an optional
-    'sha256:' prefix). Returns the path whether or not the file exists yet."""
+def _lora_blob_hex(h: str) -> str:
+    """The bare lowercase hex of a blob id (accepts an optional 'sha256:')."""
     h = (h or "").strip()
     if h.lower().startswith("sha256:"):
         h = h[7:]
-    h = h.lower()
+    return h.lower()
+
+
+def _lora_blob_path(h: str) -> str:
+    """Local path for a blob given its hex sha256 (accepts an optional
+    'sha256:' prefix). Returns the path whether or not the file exists yet.
+
+    A blob is normally one file. A text LoRA for vLLM is a DIRECTORY, so it is
+    stored unpacked under the same hash; when that directory is present it is
+    what the caller wants, because that is the path vLLM can load.
+    """
+    h = _lora_blob_hex(h)
+    unpacked = os.path.join(_lora_blob_dir(), h)
+    if os.path.isdir(unpacked):
+        return unpacked
     return os.path.join(_lora_blob_dir(), f"{h}.safetensors")
 
 
 def lora_blob_exists(h: str) -> bool:
-    return os.path.isfile(_lora_blob_path(h))
+    p = _lora_blob_path(h)
+    return os.path.isfile(p) or os.path.isdir(p)
+
+
+def _unpack_lora_bundle(data: bytes, dest: str) -> None:
+    """Extract an adapter bundle into `dest`, refusing anything that escapes it.
+
+    The bytes arrive over the network, so a member named ``../../etc/something``
+    would write outside the blob store. Names are checked here AND the extraction
+    uses tar's data filter, because either alone has been bypassed before.
+    """
+    import io
+    import tarfile
+    with tarfile.open(fileobj=io.BytesIO(data), mode="r:") as tar:
+        for info in tar.getmembers():
+            if not (info.isfile() or info.isdir()):
+                raise HTTPException(status_code=400,
+                                    detail=f"adapter bundle member {info.name!r} "
+                                           "is not a regular file")
+            target = os.path.realpath(os.path.join(dest, info.name))
+            if target != os.path.realpath(dest) and not target.startswith(
+                    os.path.realpath(dest) + os.sep):
+                raise HTTPException(status_code=400,
+                                    detail=f"adapter bundle member {info.name!r} "
+                                           "escapes the blob store")
+        os.makedirs(dest, exist_ok=True)
+        try:
+            tar.extractall(dest, filter="data")
+        except TypeError:          # filter= is newer than some runtimes
+            tar.extractall(dest)
 
 
 def save_lora_blob(data: bytes) -> tuple:
-    """Write `data` to the content-addressed store. Returns (hex_sha256, existed)."""
+    """Write `data` to the content-addressed store. Returns (hex_sha256, existed).
+
+    Tar bytes are a bundled adapter DIRECTORY (a PEFT adapter: config plus
+    weights) and are unpacked, because vLLM loads a text LoRA from a directory
+    and cannot do anything with the tar. Detected from the tar magic rather than
+    from a flag, so the sender does not have to declare the shape.
+    """
     import hashlib
+    from codai.models.text_loras import is_bundle
     h = hashlib.sha256(data).hexdigest()
-    p = _lora_blob_path(h)
+    if is_bundle(data):
+        dest = os.path.join(_lora_blob_dir(), h)
+        if os.path.isdir(dest):
+            return h, True
+        tmp = dest + ".tmp"
+        import shutil
+        shutil.rmtree(tmp, ignore_errors=True)
+        _unpack_lora_bundle(data, tmp)
+        os.replace(tmp, dest)
+        return h, False
+    p = os.path.join(_lora_blob_dir(), f"{h}.safetensors")
     if os.path.isfile(p):
         return h, True
     tmp = p + ".tmp"
@@ -2846,6 +2905,11 @@ async def lora_blob_info(hash: str, _auth=Depends(_require_api_auth)):
     if not lora_blob_exists(hash):
         raise HTTPException(status_code=404, detail="blob not found")
     p = _lora_blob_path(hash)
+    if os.path.isdir(p):
+        size = sum(os.path.getsize(os.path.join(r, f))
+                   for r, _d, fs in os.walk(p) for f in fs)
+        return {"id": f"sha256:{os.path.basename(p)}", "bytes": size,
+                "exists": True, "unpacked": True}
     return {"id": f"sha256:{os.path.basename(p)[:-len('.safetensors')]}",
             "bytes": os.path.getsize(p), "exists": True}
 

codai/api/remote_gateway.py

@@ -935,22 +935,51 @@ def sync_loras(base: str, api_key: str, body: bytes) -> bytes:
 _UPLOADED: set = set()
 
 
+#: (base, path, size, mtime) -> blob id, for adapters this process has already
+#: put on that pod. A Target is rebuilt for EVERY request, so without this the
+#: adapter was re-read and re-hashed on each one — 159 MB of disk and a round
+#: trip per request — only to learn the pod already had it. Keyed on the pod's
+#: URL, so a new pod re-sends; keyed on size+mtime, so a RETRAINED adapter at the
+#: same path is sent again rather than served stale.
+_LORA_SENT: dict = {}
+
+
 def push_lora_file(base: str, api_key: str, path: str) -> str:
-    """Put one adapter file in a remote's blob store; return its ``sha256:`` id.
+    """Put one adapter in a remote's blob store; return its ``sha256:`` id.
+
+    Takes a single file, or a PEFT adapter DIRECTORY — vLLM loads a text LoRA
+    from a directory (``adapter_config.json`` plus weights), so a trained adapter
+    is bundled into one deterministic tar and unpacked on the far side. Without
+    that, the only adapters that could reach a pod were HuggingFace repo ids.
 
     Content-addressed, so a pod that already has it is told nothing and a repeat
-    costs one HEAD-shaped check. Returns '' when the transfer fails, which the
-    caller reports rather than silently serving a model without its adapter.
+    costs one existence check — or nothing at all, once this process knows. Returns
+    '' when the transfer fails, which the caller reports rather than silently
+    serving a model without its adapter.
     """
     import hashlib
-    import requests
 
-    if not path or not os.path.isfile(path):
+    from codai.models.text_loras import bundle_dir
+
+    if not path or not os.path.exists(path):
         return ""
+    is_dir = os.path.isdir(path)
+    try:
+        stamp = _lora_stamp(path)
+    except Exception:
+        stamp = None
+    cached = _LORA_SENT.get((base, path, stamp)) if stamp else None
+    if cached:
+        return cached
     headers = {"Authorization": f"Bearer {api_key}"} if api_key else {}
     try:
-        with open(path, "rb") as fh:
-            data = fh.read()
+        if is_dir:
+            data = bundle_dir(path)
+            if not data:
+                return ""
+        else:
+            with open(path, "rb") as fh:
+                data = fh.read()
         digest = hashlib.sha256(data).hexdigest()
         have = pod_http.get(f"{base}/v1/loras/blob/{digest}", headers=headers, timeout=30)
         if have.status_code != 200:
@@ -959,15 +988,37 @@ def push_lora_file(base: str, api_key: str, path: str) -> str:
                                         "Content-Type": "application/octet-stream"},
                                timeout=1800)
             up.raise_for_status()
-            print(f"[remote-gateway] sent adapter {os.path.basename(path)} "
+            print(f"[remote-gateway] sent adapter {os.path.basename(path.rstrip('/'))}"
+                  f"{' (bundled directory)' if is_dir else ''} "
                   f"({len(data) / 1e6:.0f} MB) to {base}", flush=True)
-        return f"sha256:{digest}"
+        blob = f"sha256:{digest}"
+        if stamp:
+            _LORA_SENT[(base, path, stamp)] = blob
+        return blob
     except Exception as exc:
         print(f"[remote-gateway] could not send adapter {path} to {base}: {exc}",
               flush=True)
         return ""
 
 
+def _lora_stamp(path: str):
+    """(size, mtime) for a file, or for every file in an adapter directory.
+
+    What makes the cache safe to trust: retrain the adapter in place and the
+    stamp moves, so it is sent again instead of the pod keeping the old weights.
+    """
+    if os.path.isfile(path):
+        st = os.stat(path)
+        return (st.st_size, int(st.st_mtime))
+    out = []
+    for root, dirs, files in os.walk(path):
+        dirs.sort()
+        for name in sorted(files):
+            st = os.stat(os.path.join(root, name))
+            out.append((name, st.st_size, int(st.st_mtime)))
+    return tuple(out)
+
+
 def ensure_text_loras(base: str, api_key: str, entry: dict) -> dict:
     """Send a text model's LOCAL adapters to a coderai pod.
 
@@ -997,13 +1048,8 @@ def ensure_text_loras(base: str, api_key: str, entry: dict) -> dict:
         if not path:
             dropped.append(source)
             continue
-        if os.path.isdir(path):
-            # A PEFT adapter directory is several files; the blob store holds
-            # single files. Publishing it to HuggingFace is the way to move it.
-            dropped.append(source)
-            print(f"[remote-gateway] {source} is a directory — publish it to "
-                  "HuggingFace to use it on a pod", flush=True)
-            continue
+        # A PEFT adapter directory travels as one bundled blob; push_lora_file
+        # takes either shape.
         blob = push_lora_file(base, api_key, path)
         if blob:
             rewritten.append({**spec, "source": blob})

codai/api/vllm_worker.py

@@ -187,25 +187,99 @@ def _launch_cmd(py, cfg, host: str, port: int, model_path: str,
     return cmd
 
 
+def _base_of(entry: dict) -> str:
+    """What weights a model entry loads, for deciding which models share a base."""
+    block = entry.get("runpod") if isinstance(entry.get("runpod"), dict) else {}
+    return str(entry.get("path") or block.get("hf_repo") or entry.get("id") or "").strip().lower()
+
+
+def _sibling_lora_specs(model_config: dict) -> list:
+    """Adapters of this model AND of every other model loading the same base.
+
+    vLLM registers adapters at LAUNCH, and one vLLM serves one base model — so a
+    pod launched for the base alone could never serve an adapter of it, however
+    the request was routed. Observed as the whole point of a shared pool not
+    working: ``qwen38-awq`` creates the pool, its vLLM starts with no
+    ``--enable-lora``, and a later ``diogene-alpha`` request has nowhere to go.
+
+    Collecting siblings here makes one engine serve the base and each adapter
+    under its own served name, which is what lets different requests select a
+    different adapter with no restart.
+    """
+    from codai.models.text_loras import configured_specs
+
+    specs = list(configured_specs(model_config or {}))
+    base = _base_of(model_config or {})
+    if not base:
+        return specs
+    try:
+        from codai.admin.routes import config_manager
+        md = getattr(config_manager, "models_data", None) or {}
+    except Exception:
+        return specs
+    mine = str((model_config or {}).get("id") or "")
+    for entry in (md.get("text_models") or []):
+        if not isinstance(entry, dict) or str(entry.get("id") or "") == mine:
+            continue
+        if _base_of(entry) != base:
+            continue
+        for spec in configured_specs(entry):
+            # A sibling's adapter is served under the sibling's OWN name: that is
+            # the name the request will ask for.
+            spec = dict(spec)
+            spec["name"] = str(entry.get("id") or spec.get("name"))
+            specs.append(spec)
+    seen, unique = set(), []
+    for spec in specs:
+        if spec.get("name") in seen:
+            continue
+        seen.add(spec.get("name"))
+        unique.append(spec)
+    return unique
+
+
 def _lora_args(model_config: dict) -> list:
-    """`--enable-lora --lora-modules …` for the adapters a model config asks for."""
+    """`--enable-lora --lora-modules …` for the adapters this engine must serve."""
     try:
-        from codai.models.text_loras import configured_specs, describe
-        specs = configured_specs(model_config or {})
+        from codai.models.text_loras import describe, is_portable, local_path
+        specs = _sibling_lora_specs(model_config or {})
     except Exception:
         return []
     if not specs:
         return []
-    args = ["--enable-lora"]
-    mods = []
+    mods, usable = [], []
     for spec in specs:
-        mods.append(f"{spec['name']}={spec['source']}")
-    args += ["--lora-modules"] + mods
+        src = str(spec.get("source") or "")
+        if src.startswith(("sha256:", "name:")):
+            # An uploaded adapter is named by CONTENT id, which is not a path: it
+            # has to be looked up in the blob store first. Passing the id straight
+            # through produced `--lora-modules name=sha256:…`, which vLLM cannot
+            # open — and the launch failed minutes later, inside a pod.
+            resolved = local_path(src)
+            if not resolved:
+                print(f"[lora] {spec.get('name')!r}: adapter {src} is not in this "
+                      f"machine's blob store — serving without it", flush=True)
+                continue
+        else:
+            # A path or a repo id is passed as configured: vLLM resolves both,
+            # and second-guessing a path here would drop an adapter that a
+            # mount makes appear a moment later.
+            resolved = src
+        mods.append(f"{spec['name']}={resolved}")
+        usable.append(spec)
+    if not mods:
+        return []
+    args = ["--enable-lora", "--lora-modules"] + mods
     # vLLM rejects an adapter whose rank exceeds --max-lora-rank (default 16),
     # and the message points at the flag rather than the adapter, so raise the
     # ceiling to what vLLM supports rather than have a trained LoRA refused.
     args += ["--max-lora-rank", "64"]
-    print(f"[lora] vLLM will serve adapters: {describe(specs)}", flush=True)
+    # How many DISTINCT adapters may be active in one batch. The default is 1,
+    # so with several registered, requests naming different ones cannot batch
+    # together and serialise instead — the cost lands on latency, not on an
+    # error, which is why it goes unnoticed.
+    args += ["--max-loras", str(max(1, min(len(mods), 8)))]
+    print(f"[lora] vLLM will serve adapters: {describe(usable)}", flush=True)
     return args
 
 

codai/models/text_loras.py

@@ -144,14 +144,67 @@ def is_portable(source: str) -> bool:
         and src.count("/") == 1
 
 
+def bundle_dir(path: str) -> bytes:
+    """A PEFT adapter DIRECTORY as one deterministic tar, for the blob store.
+
+    vLLM loads a text LoRA from a directory — ``adapter_config.json`` plus the
+    weights — but the blob store holds single files, so a trained adapter could
+    not travel to a pod at all: it was reported and dropped, and the only way to
+    serve one was to publish it to HuggingFace. Tarring makes it one blob, which
+    is the thing the store can carry.
+
+    Deterministic on purpose: members sorted, uid/gid/mtime/mode fixed. The id is
+    the content hash, so the same adapter must hash the same on every call —
+    otherwise a pod would be re-sent 159 MB because a timestamp moved.
+    """
+    import io
+    import tarfile
+    if not path or not os.path.isdir(path):
+        return b""
+    names = []
+    for root, dirs, files in os.walk(path):
+        dirs.sort()
+        for name in sorted(files):
+            full = os.path.join(root, name)
+            names.append((os.path.relpath(full, path), full))
+    names.sort()
+    buf = io.BytesIO()
+    # No gzip: safetensors do not compress, and gzip would stamp an mtime.
+    with tarfile.open(fileobj=buf, mode="w", format=tarfile.USTAR_FORMAT) as tar:
+        for rel, full in names:
+            info = tar.gettarinfo(full, arcname=rel)
+            info.uid = info.gid = 0
+            info.uname = info.gname = ""
+            info.mtime = 0
+            info.mode = 0o644
+            with open(full, "rb") as fh:
+                tar.addfile(info, fh)
+    return buf.getvalue()
+
+
+def is_bundle(data: bytes) -> bool:
+    """True when these bytes are a tar — i.e. an adapter directory, not a file.
+
+    Read from the tar header's own magic rather than from a filename, so the
+    receiving end does not have to be told which shape it was sent.
+    """
+    return len(data) > 262 and data[257:262] == b"ustar"
+
+
 def blob_id(path: str) -> str:
-    """The ``sha256:<hex>`` id an adapter file will have in any blob store.
+    """The ``sha256:<hex>`` id an adapter will have in any blob store.
 
     Computed here, so a pod can be TOLD the id when it is created and be SENT the
-    bytes afterwards — the two agree without the pod having to exist yet.
+    bytes afterwards — the two agree without the pod having to exist yet. A
+    directory hashes as its bundle, so the id matches what is actually sent.
     """
     import hashlib
-    if not path or not os.path.isfile(path):
+    if not path:
+        return ""
+    if os.path.isdir(path):
+        data = bundle_dir(path)
+        return f"sha256:{hashlib.sha256(data).hexdigest()}" if data else ""
+    if not os.path.isfile(path):
         return ""
     h = hashlib.sha256()
     with open(path, "rb") as fh:

docs/remote-execution.md

@@ -579,7 +579,7 @@ Each runtime applies them its own way, and what reaches a pod differs:
 |---|---|---|
 | transformers (HF) | PEFT `load_adapter`, several at once | via a coderai pod, adapters travel with the seed |
 | llama.cpp (GGUF) | one adapter, **GGUF-converted** | needs the file; use a HF repo id or a coderai pod |
-| vLLM | `--enable-lora --lora-modules` at launch | **HuggingFace repo ids only** |
+| vLLM | `--enable-lora --lora-modules` at launch | a repo id, or an adapter sent to a coderai pod |
 | diffusers (image/video) | `load_lora_weights` | uploaded to the remote automatically, content-addressed |
 
 The asymmetry is worth knowing: a **diffusion** LoRA is sent with the request
@@ -599,9 +599,41 @@ endpoint that can receive one. The sequence:
 3. the pod resolves `sha256:<hex>` back to a file and applies it with PEFT.
 
 A **published** adapter (a HuggingFace repo id) skips all of that and goes to a
-vLLM pod, which fetches it itself — cheaper and faster, so publish adapters you
-use often. A PEFT adapter **directory** cannot be sent (the blob store holds
-single files); publish that one.
+plain vLLM pod, which fetches it itself — cheaper and faster, so publish adapters
+you use often.
+
+A PEFT adapter **directory** travels too, which matters because that is the shape
+a trainer writes: `adapter_config.json` plus the weights. It is bundled into one
+deterministic tar (sorted names, fixed uid/gid/mtime, so the content id is
+stable), stored under its hash, and unpacked on the far side — vLLM needs the
+*directory*, since the config carries the rank and target modules, and a bare
+`.safetensors` is not loadable. For ten days this was the gap that kept a trained
+client adapter off its pod, with "publish it to HuggingFace" as the only advice.
+
+#### Selecting an adapter per request
+
+vLLM registers adapters **at launch**, and one vLLM serves one base model, so the
+engine for a base must be launched knowing every adapter of it. Each model in
+`models.json` that names the same base contributes its adapter under **its own
+model id**, which is the name a request asks for:
+
+```jsonc
+{"id": "qwen38-awq",     "runpod": {"hf_repo": "…/Qwen3.8-27B-AWQ-INT4", "pool": "qwen38"}}
+{"id": "diogene-alpha",  "loras": [{"path": "/root/.coderai/loras/diogene-alpha"}],
+                         "runpod": {"hf_repo": "…/Qwen3.8-27B-AWQ-INT4", "pool": "qwen38"}}
+```
+
+One pod, one copy of the weights: `model: "qwen38-awq"` answers from the base and
+`model: "diogene-alpha"` from the adapter, request by request, with no reload. Register
+two versions of an adapter as two ids and you can A/B them on the same GPU.
+
+`--max-loras` is set to the number registered (capped at 8). Left at its default
+of 1, requests naming *different* adapters cannot share a batch and serialise
+instead — which costs latency rather than raising an error, so it goes unnoticed.
+
+A shared `pool` requires a coderai pod image: `shared_pool_key` refuses a plain
+`vllm`/`llamacpp` engine, because such a pod is launched for one model and would
+answer for another one's name with the wrong weights.
 
 Build and push the text image before using it: