an adapter id named the container, not the content (v0.3.94)

The root cause under four releases of "the adapter is not applied". Measured on
a live pod:

    catalogue says       : sha256:b8eac366…   (hash of the tar a client sent)
    this bundler's own   : sha256:2e820c…     (same content, its own packing)
    pod lookup b8eac366  : 404
    pod upload returns   : {"id":"sha256:2e820c…","existed":true}

`save_lora_blob` stored an uploaded bundle under the hash of THE BYTES THAT
ARRIVED. A tar of one adapter made by GNU tar, by this bundler, or by anything
else hashes differently — mtimes, ownership, member order — so the id recorded
in the catalogue was one nobody else could reproduce. Every later push
re-bundled the unpacked directory, stored the canonical id, and the pod looked
up the uploaded one and found nothing. It had the bytes the whole time, under
another name, so it loaded the base model and answered under the adapter's
name. Fluently, with nothing in the reply to say so.

An uploaded bundle is now unpacked FIRST and stored under the id of this
bundler's packing of it, which is what every side computes. /v1/loras/upload
returns that id, so what a client registers is what a pod can resolve.

This is also why it took four attempts: the engine flags, the seeding and the
entry propagation were each genuinely broken and each fix was real, but
underneath them the two sides were never going to agree on the id.

_load_lora_runtime had two bare `return False` paths — no adapter spec, or an
id the blob store cannot resolve — which is exactly why this hid for three
releases while every other layer was examined. Both say what happened now, and
the second says the thing that matters: the catalogue and the store disagree
about this adapter's id.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.93"
+__version__ = "0.3.94"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/loras.py

@@ -333,18 +333,36 @@ def save_lora_blob(data: bytes) -> tuple:
     from a flag, so the sender does not have to declare the shape.
     """
     import hashlib
-    from codai.models.text_loras import is_bundle
+    import shutil
+    from codai.models.text_loras import blob_id, is_bundle
     h = hashlib.sha256(data).hexdigest()
     if is_bundle(data):
-        dest = os.path.join(_lora_blob_dir(), h)
-        if os.path.isdir(dest):
-            return h, True
-        tmp = dest + ".tmp"
-        import shutil
+        # The id must name the CONTENT, not the container it arrived in. A tar
+        # of the same adapter made by GNU tar, by this bundler, or by anything
+        # else hashes differently -- mtimes, ownership, member order -- so an
+        # id taken from the uploaded bytes is one that nobody else can
+        # reproduce. Measured live: a client uploaded a tar hashing to
+        # b8eac366…, the catalogue recorded that, every subsequent push
+        # re-bundled the unpacked directory and stored 2e820c…, and the pod
+        # looked up b8eac366… and got a 404. It HAD the bytes, under another
+        # name, and so served the base model under the adapter's name.
+        #
+        # So an uploaded bundle is unpacked first and stored under the id of
+        # THIS bundler's packing of it, which is what every side computes.
+        tmp = os.path.join(_lora_blob_dir(), f"incoming-{h}.tmp")
         shutil.rmtree(tmp, ignore_errors=True)
         _unpack_lora_bundle(data, tmp)
+        canonical = (blob_id(tmp) or f"sha256:{h}").split(":", 1)[-1]
+        dest = os.path.join(_lora_blob_dir(), canonical)
+        if os.path.isdir(dest):
+            shutil.rmtree(tmp, ignore_errors=True)
+            return canonical, True
         os.replace(tmp, dest)
-        return h, False
+        if canonical != h:
+            print(f"[lora] stored an uploaded bundle as {canonical[:12]}… "
+                  f"(its own bytes hashed to {h[:12]}…): the id names the "
+                  f"content, so any sender computes the same one", flush=True)
+        return canonical, False
     p = os.path.join(_lora_blob_dir(), f"{h}.safetensors")
     if os.path.isfile(p):
         return h, True

codai/api/vllm_worker.py

@@ -553,14 +553,24 @@ def _load_lora_runtime(url: str, model_config: dict, served_name: str) -> bool:
     spec = next((s for s in _sibling_lora_specs(model_config or {})
                  if str(s.get("name") or "") == want), None)
     if spec is None:
+        print(f"[lora] {want!r} is not an adapter of this model here — nothing "
+              f"to add to the running engine", flush=True)
         return False
+    src = str(spec.get("source") or "")
     try:
         from codai.models.text_loras import local_path
-        src = str(spec.get("source") or "")
         path = local_path(src) if src.startswith(("sha256:", "name:")) else src
-    except Exception:
+    except Exception as exc:
+        print(f"[lora] could not resolve {src} ({exc})", flush=True)
         path = ""
     if not path:
+        # Said out loud because the alternative is this function returning
+        # False with no explanation, and the engine then serving the BASE
+        # under the adapter's name. That is exactly how an id mismatch between
+        # the catalogue and the blob store stayed invisible for three releases.
+        print(f"[lora] {want!r}: {src} is not in this machine's blob store — "
+              f"cannot add it to the running engine. The catalogue and the "
+              f"store disagree about this adapter's id.", flush=True)
         return False
     body = _json.dumps({"lora_name": want, "lora_path": path}).encode()
     req = urllib.request.Request(url.rstrip("/") + "/v1/load_lora_adapter",

tests/test_field_report_fixes.py

@@ -387,3 +387,91 @@ async def test_the_base_model_is_not_listed_as_an_adapter(monkeypatch):
     _catalogue(monkeypatch, [dict(BASE_MODEL)])
     monkeypatch.setattr(loras, "_loras_dir", lambda: "/nonexistent")
     assert (await loras.list_loras())["loras"] == []
+
+
+# ============================================================ adapter ids
+# An uploaded bundle was stored under the hash of THE BYTES THAT ARRIVED. A tar
+# of the same adapter made by GNU tar, by coderai's bundler, or by anything
+# else hashes differently — mtimes, ownership, member order — so the id in the
+# catalogue was one nobody else could reproduce. Measured live: the client
+# uploaded a tar hashing to b8eac366…, every later push re-bundled the unpacked
+# directory and stored 2e820c…, and the pod looked up b8eac366… and 404'd. It
+# had the bytes under another name, so it served the BASE under the adapter's
+# name — fluently, with nothing in the response to say so.
+def test_an_uploaded_bundle_is_stored_under_its_canonical_content_id(tmp_path,
+                                                                     monkeypatch):
+    import hashlib
+    import io
+    import tarfile
+
+    from codai.api import loras
+    from codai.models import text_loras
+
+    src = tmp_path / "adapter"
+    src.mkdir()
+    (src / "adapter_config.json").write_text('{"r": 8}')
+    (src / "adapter_model.safetensors").write_bytes(b"\x00" * 2048)
+
+    # A tar made by someone ELSE: different mtimes, so different bytes.
+    foreign = io.BytesIO()
+    with tarfile.open(fileobj=foreign, mode="w") as tf:
+        for f in sorted(src.iterdir()):
+            info = tarfile.TarInfo(f.name)
+            info.size = f.stat().st_size
+            info.mtime = 1700000000          # not what bundle_dir writes
+            with open(f, "rb") as fh:
+                tf.addfile(info, fh)
+    uploaded = foreign.getvalue()
+
+    store = tmp_path / "blobs"
+    store.mkdir()
+    monkeypatch.setattr(loras, "_lora_blob_dir", lambda: str(store))
+
+    h, existed = loras.save_lora_blob(uploaded)
+    assert not existed
+    # NOT the hash of the bytes that arrived…
+    assert h != hashlib.sha256(uploaded).hexdigest()
+    # …but the id any side computes from the content it now holds.
+    assert f"sha256:{h}" == text_loras.blob_id(str(store / h))
+
+
+def test_re_uploading_the_same_adapter_is_recognised(tmp_path, monkeypatch):
+    """Two different tars of one adapter must not become two blobs."""
+    import io
+    import tarfile
+
+    from codai.api import loras
+
+    src = tmp_path / "a"
+    src.mkdir()
+    (src / "adapter_config.json").write_text('{"r": 4}')
+
+    def _tar(mtime):
+        buf = io.BytesIO()
+        with tarfile.open(fileobj=buf, mode="w") as tf:
+            info = tarfile.TarInfo("adapter_config.json")
+            data = (src / "adapter_config.json").read_bytes()
+            info.size = len(data)
+            info.mtime = mtime
+            tf.addfile(info, io.BytesIO(data))
+        return buf.getvalue()
+
+    store = tmp_path / "blobs"
+    store.mkdir()
+    monkeypatch.setattr(loras, "_lora_blob_dir", lambda: str(store))
+
+    first, e1 = loras.save_lora_blob(_tar(1700000000))
+    second, e2 = loras.save_lora_blob(_tar(1800000000))
+    assert first == second and not e1 and e2
+    assert len([p for p in store.iterdir() if p.is_dir()]) == 1
+
+
+def test_a_plain_file_adapter_still_hashes_as_itself(tmp_path, monkeypatch):
+    from codai.api import loras
+    import hashlib
+    store = tmp_path / "blobs"
+    store.mkdir()
+    monkeypatch.setattr(loras, "_lora_blob_dir", lambda: str(store))
+    data = b"not a tar, just safetensors bytes"
+    h, existed = loras.save_lora_blob(data)
+    assert h == hashlib.sha256(data).hexdigest() and not existed