decide whether a GGUF is readable before evicting anything for it

Two models could not be loaded in one afternoon, and establishing why took a
diagnosis each. Neither was a memory problem; both failed late and expensively,
after other models had been turned out of the card to make room.

The second one is the reason for this. A 27B "Flash" quant passed every check a
careful operator would make -- right architecture, all 65 blocks present
including the MTP one, an mmproj beside it -- and stored 402 of its 866 tensors
in ggml type 143, a rotation-folded format from a fork. Nothing in the metadata
announces that: the type ids live in the tensor index, which no error message
reads back to you. It arrived as a generic load failure after the eviction, six
retries and a full read of the weights.

gguf_preflight answers it from the header alone and runs before the VRAM need is
even computed, so a load that was never going to work costs no other model its
residency. The bound is llama.cpp's own GGML_TYPE_COUNT rather than a list kept
here, so it tracks the installed library.

Only ids at or above that bound are refused, and the conservatism is the design.
The Python binding's GGML_TYPE_* constants are an INCOMPLETE mirror of the
compiled enum -- 0-3, 6-29, 39-42 -- so 30..38 have no constant, and 30 is BF16,
which files we serve today use and which loads perfectly. "Absent from the
binding's constants" would have rejected every one of them. An id inside the
enum that the build still cannot handle is left to llama.cpp to refuse in its own
words.

The verdict gets its own exception type because the on-demand path turns a failed
VRAM reservation into a printed warning and carries on -- correctly for a
reservation, fatally for a verdict, which would then reappear a minute later as
the original confusing failure. That path now re-raises GgufUnreadable, the same
treatment HostRamExhausted already gets, and the comment three lines below it
("a host-RAM refusal is the decision itself and must reach the caller") is why.
It reports 501, not 503: evicting and retrying cannot change the answer, so a
client that retries a 503 forever must not be invited to. The type ids ride along
in the response, being the actionable part.

The architecture is printed before the load too, so llama.cpp's "unknown model
architecture" sits beside the file it came from. Deliberately NOT a list of
supported architectures: there is no API for it, and guessing from the shared
library's strings would invent a second source of truth that can be wrong in the
direction that blocks a model which would have worked.

No fourth GGUF parser -- the tensor-type histogram rides along on the walk
gguf_mtp.read_layout already does, and a test asserts gguf_preflight has no
parser of its own. docs/serving-a-model-this-box-cannot-hold.md records the three
independent requirements, how to verify each on a new machine, the quant/size
table, why an mtp-*.gguf is not a model, and which of the three things called
"MTP support" already exist (vLLM's raw --speculative-config passthrough, ds4's
--mtp) versus which does not (a GGUF draft model, which llama-cpp-python cannot
express).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.5.43"
+__version__ = "0.5.44"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/app.py

@@ -960,6 +960,34 @@ async def host_ram_exhausted_handler(request: Request, exc: HostRamExhausted):
     )
 
 
+from codai.backends.gguf_preflight import GgufUnreadable
+
+
+@app.exception_handler(GgufUnreadable)
+async def gguf_unreadable_handler(request: Request, exc: GgufUnreadable):
+    """A load was refused because this build cannot read the file.
+
+    501, not 503: unlike the host-RAM refusal this is not transient and no
+    amount of evicting or retrying changes it. The file stores weights in a
+    quantization format the installed llama.cpp has no implementation for, so
+    the honest answer is that the server will not be able to serve this model
+    until it is replaced or the engine is upgraded — and a client that retries a
+    503 forever should not be invited to.
+
+    The type ids are carried out because they are the actionable part and
+    nothing else reports them: they live in the GGUF tensor index, which no
+    error message reads back to you.
+    """
+    return JSONResponse(
+        status_code=501,
+        content={"error": {"message": str(exc),
+                           "type": "unsupported_model_format",
+                           "param": exc.path,
+                           "code": "gguf_unreadable",
+                           "tensor_types": {str(k): v for k, v in exc.types.items()}}},
+    )
+
+
 @app.get("/v1/models", response_model=ModelList, summary="List available models", tags=["Core"])
 async def list_models():
     """List available models."""

codai/backends/gguf_mtp.py

@@ -52,12 +52,19 @@ def _read_val(f, vt):
 
 
 def read_layout(path) -> dict:
-    """Declared vs. present block layout of a GGUF.
-
-    Returns ``{arch, block_count, nextn, blocks_present, contiguous}``, or ``{}``
-    when the header cannot be read. Unlike the other GGUF probes in this package
-    this one walks the tensor *index* as well — the whole question is which
-    blocks the file actually carries — but it still reads no tensor data.
+    """Declared vs. present block layout of a GGUF, and its tensor types.
+
+    Returns ``{arch, block_count, nextn, blocks_present, contiguous,
+    tensor_types}``, or ``{}`` when the header cannot be read. Unlike the other
+    GGUF probes in this package this one walks the tensor *index* as well — the
+    whole question is which blocks the file actually carries — but it still reads
+    no tensor data.
+
+    ``tensor_types`` is the ggml type id histogram, ``{type_id: count}``. It is
+    collected here because this walk already passes every type field, and
+    :mod:`codai.backends.gguf_preflight` needs it to answer whether the loaded
+    llama.cpp can read the file at all. Four GGUF header parsers in one package
+    is three too many, so this one is shared rather than copied.
     """
     if not path:
         return {}
@@ -83,12 +90,14 @@ def read_layout(path) -> dict:
                 elif key.endswith('.nextn_predict_layers'):
                     nextn = int(val)
             blocks = set()
+            types: dict = {}
             for _ in range(n_tensors):
                 name = _read_str(f)
                 n_dims = struct.unpack('<I', f.read(4))[0]
                 f.read(8 * n_dims)                              # dims
-                f.read(4)                                       # ggml type
+                ttype = struct.unpack('<I', f.read(4))[0]       # ggml type
                 f.read(8)                                       # offset
+                types[ttype] = types.get(ttype, 0) + 1
                 if name.startswith('blk.'):
                     part = name.split('.', 2)[1]
                     if part.isdigit():
@@ -99,6 +108,7 @@ def read_layout(path) -> dict:
                 'nextn': nextn,
                 'blocks_present': len(blocks),
                 'contiguous': bool(blocks) and blocks == set(range(len(blocks))),
+                'tensor_types': types,
             }
     except Exception:
         out = {}

codai/backends/gguf_preflight.py

+"""Can this build read this file at all — answered before anything is loaded.
+
+A GGUF that the installed llama.cpp cannot read fails late and expensively. By
+the time the load is attempted, models have already been evicted to make room,
+VRAM has been reserved, and for a large file minutes have gone into reading it
+— and the error that comes back is llama.cpp's, phrased for whoever wrote
+llama.cpp. Two such files turned up on one afternoon:
+
+  * a Qwen3.8-Flash-Next quant whose architecture (``qwen4exp``) this build does
+    not implement, which cost a whole diagnosis to establish;
+  * a 27B "Flash" quant that looked perfect — right architecture, every block
+    present, an mmproj alongside — and carried 402 weight tensors in ggml type
+    143, a rotation-based format from a fork. Nothing in the file's metadata
+    says so; the type ids are in the tensor index, which no error message reads
+    back to you.
+
+The checks here are metadata-only: the GGUF header and tensor index, no tensor
+data, no model load, no eviction. That is the whole point — a check that has to
+load the model to decide is not a check that can run before loading the model.
+
+Deliberately NOT here: a list of architectures this build supports. There is no
+API for it, llama.cpp already reports ``unknown model architecture: '<name>'``
+in its own log, and the loader surfaces that. Guessing the list from the shared
+library's strings would invent a second source of truth that can be wrong in the
+direction that blocks a model which would have worked.
+"""
+
+from codai.backends.gguf_mtp import read_layout
+
+
+class GgufUnreadable(RuntimeError):
+    """This build cannot read this file, decided from its header.
+
+    Its own type, because the VRAM-reservation path turns a failed reservation
+    into a printed warning and carries on — correctly, since a reservation that
+    could not be made is often still a load that works. This is not that: it is
+    a verdict, and a verdict swallowed into a warning is a bug that reappears as
+    the original confusing failure a minute later. The host-RAM refusal is
+    handled the same way for the same reason.
+    """
+
+    def __init__(self, message, path=None, types=None):
+        super().__init__(message)
+        self.path = path
+        self.types = dict(types or {})
+
+
+def unsupported_tensor_types(path, type_count) -> dict:
+    """``{type_id: tensor_count}`` for types this build has no enum value for.
+
+    ``type_count`` is llama.cpp's own ``GGML_TYPE_COUNT``, so the bound comes
+    from the loaded library rather than from a list kept here.
+
+    Only ids at or above that bound are reported, and that conservatism is the
+    design. The Python binding's ``GGML_TYPE_*`` constants are an INCOMPLETE
+    mirror of the compiled enum -- it defines 0-3, 6-29 and 39-42, while 30 is
+    BF16, which plenty of files we serve use and which loads perfectly. Treating
+    "absent from the binding's constants" as "unsupported" would reject those
+    outright. An id below the bound that the build still cannot handle is left
+    to llama.cpp to refuse in its own words; this only catches what is provably
+    outside the enum.
+    """
+    try:
+        bound = int(type_count)
+    except (TypeError, ValueError):
+        return {}
+    if bound <= 0:
+        return {}
+    types = (read_layout(path) or {}).get('tensor_types') or {}
+    return {tid: n for tid, n in sorted(types.items()) if int(tid) >= bound}
+
+
+def explain_tensor_types(path, unsupported: dict, type_count) -> str:
+    """Why this file cannot be read, in terms of the file rather than of ggml."""
+    layout = read_layout(path) or {}
+    total = sum((layout.get('tensor_types') or {}).values()) or 0
+    affected = sum(unsupported.values())
+    listed = ", ".join(f"type {tid} ({n} tensors)"
+                       for tid, n in sorted(unsupported.items()))
+    return (
+        f"this GGUF stores {affected} of its {total} tensors in a quantization "
+        f"format the installed llama.cpp does not implement: {listed}. The "
+        f"highest type this build knows is {int(type_count) - 1} "
+        f"(GGML_TYPE_COUNT={int(type_count)}), so these come from a fork or a "
+        f"newer ggml -- a rotation-folded quant (PRISM and similar) is the usual "
+        f"source, and nothing in the file's metadata announces it. Refusing here "
+        f"rather than after eviction and a full read of the weights. Use a quant "
+        f"in a standard format, or run it on a build that implements these types."
+    )
+
+
+def architecture(path) -> str:
+    """``general.architecture``, or '' when it cannot be read.
+
+    Logged before a load so that llama.cpp's own ``unknown model architecture``
+    is attributable to a name without re-reading the file by hand.
+    """
+    return str((read_layout(path) or {}).get('arch') or '')

codai/backends/vulkan.py

@@ -1716,6 +1716,19 @@ class VulkanBackend(ModelBackend):
         # — indistinguishable, from the outside, from a truncated download. When
         # the file's shape proves that is what happened, correct the count here;
         # see codai/backends/gguf_mtp.py for the conditions.
+        # Say which architecture is being attempted. llama.cpp reports an
+        # architecture it does not implement as "unknown model architecture:
+        # '<name>'", which names the thing but not the file; printing it here
+        # means the two are already side by side in the log instead of needing
+        # the GGUF header read by hand to connect them.
+        try:
+            from codai.backends import gguf_preflight as _pf
+            _arch = _pf.architecture(model_path)
+            if _arch:
+                print(f"  arch         : {_arch}")
+        except Exception:
+            pass
+
         _kv_over = dict(_raw_cfg.get('kv_overrides') or {})
         if bool(_raw_cfg.get('mtp_block_fix', True)):
             from codai.backends import gguf_mtp

codai/models/manager.py

@@ -311,6 +311,45 @@ _SEED_CACHE = None
 _REMOTE_BACKENDS = ("runpod", "remote", "node", "cluster")
 
 
+def _gguf_unreadable_reason(*names):
+    """``(why, path, types)`` when llama.cpp cannot read this GGUF, else None.
+
+    Called before eviction, because a model that cannot possibly load has no
+    business turning other models out of the card first. Measured on zeiss: a
+    27B quant that looked perfect — right architecture, every block present, an
+    mmproj beside it — stored 402 of its 866 tensors in ggml type 143, a format
+    from a fork. Nothing in the metadata says so, and the failure arrived after
+    the eviction, six load attempts and a full read of the weights.
+
+    Returns '' for anything it cannot decide, which is most things: a non-GGUF
+    model, a path that is not local, a header it cannot parse, or a process
+    without llama_cpp (the front has no CUDA libraries, so the import fails
+    there and the check simply does not run — the engine, which is where this
+    is reached before a GGUF load, has it).
+    """
+    import os as _os
+    for name in names:
+        entry = _model_entry_for(name) if name else None
+        path = (entry or {}).get('path') or (name if isinstance(name, str) else '')
+        if not isinstance(path, str) or not path.lower().endswith('.gguf'):
+            continue
+        if not _os.path.isfile(path):
+            continue
+        try:
+            import llama_cpp as _lc
+            from codai.backends import gguf_preflight as _pf
+            bound = getattr(_lc, 'GGML_TYPE_COUNT', None)
+            if not bound:
+                return ''
+            bad = _pf.unsupported_tensor_types(path, bound)
+            if bad:
+                return (_pf.explain_tensor_types(path, bad, bound), path, bad)
+        except Exception:
+            return None
+        return None
+    return None
+
+
 def _model_entry_for(model_name: str):
     """The models.json entry (dict) matching ``model_name`` by path / basename /
     alias / id, or None."""
@@ -2320,8 +2359,15 @@ class MultiModelManager:
                 # direct on-demand path, which otherwise loads on top of the
                 # current model and OOMs (e.g. switching to a larger model).
                 if inst_num == 1:
+                    from codai.backends.gguf_preflight import GgufUnreadable
                     try:
                         self.ensure_vram_for(model_name, requested_id=requested_id)
+                    except GgufUnreadable:
+                        # A verdict, not a reservation failure: this build cannot
+                        # read the file at all, so the load below would fail a
+                        # minute later with llama.cpp's own wording after evicting
+                        # whatever was resident. Let it reach the caller.
+                        raise
                     except Exception as _ev_e:
                         print(f"  (ensure_vram_for warning: {_ev_e})")
                     # Outside that except on purpose: a VRAM reservation failing
@@ -6116,6 +6162,14 @@ class MultiModelManager:
         for _k in (model_key, resolved_name, requested_id):
             if _k and _k in self.models:
                 return
+        # Before anything is turned out of the card: can this file be read at
+        # all? Cheap (header + tensor index, no weights) and it keeps a load
+        # that was never going to work from costing every other resident model.
+        _verdict = _gguf_unreadable_reason(model_key, resolved_name, requested_id)
+        if _verdict:
+            from codai.backends.gguf_preflight import GgufUnreadable
+            _why, _path, _types = _verdict
+            raise GgufUnreadable(f"{model_key}: {_why}", path=_path, types=_types)
         needed_gb = self._get_model_used_vram_gb(model_key, resolved_name, requested_id)
         if extra_vram_gb and extra_vram_gb > 0:
             needed_gb += extra_vram_gb

docs/serving-a-model-this-box-cannot-hold.md

+# When a model does not fit, and what has to be true before it does
+
+Written from one afternoon spent establishing why `Qwen3.8-Flash-Next` would not
+serve. The conclusion was that nothing about it was a memory problem on the
+machine it was tried on, and that the next machine needs three separate things
+to be true, not one. This is the checklist, so the diagnosis is not repeated.
+
+## The three independent requirements
+
+A model needs all three. Fixing the one you notice first is the usual mistake.
+
+**1. The engine must implement its architecture.** This is not about hardware
+and no amount of memory changes it. Qwen3.8-Flash-Next is
+`Qwen4ExpForConditionalGeneration` / `model_type: qwen4_exp`, and on the build
+it was tried against:
+
+```
+llama.cpp implemented:  qwen35, qwen3moe, qwen3next, qwen3vlmoe
+vLLM registered:        Qwen3NextForCausalLM, Qwen3NextMTP,
+                        Qwen3VLForConditionalGeneration,
+                        Qwen3VLMoeForConditionalGeneration
+```
+
+Neither carried it. llama.cpp says `unknown model architecture: '<name>'` and
+the loader surfaces that; the architecture is also printed before the load, so
+the name and the file are already side by side in the log. **To check a machine
+before trying:** `strings …/libllama.so* | grep -ix <arch>` for the GGUF path,
+and grep vLLM's `model_executor/models/registry.py` for the HF class name.
+
+**2. The quantization must be one the engine can read.** Separate from the
+architecture, and invisible in the metadata. A 27B "Flash" quant passed every
+check an operator would make — right architecture, all 65 blocks present
+including the MTP one, an mmproj beside it — and stored 402 of its 866 tensors
+in **ggml type 143**, a rotation-folded format from a fork. The type ids live in
+the tensor index; nothing announces them.
+
+`codai/backends/gguf_preflight.py` now answers this from the header alone,
+before eviction, and names the type and how many tensors use it. The bound is
+llama.cpp's own `GGML_TYPE_COUNT`, so it tracks the installed library rather
+than a list kept in the repo.
+
+**3. The memory has to hold it.** Only here does the machine matter. For
+Qwen3.8-Flash-Next (125B-A5B, 360 GB in bf16 across 131 shards):
+
+| quant | size | | quant | size |
+|---|---|---|---|---|
+| UD-IQ1_S | 72.5 GB | | UD-IQ4_XS | 93.7 GB |
+| UD-IQ1_M | 74.5 GB | | UD-Q4_K_XL | 111.3 GB |
+| UD-Q2_K_XL | 78.9 GB | | Q8_0 | 188.2 GB |
+| UD-IQ3_XXS | 82.0 GB | | BF16 | 354.0 GB |
+
+An expert-pruned variant (`REAP128-86B`) exists at 48.9 GB, IQ4_XS only, and is
+still `qwen4exp`. Budget VRAM + host RAM against the admission floor, not
+against total RAM — see `memory-accounting.md`.
+
+## The MTP file is not a model
+
+`mtp-*.gguf` files hold the NextN / multi-token-prediction draft head **only**.
+One was 2.7 GB and contained a single block, `blk.48` of a declared 49: no
+embedding, no layers, nothing to run. It is an accessory to a base checkpoint,
+not something to serve, and `gguf_mtp.block_count_override` deliberately refuses
+to reinterpret one into a one-layer model.
+
+Three different things get called "MTP support":
+
+- **A quant that dropped its MTP block** but still declares it. llama.cpp asks
+  for a block the file has not got and reports `missing tensor
+  'blk.N.attn_norm.weight'`, which reads like a corrupt download. Handled —
+  `gguf_mtp` corrects the declared count when the file's shape proves that is
+  what happened.
+- **Speculative decoding through MTP.** Available on **vLLM** today: the
+  `speculative` setting passes raw JSON to `--speculative-config`, precisely so
+  a draft method can be configured without a code change. Also on **ds4**
+  (`--mtp`). **Not** on the llama.cpp path: `llama-cpp-python` exposes
+  `draft_model`, but its only implementation is `LlamaPromptLookupDecoding`, a
+  prompt-lookup n-gram drafter — there is no way to hand it a second GGUF, even
+  though the C library has the plumbing.
+- **An MTP block inside the main GGUF**, which the architecture's own code uses.
+  This build wants it: it asked for `blk.64` rather than stopping at 63.
+
+## Moving one of these to a bigger machine
+
+1. Confirm requirement 1 on that machine, for the engine you intend to use.
+   Nothing else matters until this passes.
+2. Pick a quant and check requirement 3 against VRAM + host RAM, remembering the
+   host-RAM admission floor.
+3. Download the **base**, not the `MTP/` folder.
+4. Set the card's `backend` to match the engine that will serve it. A GGUF path
+   with `backend: vllm` fails at the front with a pin conflict before anything
+   opens the file — which looks like an engine fault and is a config one.
+5. For vLLM, MTP speculative decoding goes in `speculative` as JSON.
+
+The preflight in requirement 2 runs by itself and will say if the quant is
+unreadable, before it costs an eviction.