runpod: a preferred GPU, not the only GPU (v0.3.1)

`gpu_type` was a filter: "NVIDIA A40" meant an A40 or nothing, so when RunPod
had no A40 free the request failed rather than taking a card that would have
served it. Now `gpu_types` is an ordered PREFERENCE and `allow_other_gpus` lets
the search fall through to anything else that still satisfies min_vram_gb and
max_hourly_usd.

A preferred card ranks ahead of every fallback whatever selection_criteria
says — otherwise "cheaper" would put the fallback first and naming a card would
mean nothing. Each ranked option carries `preferred`, and provisioning a card
nobody asked for prints "[FALLBACK — no preferred card available]": correct
behaviour, but never silent.

Off by default. A pinned card is sometimes pinned for a reason, and an install
that quietly started renting something else would be a surprise on the invoice.

Name matching is exact on the id or display name, case-insensitive, tolerating
a missing "NVIDIA " prefix, and deliberately NOT a substring: "A40" would
otherwise match "NVIDIA RTX A4000" — 16 GB for a job that asked for 48. GPU
names split on commas only, because "NVIDIA RTX A6000" has spaces in it.

Also: the admin handler rebuilds the runpod block from a whitelist, so all six
keys added today (global_volume_id, data_centers, gpu_types, allow_other_gpus,
allow_client_scale, client_scale_ttl_s) were being dropped the next time anyone
saved a model from the GUI — taking the warm schedule and the volume with them.
They are in the whitelist now, with the two ordered lists kept in order.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.0"
+__version__ = "0.3.1"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -3105,7 +3105,9 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                   "network_volume_id", "volume_mount_path", "volume_path",
                   "venv_name", "slim_image",
                   # region + warm-pod schedule
-                  "data_center", "schedule_start", "schedule_end", "schedule_tz"):
+                  "data_center", "schedule_start", "schedule_end", "schedule_tz",
+                  # a GLOBAL (region-independent) volume
+                  "global_volume_id"):
             v = src.get(k)
             if isinstance(v, str) and v.strip():
                 rpo[k] = v.strip()
@@ -3119,6 +3121,20 @@ async def api_model_configure(request: Request, username: str = Depends(require_
             ct = None
         if ct:
             rpo["cloud_types"] = ct
+        # Ordered lists: regions to try, and preferred GPUs. Order IS meaning
+        # here (it is the fallback order), so neither is sorted or upper-cased
+        # beyond what the region ids need. GPU names split on commas only —
+        # "NVIDIA RTX A6000" has spaces in it.
+        dcs = src.get("data_centers")
+        if isinstance(dcs, str):
+            dcs = [d.strip() for d in dcs.replace(",", " ").split() if d.strip()]
+        if isinstance(dcs, list) and dcs:
+            rpo["data_centers"] = [str(d).strip().upper() for d in dcs if str(d).strip()]
+        gts = src.get("gpu_types")
+        if isinstance(gts, str):
+            gts = [g.strip() for g in gts.split(",") if g.strip()]
+        if isinstance(gts, list) and gts:
+            rpo["gpu_types"] = [str(g).strip() for g in gts if str(g).strip()]
         # schedule_days: list of 0=Mon…6=Sun, or a comma string of those/names
         sd = src.get("schedule_days")
         if isinstance(sd, str):
@@ -3127,7 +3143,10 @@ async def api_model_configure(request: Request, username: str = Depends(require_
             rpo["schedule_days"] = [str(d).strip().lower() for d in sd if str(d).strip()]
         # bools
         for k in ("allow_spot", "allow_open_pod", "sticky_sessions", "keep_warm",
-                  "model_url_is_tar", "venv_on_volume", "schedule_enabled"):
+                  "model_url_is_tar", "venv_on_volume", "schedule_enabled",
+                  # rent a different card when no preferred one is free, and let
+                  # a client lease warm pods
+                  "allow_other_gpus", "allow_client_scale"):
             if k in src and src.get(k) is not None and src.get(k) != "":
                 sv = src.get(k)
                 rpo[k] = (sv.lower() in ("1", "true", "on", "yes")
@@ -3139,7 +3158,7 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                   # how hard the POD's own server may be driven
                   "pod_max_parallel_requests", "pod_queue_max_size",
                   "boot_timeout_s", "load_timeout_s", "min_workers", "max_workers",
-                  "gpu_count"):
+                  "gpu_count", "client_scale_ttl_s"):
             v = src.get(k)
             if v not in (None, ""):
                 try:

codai/api/runpod_worker.py

@@ -42,6 +42,18 @@ class RunpodModelConfig:
     cloud_types: list = field(default_factory=lambda: ["SECURE"])   # allowed pools
     selection_criteria: str = "cheaper"      # cheaper | faster
     gpu_type: str = ""                       # explicit RunPod gpuTypeId (optional)
+    # Ordered PREFERENCE instead of a single card: tried best-first, and with
+    # allow_other_gpus the search falls through to anything else that fits.
+    # `gpu_type` above is folded in as a single-entry preference.
+    gpu_types: list = field(default_factory=list)
+    # When every preferred card is unavailable, may we rent a different one that
+    # still satisfies min_vram_gb and max_hourly_usd? OFF by default: a pinned
+    # card is sometimes pinned for a reason (an architecture a kernel needs, or
+    # a price), and silently renting something else would be a surprise. ON is
+    # what you want for "serve the request, A40 or not" — and then the VRAM
+    # floor and the $/hr ceiling are the real constraints, so check that
+    # something actually fits under them.
+    allow_other_gpus: bool = False
     min_vram_gb: float = 0.0
     max_hourly_usd: float = 0.0              # per-model $/hr GPU ceiling (0 = none)
     # GPUs per pod. RunPod rents multi-GPU pods, and a 154 GB model on one 80 GB
@@ -1366,6 +1378,43 @@ def _as_int(v, default=0):
 _DAY_NAMES = ("mon", "tue", "wed", "thu", "fri", "sat", "sun")
 
 
+def _parse_name_list(value) -> list:
+    """An ordered list of GPU names from a list or a comma string.
+
+    Comma only, not whitespace: a GPU display name has spaces in it ("NVIDIA
+    RTX A6000"), so splitting on those would turn one card into three.
+    """
+    if value in (None, ""):
+        return []
+    items = value if isinstance(value, (list, tuple)) else str(value).split(",")
+    out = []
+    for raw in items:
+        tok = str(raw).strip()
+        if tok and tok not in out:
+            out.append(tok)
+    return out
+
+
+def _gpu_matches(gpu: dict, want: str) -> bool:
+    """Whether a catalogue entry is the card `want` names.
+
+    Exact on the id or the display name, case-insensitively, and tolerant of a
+    missing "NVIDIA " prefix so "A40" finds "NVIDIA A40". NOT a substring match:
+    "A40" would otherwise also match "NVIDIA RTX A4000", which is a 16 GB card
+    where someone asked for a 48 GB one.
+    """
+    w = (want or "").strip().lower()
+    if not w:
+        return False
+    for field_ in (gpu.get("id"), gpu.get("display_name")):
+        name = str(field_ or "").strip().lower()
+        if not name:
+            continue
+        if name == w or name.removeprefix("nvidia ") == w.removeprefix("nvidia "):
+            return True
+    return False
+
+
 def _parse_dc_list(value) -> list:
     """A region allow-list from a list or a comma/space string, order kept.
 
@@ -1530,6 +1579,8 @@ def parse_model_runpod(block: Optional[dict]) -> RunpodModelConfig:
     cfg.network_volume_id = (b.get("network_volume_id") or "").strip()
     cfg.global_volume_id = (b.get("global_volume_id") or "").strip()
     cfg.data_centers = _parse_dc_list(b.get("data_centers"))
+    cfg.gpu_types = _parse_name_list(b.get("gpu_types"))
+    cfg.allow_other_gpus = _as_bool(b.get("allow_other_gpus"), cfg.allow_other_gpus)
     if cfg.network_volume_id:
         # RunPod offers network volumes on Secure Cloud only; leaving COMMUNITY
         # in the list just produces candidates the attach would reject.
@@ -1691,11 +1742,24 @@ def _rank_gpus(client, mcfg: "RunpodModelConfig", account_cfg) -> list:
     # holds N times the VRAM, so both the ceiling and the VRAM floor are checked
     # against the pod, not the card — or a 4-GPU pod would pass a $1/hr ceiling
     # at $0.90 a card and bill $3.60.
-    explicit = (mcfg.gpu_type or getattr(account_cfg, "default_gpu_type", "") or "").strip()
-
-    cands = []   # (price, is_spot, cloud_type, gpu, on_demand_price)
+    # Preference, not a filter. The old single `gpu_type` was a hard filter, so
+    # "A40" meant "A40 or nothing" — and when RunPod had no A40 the request
+    # failed instead of taking a card that fit just as well.
+    prefer = ([p for p in (getattr(mcfg, "gpu_types", None) or []) if str(p).strip()]
+              or ([mcfg.gpu_type] if mcfg.gpu_type.strip() else [])
+              or ([getattr(account_cfg, "default_gpu_type", "")]
+                  if str(getattr(account_cfg, "default_gpu_type", "") or "").strip()
+                  else []))
+    allow_other = bool(getattr(mcfg, "allow_other_gpus", False))
+
+    cands = []   # (pref_rank, price, is_spot, cloud_type, gpu, on_demand_price)
     for g in catalog:
-        if explicit and g["id"] != explicit and g["display_name"] != explicit:
+        rank = len(prefer)
+        for idx_p, want in enumerate(prefer):
+            if _gpu_matches(g, want):
+                rank = idx_p
+                break
+        if prefer and rank == len(prefer) and not allow_other:
             continue
         mem = (g.get("memory_gb") or 0) * n
         if mcfg.min_vram_gb and mem < mcfg.min_vram_gb:
@@ -1712,26 +1776,31 @@ def _rank_gpus(client, mcfg: "RunpodModelConfig", account_cfg) -> list:
                 price = price * n
                 if ceiling > 0 and price > ceiling:
                     continue
-                cands.append((price, is_spot, ct, g, (on_demand or price) * n))
+                cands.append((rank, price, is_spot, ct, g, (on_demand or price) * n))
     if not cands:
         raise RunpodError(
             f"RunPod: no GPU in pools {allowed} with ≥{mcfg.min_vram_gb}GB under "
             f"${ceiling}/hr (spot={'on' if mcfg.allow_spot else 'off'}"
-            + (f", gpu={explicit}" if explicit else "") + ").")
+            + (f", preferred={prefer}" if prefer else "")
+            + (", allow_other_gpus=off — set it to let another card that fits "
+               "serve the request" if prefer and not allow_other else "") + ").")
 
     if (mcfg.selection_criteria or "cheaper").lower() == "faster":
         # Reliability + throughput: SECURE before COMMUNITY, on-demand before spot,
         # more VRAM (throughput proxy) first, then cheapest.
         ct_rank = {"SECURE": 0, "COMMUNITY": 1}
-        cands.sort(key=lambda c: (ct_rank.get(c[2], 9), c[1] is True,
-                                  -(c[3].get("memory_gb") or 0), c[0]))
+        cands.sort(key=lambda c: (c[0], ct_rank.get(c[3], 9), c[2] is True,
+                                  -(c[4].get("memory_gb") or 0), c[1]))
     else:  # cheaper
-        cands.sort(key=lambda c: (c[0], c[1] is False))
+        cands.sort(key=lambda c: (c[0], c[1], c[2] is False))
+    # A preferred card always sorts ahead of a fallback, whatever the criteria:
+    # the point of naming one is to get it when it is there.
     return [{"gpu_type_id": g["id"],
              "display_name": (f"{n}x {g['display_name']}" if n > 1 else g["display_name"]),
              "memory_gb": (g.get("memory_gb") or 0) * n, "cloud_type": ct, "price": price,
-             "is_spot": is_spot, "bid": on_demand if is_spot else 0.0, "gpu_count": n}
-            for (price, is_spot, ct, g, on_demand) in cands]
+             "is_spot": is_spot, "bid": on_demand if is_spot else 0.0, "gpu_count": n,
+             "preferred": rank < len(prefer)}
+            for (rank, price, is_spot, ct, g, on_demand) in cands]
 
 
 def _select_gpu(client, mcfg: "RunpodModelConfig", account_cfg) -> dict:
@@ -2158,9 +2227,13 @@ class RunpodPodPool:
                     + str(self.model_key)[:20].replace("/", "_").replace(" ", "_")
                     + "-" + uuid.uuid4().hex[:6])
             _dc = sel.get("_dc") or self._data_center()
+            # Say so when this is not the card that was asked for. Renting a
+            # different GPU than the configuration names is correct here — it is
+            # what allow_other_gpus is for — but it must never be silent.
+            _alt = "" if sel.get("preferred", True) else " [FALLBACK — no preferred card available]"
             print(f"[runpod] provisioning pod for {self.model_key!r}: {sel['display_name']} "
                   f"({sel['cloud_type']}{'/spot' if sel['is_spot'] else ''}) ${sel['price']}/hr"
-                  + (f" in {_dc}" if _dc else " (region: RunPod's choice)"),
+                  + (f" in {_dc}" if _dc else " (region: RunPod's choice)") + _alt,
                   flush=True)
             try:
                 _disk_gb, _disk_note = disk_for(

docs/orchestrator-runpod-guide.md

@@ -608,6 +608,8 @@ Four numbers decide the shape of a pool:
 | `client_scale_ttl_s` | how long such a lease lasts unless renewed (default 1800). A lease that never expired would be a bill that never stopped. |
 | `data_centers` | several acceptable regions, tried in order on a capacity miss — how you say "anywhere in the EU". A single `data_center` pins one region; blank lets RunPod pick any region **on earth**. |
 | `global_volume_id` | a RunPod **global** volume (region-independent). Unlike `network_volume_id` it does NOT pin the pod's region, which is the only reason to use one. |
+| `gpu_types` | ordered GPU **preference**, best first (comma-separated; names have spaces, so commas only). |
+| `allow_other_gpus` | when no preferred card is free, rent another that still fits `min_vram_gb` and `max_hourly_usd`. Off by default. |
 
 **Then the two that are easy to miss**, and the reason a pod can look slow while
 its GPU is idle:
@@ -697,6 +699,44 @@ Caveats worth knowing before you rely on it:
 
 ---
 
+### When the card you asked for is not available
+
+`gpu_type` is a **filter**, not a preference: `"NVIDIA A40"` has always meant
+"an A40 or nothing", so when RunPod has no A40 free the request fails rather
+than taking a card that would have served it just as well.
+
+To let it fall through:
+
+```json
+"gpu_types": "NVIDIA A40,NVIDIA RTX A6000",
+"allow_other_gpus": true,
+"min_vram_gb": 40,
+"max_hourly_usd": 1.20
+```
+
+* `gpu_types` is an ordered **preference**. A preferred card always ranks ahead
+  of a fallback, whatever `selection_criteria` says — the point of naming one is
+  to get it when it is there.
+* `allow_other_gpus` is what permits anything else. It is **off by default**: a
+  pinned card is sometimes pinned for a reason, and quietly renting a different
+  one would be a surprise on the invoice.
+* **`min_vram_gb` and `max_hourly_usd` are the real constraints** once the
+  fallback is on, and this is the part that catches people. With
+  `max_hourly_usd: 0.60`, an A40 at $0.59 fits and nothing else does — so the
+  fallback finds nothing and you are back where you started. Raise the ceiling
+  to the most you are willing to pay for the *alternative*, and keep
+  `min_vram_gb` honest so a cheap 16 GB card cannot win a job that needs 48.
+* A fallback is never silent. The provisioning line says
+  `[FALLBACK — no preferred card available]`, and each ranked option carries a
+  `preferred` flag.
+
+Name matching is exact on the id or display name, case-insensitive, and
+tolerates a missing `NVIDIA ` prefix — so `A40` finds `NVIDIA A40`. It is
+deliberately **not** a substring match: `A40` would otherwise also match
+`NVIDIA RTX A4000`, a 16 GB card, for a job that asked for 48.
+
+---
+
 ### Let a client ask for pods
 
 The autoscaler reacts to load: it adds a pod once the least-loaded one already