know which card, size per card, and never load a remote model here (v0.4.13)

Three things, and the third is the reason the first two were impossible.

**A mixed fleet was distributed as if every pod were the same.**
`PodPool._pick` was `min(free, key=inflight)` and growth was
`load / scale_up_inflight_per_pod` -- both card-blind. An A100 and an A40 were
handed the same number of pages, so the fast card drained its share and idled
while the slow one straggled; and a document completes only when its LAST page
lands, so the slowest pod set the latency the client saw. The scaler had the
matching fault in reverse: a fleet of bigger cards was still sized as if each
were the preferred one, which is the same arithmetic that reached 94 pods at
10 % utilisation.

Now: weighted least-connections, an in-flight cap scaled per pod, and growth
that divides demand by the fleet's real capacity.

**`gpu_limits`: per-card numbers, nothing multiplied.**

    "gpu_limits": {
      "NVIDIA A100-SXM4-80GB": {"parallel": 186, "queue": 744, "scale_up": 93},
      "NVIDIA A40":            {"parallel": 64,  "queue": 256, "scale_up": 32}
    }

A scaled single number cannot stay right: the model-level figures describe ONE
card, and which card that was came from the first entry of `gpu_types` -- so
changing the RENTAL preference silently re-scaled the concurrency, and a figure
measured on an A40 came to describe an A100 by a factor nobody had measured. A
table states what was measured, per card. The share of load is DERIVED from it
(two cards with depths 32 and 93 are given work in that ratio) rather than
configured twice, with `gpu_weights` and then the VRAM ratio as fallbacks for
cards nobody has measured yet. Editable per model in the admin UI; a row with a
card name and no numbers is dropped, because "not measured yet" must not be
saved as a limit of zero.

`parallel`/`queue` are applied AFTER the candidate card is chosen. The launch
plan is built before, so its env carried the model-level number whatever card
won -- and the gate is env, set once, so a pod rented on a 3x card spent its
whole life admitting a 1x card's work.

**An adopted pod recorded `gpu="(adopted)"`.** A pool that cannot name a pod's
card cannot size it, weight it, or report the fleet: a whole day's status output
read `by_gpu: {"(adopted)": 32}`. `list_pods`/`get_pod` now ask RunPod for
`gpuCount` and `machine { gpuDisplayName gpuTypeId gpuType { memoryInGb } }`
(every field probed against the live schema first), the registry stores the
card and its size so adoption needs no API call, and an unreadable card costs a
weight of 1.0, never a pod.

**And a catalogue entry could crash an engine.** Adding a `vision_models` row
for a model served on RunPod quarantined the cpu engine for ninety minutes:
its id is not a path, not a URL and not a repo id, so resolution fell through
to "assume a repo id", then to "assume a URL", and
`requests.get("qwen38-vision")` raised MissingSchema -- five exits in 120 s,
QUARANTINED, taking /v1/ocr and every pool with it while `/healthz` still
answered `ok: true` because the FRONT was up. Two independent guards:

* a model pinned to a backend that serves it ELSEWHERE is never resolved,
  downloaded or cached here -- which is also what lets one row run on rented
  pods and load locally on a box with a GPU, the pin being the whole
  difference and the section not overriding it;
* a name that is none of the three resolvable kinds is refused with a sentence
  naming all three and pointing at `backend`.

The pin does NOT make local weights useless: `resolve_model_source` is
configurable (`auto | hf | url | upload`) and reads the entry directly, so an
id still means "the pod downloads it" and a local path still means "push it".
Tested, so a later change cannot couple the two.

43 new tests. 3411 passed, 13 pre-existing failures (broker, studio, manual
client) untouched.
Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
parent 6f74f917
......@@ -16,7 +16,7 @@
# Canonical product version for CoderAI — single source of truth. Both the API
# metadata and the admin web UI read from here.
__version__ = "0.4.12"
__version__ = "0.4.13"
__tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
# Configure the CUDA caching allocator BEFORE torch is imported anywhere.
......
......@@ -3202,6 +3202,31 @@ async def api_model_configure(request: Request, username: str = Depends(require_
env = src.get("env")
if isinstance(env, dict) and env:
rpo["env"] = {str(k): str(v) for k, v in env.items()}
# PER-CARD limits. A mixed fleet needs a number per card rather than one
# number scaled: a figure measured on one GPU does not describe another.
# A row carrying a card name but no numbers is DROPPED -- "not measured
# yet" must not be saved as a limit of zero.
gl = src.get("gpu_limits")
if isinstance(gl, dict):
out = {}
for card, lim in gl.items():
name = str(card or "").strip()
if not name or not isinstance(lim, dict):
continue
row = {}
for key in ("parallel", "queue", "scale_up", "inflight"):
if lim.get(key) in (None, ""):
continue
try:
v = int(lim.get(key))
except (TypeError, ValueError):
continue
if v >= 0:
row[key] = v
if row:
out[name] = row
if out:
rpo["gpu_limits"] = out
if rpo:
entry["runpod"] = rpo
else:
......
......@@ -963,7 +963,24 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
</div>
</div>
<div class="form-row" style="margin:.25rem 0 0">
<span class="form-hint" style="font-size:11px">How many requests the pod's own server admits at once. The defaults are sized for a shared workstation and will throttle a dedicated card long before its GPU is busy — everything over the limit gets a 429. Set these to roughly the remote engine's batch width (e.g. vLLM's <code>max_num_seqs</code>).</span>
<span class="form-hint" style="font-size:11px">How many requests the pod's own server admits at once. The defaults are sized for a shared workstation and will throttle a dedicated card long before its GPU is busy — everything over the limit gets a 429. Set these to roughly the remote engine's batch width (e.g. vLLM's <code>max_num_seqs</code>). The numbers above apply to any card without a row in the table below.</span>
</div>
<div class="form-row" style="margin:.5rem 0 0">
<label class="form-label">Per-card limits <span style="font-weight:400;opacity:.7">(one row per GPU)</span></label>
<table style="width:100%;font-size:12px;border-collapse:collapse" id="cfg-rp-gpu-limits-table">
<thead><tr style="text-align:left;opacity:.75">
<th style="padding:.15rem .25rem">GPU</th>
<th style="padding:.15rem .25rem">parallel</th>
<th style="padding:.15rem .25rem">queue</th>
<th style="padding:.15rem .25rem">scale&nbsp;up at</th>
<th style="padding:.15rem .25rem">in&#8209;flight cap</th>
<th></th>
</tr></thead>
<tbody id="cfg-rp-gpu-limits-rows"></tbody>
</table>
<button type="button" class="btn btn-secondary" style="margin-top:.35rem;font-size:12px"
onclick="rpGpuLimitAddRow()">+ add a card</button>
<span class="form-hint" style="font-size:11px">A mixed fleet needs a number per card, not one number scaled: a figure measured on one GPU does not describe another, and leaving it to be multiplied means changing the rental preference silently re-scales it. <b>Leave a card out until you have measured it</b> — it then falls back to the numbers above (or to its VRAM ratio). <code>scale up at</code> is the in-flight depth at which that card justifies renting another pod, and it also decides that card&rsquo;s share of the load: two cards with depths 32 and 93 are given work in that ratio.</span>
</div>
<div class="form-row" style="margin:.5rem 0 0">
<label class="form-label">Region (data centre)</label>
......@@ -4937,6 +4954,7 @@ function _populateRunpod(s){
set('cfg-rp-schedule-tz', rp.schedule_tz || '');
set('cfg-rp-data-center', rp.data_center || '');
set('cfg-rp-pod-parallel', rp.pod_max_parallel_requests != null ? rp.pod_max_parallel_requests : '');
rpGpuLimitsLoad(rp.gpu_limits || {});
set('cfg-rp-pod-queue', rp.pod_queue_max_size != null ? rp.pod_queue_max_size : '');
(function(){
// Days come back normalised to 0=Mon…6=Sun; an empty list means every day,
......@@ -5156,6 +5174,7 @@ function _collectRunpod(){
.map(function(cb){ return cb.value; }),
data_center: (val('cfg-rp-data-center') || '').trim(),
pod_max_parallel_requests: intOrU('cfg-rp-pod-parallel'),
gpu_limits: rpGpuLimitsCollect(),
pod_queue_max_size: intOrU('cfg-rp-pod-queue'),
};
return o;
......@@ -5522,5 +5541,50 @@ async function testRunModel(){
}
}
function rpGpuLimitAddRow(card, lim) {
const body = document.getElementById('cfg-rp-gpu-limits-rows');
if (!body) return;
lim = lim || {};
const tr = document.createElement('tr');
const cell = (html) => { const td = document.createElement('td');
td.style.padding = '.1rem .25rem'; td.innerHTML = html; return td; };
tr.appendChild(cell('<input type="text" class="form-input rp-gl-card" style="width:100%;font-size:12px"'
+ ' placeholder="NVIDIA A40" value="' + (card || '').replace(/"/g, '&quot;') + '">'));
for (const k of ['parallel', 'queue', 'scale_up', 'inflight']) {
tr.appendChild(cell('<input type="number" min="0" class="form-input rp-gl-' + k
+ '" style="width:100%;font-size:12px" value="'
+ (lim[k] != null ? lim[k] : '') + '">'));
}
tr.appendChild(cell('<button type="button" class="btn btn-secondary" style="font-size:11px"'
+ ' onclick="this.closest(\'tr\').remove()">remove</button>'));
body.appendChild(tr);
}
function rpGpuLimitsLoad(limits) {
const body = document.getElementById('cfg-rp-gpu-limits-rows');
if (!body) return;
body.innerHTML = '';
Object.keys(limits || {}).sort().forEach(card => rpGpuLimitAddRow(card, limits[card]));
}
function rpGpuLimitsCollect() {
// A row with a card name but no numbers is DROPPED rather than saved as a
// row of zeros: an empty row means "not measured yet", and a zero would be
// read as a real limit.
const out = {};
document.querySelectorAll('#cfg-rp-gpu-limits-rows tr').forEach(tr => {
const card = (tr.querySelector('.rp-gl-card') || {}).value || '';
if (!card.trim()) return;
const row = {};
for (const k of ['parallel', 'queue', 'scale_up', 'inflight']) {
const el = tr.querySelector('.rp-gl-' + k);
const v = el && el.value !== '' ? parseInt(el.value, 10) : null;
if (v != null && !isNaN(v) && v >= 0) row[k] = v;
}
if (Object.keys(row).length) out[card.trim()] = row;
});
return out;
}
</script>
{% endblock %}
......@@ -133,6 +133,45 @@ def pod_console_url(pod_id: str) -> str:
return f"https://www.runpod.io/console/pods/{pod_id}"
def _machine_fields(p: dict) -> dict:
"""``{gpu, gpu_id, gpu_count, vram_gb}`` from a pod's machine, defensively.
RunPod reports the card on the MACHINE, and a pod whose machine record is
missing (one still provisioning, one being torn down) must not break a
listing the reaper depends on -- so every field degrades to a blank rather
than raising.
"""
m = (p.get("machine") or {}) if isinstance(p, dict) else {}
gt = (m.get("gpuType") or {}) if isinstance(m, dict) else {}
try:
n = max(1, int(p.get("gpuCount") or 1))
except (TypeError, ValueError):
n = 1
try:
per_card = float(gt.get("memoryInGb") or 0.0)
except (TypeError, ValueError):
per_card = 0.0
return {"gpu": str(m.get("gpuDisplayName") or ""),
"gpu_id": str(m.get("gpuTypeId") or ""),
"gpu_count": n,
# The pod's TOTAL VRAM: a 2-card pod holds twice the KV cache.
"vram_gb": per_card * n}
def get_pod_machine(client, pod_id: str) -> dict:
"""``{gpu, gpu_id, gpu_count, vram_gb}`` for one pod, or blanks.
Used when ADOPTING a pod this process did not rent: the registry may know
the card (we recorded it when we rented it), and when it does not, the only
other source is the account.
"""
try:
return {k: v for k, v in (client.get_pod(pod_id) or {}).items()
if k in ("gpu", "gpu_id", "gpu_count", "vram_gb")}
except Exception:
return {}
class RunpodClient:
def __init__(self, cfg):
self._cfg = cfg
......@@ -456,6 +495,8 @@ class RunpodClient:
desiredStatus
lastStatusChange
costPerHr
gpuCount
machine { gpuDisplayName gpuTypeId gpuType { memoryInGb } }
runtime {
uptimeInSeconds
ports { ip isIpPublic privatePort publicPort type }
......@@ -467,6 +508,7 @@ class RunpodClient:
rt = p.get("runtime") or {}
ports = rt.get("ports") or []
return {
**_machine_fields(p),
"id": p.get("id") or pod_id,
"status": p.get("desiredStatus"),
# The one field that says WHY a pod is EXITED. It reads like
......@@ -531,6 +573,8 @@ class RunpodClient:
query Pods {
myself {
pods { id name desiredStatus costPerHr imageName lastStatusChange
gpuCount
machine { gpuDisplayName gpuTypeId gpuType { memoryInGb } }
runtime { uptimeInSeconds } }
}
}"""
......@@ -542,6 +586,11 @@ class RunpodClient:
"id": p.get("id"), "name": p.get("name"),
"status": p.get("desiredStatus"), "cost_per_hr": p.get("costPerHr"),
"image": p.get("imageName"),
# WHICH CARD. A pool that cannot name a pod's card cannot size
# its concurrency, weight its share of the load, or report the
# fleet honestly -- adopted pods were listed as "(adopted)" and
# counted as one anonymous kind.
**_machine_fields(p),
"last_change": p.get("lastStatusChange") or "",
}
# Absent while a pod is still being scheduled, which is exactly when
......
This diff is collapsed.
......@@ -255,6 +255,23 @@ def download_model(model_path: str, cache_dir: Optional[str] = None, file_patter
Returns:
Path to downloaded model, or None on failure
"""
# NEITHER a URL NOR a repo id: say so, rather than handing a bare name to
# requests.get and letting it raise MissingSchema from four frames deep.
#
# That is not a cosmetic difference. A `vision_models` entry named
# `qwen38-vision` (no slash, so not a repo id; no scheme, so not a URL)
# reached here at engine startup, requests raised, and the engine
# crash-looped five times in 120 s and was QUARANTINED -- taking /v1/ocr
# and every RunPod pool with it for ninety minutes. A name nobody can
# resolve is a configuration mistake, and it must read like one.
if not is_huggingface_model_id(model_path) and not str(model_path).startswith(
("http://", "https://")):
print(f"[cache] cannot fetch {model_path!r}: it is not a local path, not "
f"a URL (no http:// or https://), and not a HuggingFace repo id "
f"(no 'owner/name'). Nothing says where these weights are. If this "
f"model is served remotely, pin it with `backend` so nothing tries "
f"to load it here.", flush=True)
return None
# Check if it's a HuggingFace model ID
if is_huggingface_model_id(model_path):
# Use HF cache if no specific cache_dir provided
......
......@@ -303,6 +303,12 @@ def _env_seed_entries() -> list:
_SEED_CACHE = None
#: Backends that serve a model somewhere else. A model pinned to one of these
#: has no local weights to resolve, so nothing here should download, cache or
#: path-resolve it -- on ANY install, whichever section lists it.
_REMOTE_BACKENDS = ("runpod", "remote", "node", "cluster")
def _model_entry_for(model_name: str):
"""The models.json entry (dict) matching ``model_name`` by path / basename /
alias / id, or None."""
......@@ -2991,6 +2997,29 @@ class MultiModelManager:
"""
from codai.models.cache import is_huggingface_model_id
# 0. A model pinned to a REMOTE backend has no local weights, by
# definition. Resolving one here downloads something this install will
# never run, and for a name that is not a repo id it used to reach
# requests.get and raise -- which crash-looped an engine into
# quarantine over a catalogue entry (see codai/models/cache for the
# incident).
#
# This is also what lets ONE catalogue entry serve both kinds of
# install: the same `vision_models` row runs on rented pods where it is
# pinned `backend: runpod`, and loads locally on a box with a GPU where
# it is not. The pin is the whole difference, and the section a model
# is listed in must not override it.
pin = ""
try:
entry = _model_entry_for(model_path) or {}
pin = str(entry.get("backend") or "").strip().lower()
except Exception:
pin = ""
if pin in _REMOTE_BACKENDS:
print(f"[models] {model_path!r} is pinned to {pin!r}, which serves it "
f"elsewhere — not resolving local weights for it", flush=True)
return model_path
# 1. Check if it's a local file
if os.path.isfile(model_path):
print(f"Using local model: {model_path}")
......
"""A catalogue entry must not be able to crash an engine.
The incident, 2026-10-16 on the Digesta orchestrator: a `vision_models` row was
added for a model served on RunPod, pinned `backend: runpod`. At engine startup
something resolved its path; the id (`qwen38-vision`) is not a local file, not a
URL and not a HuggingFace repo id, so the resolver fell through to "assume it's
a repo id", `download_model` fell through to "then it must be a URL", and
`requests.get("qwen38-vision")` raised MissingSchema. The engine crash-looped
five times in 120 s and was QUARANTINED, taking /v1/ocr and every RunPod pool
down with it for about ninety minutes. `/healthz` still answered `ok: true`,
because the FRONT was up.
Two independent guards, because either alone would have prevented it:
* a model pinned to a backend that serves it elsewhere has no local weights, so
nothing resolves, downloads or caches one -- which is also what lets ONE
catalogue row run on rented pods here and load locally on a box with a GPU;
* a name that is none of the three resolvable kinds is refused with a sentence
naming all three, not with a stack trace from inside requests.
"""
import types
import pytest
@pytest.fixture
def catalogue(monkeypatch):
import codai.admin.routes as routes
def _set(md):
monkeypatch.setattr(routes, "config_manager",
types.SimpleNamespace(models_data=md), raising=False)
return _set
# ------------------------------------------------- the pin wins over loading #
@pytest.mark.parametrize("pin", ["runpod", "remote", "node", "cluster", "RunPod"])
def test_a_remote_pinned_model_is_never_resolved_locally(catalogue, pin):
from codai.models.manager import multi_model_manager
catalogue({"vision_models": [{"id": "qwen38-vision", "backend": pin,
"runpod": {"hf_repo": "owner/Qwen"}}]})
assert multi_model_manager.load_model("qwen38-vision") == "qwen38-vision"
def test_the_same_row_WOULD_load_locally_without_the_pin(catalogue, monkeypatch):
"""The point of the pin: one catalogue entry, two kinds of install."""
from codai.models import manager as mm
catalogue({"vision_models": [{"id": "owner/Qwen-VL"}]}) # no backend
assert mm.multi_model_manager.load_model("owner/Qwen-VL") == "owner/Qwen-VL"
def test_local_weights_on_a_pinned_entry_are_still_meaningful(catalogue, tmp_path):
"""The pin stops THIS install loading them. It does not make them useless.
Weights that exist here can still reach a pod: `resolve_model_source` is
configurable (`source: auto | hf | url | upload`) and reads the entry
DIRECTLY -- `entry["path"]`, `mcfg.hf_repo`, `mcfg.model_url` -- so a local
path is what `upload` pushes and what a repo id is recovered from. Neither
goes through the resolver guarded above, and this test exists so a later
change cannot quietly couple them.
"""
from codai.api.runpod_worker import resolve_model_source
import types
from codai.models.manager import multi_model_manager
f = tmp_path / "w.gguf"
f.write_bytes(b"x")
entry = {"id": str(f), "path": str(f), "backend": "runpod"}
catalogue({"text_models": [entry]})
# Nothing is loaded here...
assert multi_model_manager.load_model(str(f)) == str(f)
# ...and every way of getting it onto a pod still works, by configuration.
up = types.SimpleNamespace(source="upload", hf_repo="", model_url="",
served_model="")
assert resolve_model_source(entry, up) == ("upload", str(f))
hf = types.SimpleNamespace(source="hf", hf_repo="owner/name", model_url="",
served_model="")
assert resolve_model_source(entry, hf) == ("hf", "owner/name")
url = types.SimpleNamespace(source="url", hf_repo="",
model_url="https://h/m.gguf", served_model="")
assert resolve_model_source(entry, url) == ("url", "https://h/m.gguf")
def test_an_id_is_recovered_so_the_pod_can_download_it(catalogue):
""""auto": a repo id is often recoverable even when the path is local, and
then the POD downloads it rather than anything being pushed."""
import types
from codai.api.runpod_worker import resolve_model_source
entry = {"id": "x", "path": "/local/only/weights", "backend": "runpod"}
auto = types.SimpleNamespace(source="auto", hf_repo="owner/name",
model_url="", served_model="")
assert resolve_model_source(entry, auto) == ("hf", "owner/name")
def test_an_unknown_model_is_unaffected(catalogue):
from codai.models.manager import multi_model_manager
catalogue({})
# No entry, so no pin: resolution proceeds as it always did (and the name
# below is a repo id, so it resolves to itself without any network).
assert multi_model_manager.load_model("owner/name") == "owner/name"
def test_a_broken_catalogue_cannot_block_loading(monkeypatch):
import codai.admin.routes as routes
from codai.models.manager import multi_model_manager
monkeypatch.setattr(routes, "config_manager", None, raising=False)
assert multi_model_manager.load_model("owner/name") == "owner/name"
# --------------------------------------- an unresolvable name is a sentence #
def test_a_bare_name_is_refused_not_fetched(capsys):
"""`requests.get("qwen38-vision")` is what killed the engine."""
from codai.models.cache import download_model
assert download_model("qwen38-vision") is None
out = capsys.readouterr().out
assert "not a local path" in out and "not a URL" in out
assert "HuggingFace repo id" in out
# ...and it points at the fix rather than only at the symptom.
assert "backend" in out
def test_a_bare_name_never_reaches_requests(monkeypatch):
import requests
from codai.models.cache import download_model
def _boom(*a, **k):
raise AssertionError("requests.get must not be called for a bare name")
monkeypatch.setattr(requests, "get", _boom)
assert download_model("qwen38-vision") is None
def test_a_real_url_still_reaches_requests(monkeypatch):
import requests
from codai.models.cache import download_model
seen = {}
class _Resp:
headers = {"content-length": "0"}
status_code = 200
def raise_for_status(self):
return None
def iter_content(self, chunk_size=0):
return iter(())
def _get(url, **k):
seen["url"] = url
return _Resp()
monkeypatch.setattr(requests, "get", _get)
# A unique URL per run: the cache is keyed by a hash of it, and a hit from
# an earlier run would skip the fetch and make this test pass for the wrong
# reason (it did, once).
import uuid
url = f"https://example.invalid/{uuid.uuid4().hex}.gguf"
download_model(url)
assert seen.get("url") == url
def test_a_repo_id_does_not_take_the_url_path(monkeypatch):
import requests
from codai.models.cache import download_model
def _boom(*a, **k):
raise AssertionError("a repo id must not be fetched as a URL")
monkeypatch.setattr(requests, "get", _boom)
# It will fail to reach HuggingFace in a sandbox; what matters is that it
# never treats 'owner/name' as a URL.
try:
download_model("owner/name")
except Exception:
pass
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment