know which card, size per card, and never load a remote model here (v0.4.13)
Three things, and the third is the reason the first two were impossible.
**A mixed fleet was distributed as if every pod were the same.**
`PodPool._pick` was `min(free, key=inflight)` and growth was
`load / scale_up_inflight_per_pod` -- both card-blind. An A100 and an A40 were
handed the same number of pages, so the fast card drained its share and idled
while the slow one straggled; and a document completes only when its LAST page
lands, so the slowest pod set the latency the client saw. The scaler had the
matching fault in reverse: a fleet of bigger cards was still sized as if each
were the preferred one, which is the same arithmetic that reached 94 pods at
10 % utilisation.
Now: weighted least-connections, an in-flight cap scaled per pod, and growth
that divides demand by the fleet's real capacity.
**`gpu_limits`: per-card numbers, nothing multiplied.**
"gpu_limits": {
"NVIDIA A100-SXM4-80GB": {"parallel": 186, "queue": 744, "scale_up": 93},
"NVIDIA A40": {"parallel": 64, "queue": 256, "scale_up": 32}
}
A scaled single number cannot stay right: the model-level figures describe ONE
card, and which card that was came from the first entry of `gpu_types` -- so
changing the RENTAL preference silently re-scaled the concurrency, and a figure
measured on an A40 came to describe an A100 by a factor nobody had measured. A
table states what was measured, per card. The share of load is DERIVED from it
(two cards with depths 32 and 93 are given work in that ratio) rather than
configured twice, with `gpu_weights` and then the VRAM ratio as fallbacks for
cards nobody has measured yet. Editable per model in the admin UI; a row with a
card name and no numbers is dropped, because "not measured yet" must not be
saved as a limit of zero.
`parallel`/`queue` are applied AFTER the candidate card is chosen. The launch
plan is built before, so its env carried the model-level number whatever card
won -- and the gate is env, set once, so a pod rented on a 3x card spent its
whole life admitting a 1x card's work.
**An adopted pod recorded `gpu="(adopted)"`.** A pool that cannot name a pod's
card cannot size it, weight it, or report the fleet: a whole day's status output
read `by_gpu: {"(adopted)": 32}`. `list_pods`/`get_pod` now ask RunPod for
`gpuCount` and `machine { gpuDisplayName gpuTypeId gpuType { memoryInGb } }`
(every field probed against the live schema first), the registry stores the
card and its size so adoption needs no API call, and an unreadable card costs a
weight of 1.0, never a pod.
**And a catalogue entry could crash an engine.** Adding a `vision_models` row
for a model served on RunPod quarantined the cpu engine for ninety minutes:
its id is not a path, not a URL and not a repo id, so resolution fell through
to "assume a repo id", then to "assume a URL", and
`requests.get("qwen38-vision")` raised MissingSchema -- five exits in 120 s,
QUARANTINED, taking /v1/ocr and every pool with it while `/healthz` still
answered `ok: true` because the FRONT was up. Two independent guards:
* a model pinned to a backend that serves it ELSEWHERE is never resolved,
downloaded or cached here -- which is also what lets one row run on rented
pods and load locally on a box with a GPU, the pin being the whole
difference and the section not overriding it;
* a name that is none of the three resolvable kinds is refused with a sentence
naming all three and pointing at `backend`.
The pin does NOT make local weights useless: `resolve_model_source` is
configurable (`auto | hf | url | upload`) and reads the entry directly, so an
id still means "the pod downloads it" and a local path still means "push it".
Tested, so a later change cannot couple the two.
43 new tests. 3411 passed, 13 pre-existing failures (broker, studio, manual
client) untouched.
Co-Authored-By: 
Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
Showing
Please
register
or
sign in
to comment