-
Stefy Lanza (nextime / spora ) authored
Three things, and the third is the reason the first two were impossible. **A mixed fleet was distributed as if every pod were the same.** `PodPool._pick` was `min(free, key=inflight)` and growth was `load / scale_up_inflight_per_pod` -- both card-blind. An A100 and an A40 were handed the same number of pages, so the fast card drained its share and idled while the slow one straggled; and a document completes only when its LAST page lands, so the slowest pod set the latency the client saw. The scaler had the matching fault in reverse: a fleet of bigger cards was still sized as if each were the preferred one, which is the same arithmetic that reached 94 pods at 10 % utilisation. Now: weighted least-connections, an in-flight cap scaled per pod, and growth that divides demand by the fleet's real capacity. **`gpu_limits`: per-card numbers, nothing multiplied.** "gpu_limits": { "NVIDIA A100-SXM4-80GB": {"parallel": 186, "queue": 744, "scale_up": 93}, "NVIDIA A40": {"parallel": 64, "queue": 256, "scale_up": 32} } A scaled single number cannot stay right: the model-level figures describe ONE card, and which card that was came from the first entry of `gpu_types` -- so changing the RENTAL preference silently re-scaled the concurrency, and a figure measured on an A40 came to describe an A100 by a factor nobody had measured. A table states what was measured, per card. The share of load is DERIVED from it (two cards with depths 32 and 93 are given work in that ratio) rather than configured twice, with `gpu_weights` and then the VRAM ratio as fallbacks for cards nobody has measured yet. Editable per model in the admin UI; a row with a card name and no numbers is dropped, because "not measured yet" must not be saved as a limit of zero. `parallel`/`queue` are applied AFTER the candidate card is chosen. The launch plan is built before, so its env carried the model-level number whatever card won -- and the gate is env, set once, so a pod rented on a 3x card spent its whole life admitting a 1x card's work. **An adopted pod recorded `gpu="(adopted)"`.** A pool that cannot name a pod's card cannot size it, weight it, or report the fleet: a whole day's status output read `by_gpu: {"(adopted)": 32}`. `list_pods`/`get_pod` now ask RunPod for `gpuCount` and `machine { gpuDisplayName gpuTypeId gpuType { memoryInGb } }` (every field probed against the live schema first), the registry stores the card and its size so adoption needs no API call, and an unreadable card costs a weight of 1.0, never a pod. **And a catalogue entry could crash an engine.** Adding a `vision_models` row for a model served on RunPod quarantined the cpu engine for ninety minutes: its id is not a path, not a URL and not a repo id, so resolution fell through to "assume a repo id", then to "assume a URL", and `requests.get("qwen38-vision")` raised MissingSchema -- five exits in 120 s, QUARANTINED, taking /v1/ocr and every pool with it while `/healthz` still answered `ok: true` because the FRONT was up. Two independent guards: * a model pinned to a backend that serves it ELSEWHERE is never resolved, downloaded or cached here -- which is also what lets one row run on rented pods and load locally on a box with a GPU, the pin being the whole difference and the section not overriding it; * a name that is none of the three resolvable kinds is refused with a sentence naming all three and pointing at `backend`. The pin does NOT make local weights useless: `resolve_model_source` is configurable (`auto | hf | url | upload`) and reads the entry directly, so an id still means "the pod downloads it" and a local path still means "push it". Tested, so a later change cannot couple the two. 43 new tests. 3411 passed, 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
51ffc4e1