• Stefy Lanza (nextime / spora )'s avatar
    know which card, size per card, and never load a remote model here (v0.4.13) · 51ffc4e1
    Stefy Lanza (nextime / spora ) authored
    Three things, and the third is the reason the first two were impossible.
    
    **A mixed fleet was distributed as if every pod were the same.**
    `PodPool._pick` was `min(free, key=inflight)` and growth was
    `load / scale_up_inflight_per_pod` -- both card-blind. An A100 and an A40 were
    handed the same number of pages, so the fast card drained its share and idled
    while the slow one straggled; and a document completes only when its LAST page
    lands, so the slowest pod set the latency the client saw. The scaler had the
    matching fault in reverse: a fleet of bigger cards was still sized as if each
    were the preferred one, which is the same arithmetic that reached 94 pods at
    10 % utilisation.
    
    Now: weighted least-connections, an in-flight cap scaled per pod, and growth
    that divides demand by the fleet's real capacity.
    
    **`gpu_limits`: per-card numbers, nothing multiplied.**
    
        "gpu_limits": {
          "NVIDIA A100-SXM4-80GB": {"parallel": 186, "queue": 744, "scale_up": 93},
          "NVIDIA A40":            {"parallel": 64,  "queue": 256, "scale_up": 32}
        }
    
    A scaled single number cannot stay right: the model-level figures describe ONE
    card, and which card that was came from the first entry of `gpu_types` -- so
    changing the RENTAL preference silently re-scaled the concurrency, and a figure
    measured on an A40 came to describe an A100 by a factor nobody had measured. A
    table states what was measured, per card. The share of load is DERIVED from it
    (two cards with depths 32 and 93 are given work in that ratio) rather than
    configured twice, with `gpu_weights` and then the VRAM ratio as fallbacks for
    cards nobody has measured yet. Editable per model in the admin UI; a row with a
    card name and no numbers is dropped, because "not measured yet" must not be
    saved as a limit of zero.
    
    `parallel`/`queue` are applied AFTER the candidate card is chosen. The launch
    plan is built before, so its env carried the model-level number whatever card
    won -- and the gate is env, set once, so a pod rented on a 3x card spent its
    whole life admitting a 1x card's work.
    
    **An adopted pod recorded `gpu="(adopted)"`.** A pool that cannot name a pod's
    card cannot size it, weight it, or report the fleet: a whole day's status output
    read `by_gpu: {"(adopted)": 32}`. `list_pods`/`get_pod` now ask RunPod for
    `gpuCount` and `machine { gpuDisplayName gpuTypeId gpuType { memoryInGb } }`
    (every field probed against the live schema first), the registry stores the
    card and its size so adoption needs no API call, and an unreadable card costs a
    weight of 1.0, never a pod.
    
    **And a catalogue entry could crash an engine.** Adding a `vision_models` row
    for a model served on RunPod quarantined the cpu engine for ninety minutes:
    its id is not a path, not a URL and not a repo id, so resolution fell through
    to "assume a repo id", then to "assume a URL", and
    `requests.get("qwen38-vision")` raised MissingSchema -- five exits in 120 s,
    QUARANTINED, taking /v1/ocr and every pool with it while `/healthz` still
    answered `ok: true` because the FRONT was up. Two independent guards:
    
    * a model pinned to a backend that serves it ELSEWHERE is never resolved,
      downloaded or cached here -- which is also what lets one row run on rented
      pods and load locally on a box with a GPU, the pin being the whole
      difference and the section not overriding it;
    * a name that is none of the three resolvable kinds is refused with a sentence
      naming all three and pointing at `backend`.
    
    The pin does NOT make local weights useless: `resolve_model_source` is
    configurable (`auto | hf | url | upload`) and reads the entry directly, so an
    id still means "the pod downloads it" and a local path still means "push it".
    Tested, so a later change cannot couple the two.
    
    43 new tests. 3411 passed, 13 pre-existing failures (broker, studio, manual
    client) untouched.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    51ffc4e1
test_remote_pinned_never_loads_locally.py 7.21 KB