• Stefy Lanza (nextime / spora )'s avatar
    runpod: a request may narrow which cards a lease accepts (v0.3.54) · 3467f293
    Stefy Lanza (nextime / spora ) authored
    Per model, like every other pod setting, and bounded so that it can only ever
    NARROW what the operator configured.
    
    A pool with a preference and allow_other_gpus: true takes another card when the
    preferred one is not free -- usually right, occasionally exactly wrong. Observed
    2026-10-13: gpu_types [NVIDIA A40] fell back to an L40S at $1.09/hr against the
    A40's $0.59, correct per the configuration and useless to a caller measuring
    throughput, since a number taken on a different card is not comparable with the
    one it is being compared to. The only way to get the A40 was to edit the model's
    configuration -- changing it for everybody, permanently, for one request.
    
    POST /v1/runpod/scale now takes `gpu_types` (list or comma string, in preference
    order) and `allow_other_gpus`. The rule is one-directional:
    
      - a card the operator did not list is DROPPED and named in `notes`;
      - narrowing to only cards the operator forbids is a 400 that names what IS
        allowed -- never a silent fall back to the default, which would be the
        opposite of what the caller asked for;
      - allow_other_gpus may be turned OFF by a request, never ON;
      - max_hourly_usd, min_vram_gb, cloud_types and max_pods are the operator's
        alone and are not reachable from a request at all.
    
    Held on the LEASE, not the pool: an "A40 only" cannot outlive the 30 minutes it
    asked for, releasing the lease releases its constraints, and provisioning reads
    the narrowing per pod so a lapsed lease stops constraining the very next one.
    There is deliberately no path from a request to a model's configuration.
    
    Reported back in the answer and on /v1/runpod/status as `client_gpu_narrow`,
    because a constraint silently dropped is worse than one refused -- the caller
    would read its numbers as having come from the card it asked for -- and a pod on
    an unexpected card should have a visible cause, not only a bill.
    
    Tests: 32 new, including that the cost ceiling and the VRAM floor still refuse a
    pinned card, that the operator's limits are not even mentioned in the narrowing
    code, and that None (not False) is the default for both fields -- False would
    silently turn every caller's fallback off.
    
    Also fixed a source-window test that anchored on `_rank_gpus`'s full argument
    list and broke the moment a parameter was added; it now anchors on the
    assignment, since the arguments are not what it is testing.
    
    Full suite: 25 failures, all pre-existing and identical to baseline.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    3467f293
test_runpod_request_gpu_choice.py 12.6 KB