• Stefy Lanza (nextime / spora )'s avatar
    ratelimit: 60 req/min was measured as a GPU cliff, and throttled rented pods (v0.3.38) · cd133742
    Stefy Lanza (nextime / spora ) authored
    `/v1/chat/completions` was capped at 60 requests per 60 s per client IP as a
    module constant — no config, no env, no way to move it.
    
    On a RENTED POD that cap is the orchestrator's own: a pod has exactly one client,
    the install paying for it by the hour, so every request shares one bucket. An
    olmOCR pod measured 3.45 pages/s (207/min) and was being refused above 60.
    
    It also produced three wrong measurements that were reported as hardware facts.
    Concurrency ladders ran every level inside one window, so refusals began exactly
    where the cumulative count crossed 60:
    
        surya   16+24=40, then 20 of 32 pass, 12 refused   (13 measured)
        surya   +48 = 72 over,  all 48 refused             (all 48 measured)
        chandra 8+16+24=48, then 12 pass, 20 refused       (21 measured)
        olmOCR  same ladder, same arithmetic               (21 measured, twice)
    
    "Both peak at 16 and both fail outright by 32 — a cliff" was the rate limiter.
    Paced across windows, olmOCR runs N=32 with ZERO errors at its best throughput;
    it is flat at ~3.4 pages/s from N=8 to N=32 while p50 goes 2.3s -> 7.9s, which is
    saturation at 8, not a cliff at 32.
    
    So: the limits take env overrides (CODERAI_SERVER_RATE_LIMITS="prefix=count/window",
    CODERAI_SERVER_RATE_LIMIT_ENABLED=0), and a pod is launched with the limiter OFF —
    its admission is pod_max_parallel_requests + pod_queue_max_size, which describe
    the GPU rather than guessing at an abusive client that cannot exist there.
    
    docs/ocr.md keeps the old table with the refusals marked as refusals, because the
    numbers themselves are still the only ones measured and the reading of them was
    the error.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    cd133742
test_text_lora_to_pod.py 16.8 KB