• Stefy Lanza (nextime / spora )'s avatar
    httpx's default connection pool was the throughput ceiling (v0.4.36) · 9ecc4122
    Stefy Lanza (nextime / spora ) authored
    The front forwards to an engine over a client built with `pool=None` --
    wait forever for a free connection -- and httpx defaults to
    `max_connections=100`. Together those two defaults cap everything the
    front forwards at 100 in flight and make the 101st wait indefinitely.
    Nothing anywhere reported a limit, because nothing had set one.
    
    Measured on a live orchestrator mid-campaign, client offering 512 pages:
    
        established to the front    514
        forwarded to the engine     100    <- the cap
        pages/s                    0.42    (fleet sized for ~3.3)
        mean page latency          238 s   (service time measured at ~19 s)
        cost per page              12x the measured baseline
    
    The pods were rented, healthy, and unreachable. The 200 s of that latency
    that is not service time was pages queueing for a connection.
    
    It also wedged the control plane, which is how this was finally found.
    /v1/runpod/status is not under /admin/api/, so it took the same unbounded
    path and never answered at all -- http=000 at 30 s, three times in a row,
    while /healthz answered in 2.1 ms. Two consequences, both worse than a
    blind admin page: a lease could not be RELEASED while the fleet was busy,
    so pods kept billing; and a client sizing its in-flight window from
    `summary.pods_ready` -- which is what the throughput guide tells it to do
    -- was flying blind as a direct result of the thing it was measuring.
    
      * the forwarding pool is sized from `ocr.max_concurrency` plus headroom,
        floored at 256 and capped at 8192. An unreadable config falls back to
        the floor, never to httpx's default: silently reinstating the limit
        this removes would be the worst possible fallback.
      * waiting for a connection is bounded at 30 s, so exhaustion surfaces as
        a retryable 502 rather than a socket that never answers. The READ
        timeout stays unbounded -- a page may legitimately take minutes, and
        confusing the queue with the work would truncate real pages.
      * control-plane paths (fleet status, scale, release, healthz, metrics)
        take a separate 16-connection pool that request traffic may not use.
        A pod that cannot be released because the release cannot get a
        connection keeps billing.
    
    This supersedes part of v0.4.34's diagnosis. The control pool and the
    bounded snapshot there were real fixes to real defects, but they were in
    the ENGINE app, and the wedge was in front of it.
    
    18 tests.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    9ecc4122
test_front_forward_pool_ceiling.py 5.99 KB