-
Stefy Lanza (nextime / spora ) authored
The front forwards to an engine over a client built with `pool=None` -- wait forever for a free connection -- and httpx defaults to `max_connections=100`. Together those two defaults cap everything the front forwards at 100 in flight and make the 101st wait indefinitely. Nothing anywhere reported a limit, because nothing had set one. Measured on a live orchestrator mid-campaign, client offering 512 pages: established to the front 514 forwarded to the engine 100 <- the cap pages/s 0.42 (fleet sized for ~3.3) mean page latency 238 s (service time measured at ~19 s) cost per page 12x the measured baseline The pods were rented, healthy, and unreachable. The 200 s of that latency that is not service time was pages queueing for a connection. It also wedged the control plane, which is how this was finally found. /v1/runpod/status is not under /admin/api/, so it took the same unbounded path and never answered at all -- http=000 at 30 s, three times in a row, while /healthz answered in 2.1 ms. Two consequences, both worse than a blind admin page: a lease could not be RELEASED while the fleet was busy, so pods kept billing; and a client sizing its in-flight window from `summary.pods_ready` -- which is what the throughput guide tells it to do -- was flying blind as a direct result of the thing it was measuring. * the forwarding pool is sized from `ocr.max_concurrency` plus headroom, floored at 256 and capped at 8192. An unreadable config falls back to the floor, never to httpx's default: silently reinstating the limit this removes would be the worst possible fallback. * waiting for a connection is bounded at 30 s, so exhaustion surfaces as a retryable 502 rather than a socket that never answers. The READ timeout stays unbounded -- a page may legitimately take minutes, and confusing the queue with the work would truncate real pages. * control-plane paths (fleet status, scale, release, healthz, metrics) take a separate 16-connection pool that request traffic may not use. A pod that cannot be released because the release cannot get a connection keeps billing. This supersedes part of v0.4.34's diagnosis. The control pool and the bounded snapshot there were real fixes to real defects, but they were in the ENGINE app, and the wedge was in front of it. 18 tests. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
9ecc4122