• Stefy Lanza (nextime / spora )'s avatar
    the shared pod client capped the whole fleet at 256 connections (v0.4.44) · 9e1f16a7
    Stefy Lanza (nextime / spora ) authored
    `_pool_size()` is a PER-POD figure -- what `requests` means by
    `pool_maxsize`, which keeps one pool per host. The async client that
    forwards every page to every pod is ONE httpx.AsyncClient for the whole
    fleet, and httpx's `max_connections` is a TOTAL across hosts. Passing
    the per-pod 256 to it capped every page forward in the install at 256.
    
    Measured 2026-10-19 on a 31-pod fleet: 1,583 pages holding a pod lease,
    284 TCP connections to pods in total -- six pods with 25-49 connections,
    most with exactly one. Renting more pods added cards and no throughput.
    Pages that waited too long for a connection failed with an empty-message
    timeout, reported as the pod being unreachable, and healthy pods were
    quarantined for it.
    
    The async client is now sized from the page ceiling the install is
    configured for (ocr.max_concurrency + headroom, capped at 16384): 5120
    at the current 4096. The sync per-host pool is unchanged; per host is
    the right meaning there. An existing test asserted the shared client's
    limit equalled the per-pod figure -- it pinned the bug, and now asserts
    the fleet total.
    Co-Authored-By: 's avatarClaude Opus 5.5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    9e1f16a7
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...