• Stefy Lanza (nextime / spora )'s avatar
    the admission gate must not be a hidden throughput ceiling (v0.4.25) · c3f4a2c0
    Stefy Lanza (nextime / spora ) authored
    server.max_parallel_requests defaults to 2 in flight and 6 queued. That is
    right for one local card answering chat. For an OCR install dispatching pages
    to rented pods it is the throughput ceiling, and nothing points at it: the pool
    is sized in pages (ocr.max_concurrency, max_inflight_per_pod), a pod is handed
    its own much larger limit through env, and the front in front of all of it
    keeps admitting two requests.
    
    Measured on one A40 with ocr.max_concurrency=4096 and the pod allowed 64: the
    pod sat at `Running: 1, Waiting: 0` -- asking for work and not getting it --
    while 219 pages took 1210s, the same as reading them one at a time. Raising the
    gate took the pod to 8-11 concurrent sequences on the same documents. The
    `[queue] max_parallel_requests=2` line was printed at every boot throughout and
    read past, because a number with no consequence attached does not look like a
    problem.
    
    Two changes, neither of which overrides a deliberate choice:
    
    * When OCR is enabled and the operator has NOT written the key, the gate is
      derived from the page budget they did ask for (page_budget/8, floored at 16
      and capped at 256) rather than taking a default chosen for a different
      deployment. An explicit value is obeyed however small, and so is an env
      override, since env is the only channel a rented pod has. The cap is there
      because the remaining bound is memory, not CPU: an admitted request holds its
      whole upload plus the per-page PDFs split from it.
    * When the gate is explicitly set below what the configured page budget needs,
      a startup diagnostic says so and names the setting to change, instead of
      leaving the operator to infer it from pods that look idle.
    
    ConfigManager now records which server keys were actually written, because an
    explicit 2 and an absent 2 are indistinguishable from the parsed value and must
    be treated differently.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    c3f4a2c0
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...