• Stefy Lanza (nextime / spora )'s avatar
    an operator may decline a self-harming page budget (v0.4.22) · 0ed00f5b
    Stefy Lanza (nextime / spora ) authored
    A per-request `max_tokens` wins over the engine's own default and over the
    operator's configured budget. That is right -- only the caller knows whether it
    is sending a four-page contract or a fifty-page judgment -- and it stays the
    default.
    
    But it cost a run, and the operator had no say. A client sent
    `max_tokens: 3072` on every page of a corpus whose pages need just over that.
    Each page therefore generated to the cap, stopped mid-text, and paid a
    continuation that RE-SENT the first 3072 tokens to retrieve the last few
    hundred; by the second cut-off the model was often repeating, so the page was
    refused a resume and re-run whole at a higher temperature.
    
    Measured across 594 continuations on the live fleet: 532 pages reached segment
    2, 44 segment 3, 11 segment 4, 7 segment 5. That distribution is the signature
    of a budget MARGINALLY too small, which is the most expensive kind -- nearly
    every page pays double, rather than a few pages paying a lot. The cards sat at
    `GPU KV cache usage: 1.4%` with `Running: 0-4 reqs` while the pool grew to 65
    pods at $47/hr, because the time was going on sequential round trips and not on
    inference. Adding pods could not help and did not.
    
    So `<engine>_max_tokens_floor` (surya, chandra, olmocr): an operator may state
    a minimum. Off by default -- an install that silently overrode a caller would
    be lying about its API -- never LOWERS a budget, because a caller asking for
    more knows something the floor does not, and never invents one where the caller
    asked for nothing (omitted means "use the configured value", which may well be
    higher) or asked for 0 (which means unlimited, so raising it would cap it).
    
    Applied once at the top of `ocr_document`, before either pipeline sees the
    opts, so a local read and a pod forward cannot disagree about the budget. Said
    once per engine per five minutes with both numbers and the reason, because a
    5,000-document run would otherwise write the line 30,000 times, and because an
    operator should not have to find this commit to learn why a small budget is
    expensive. Seeded to pods as `CODERAI_OCR_<ENGINE>_MAX_TOKENS_FLOOR`.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    0ed00f5b
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...