• Stefy Lanza (nextime / spora )'s avatar
    ocr: per-request token and continuation budgets, and a safe ceiling (v0.3.49) · 88add444
    Stefy Lanza (nextime / spora ) authored
    /v1/ocr took engine, dpi, lang, structured, schema and detect. FastAPI drops
    unknown form fields SILENTLY, so a caller sending max_tokens=8000 on every
    request was discarded without a word and every page ran at the configured
    value. Both budgets are now real request parameters.
    
    One global cap cannot serve a public deployment: a four-page contract and a
    fifty-page judgment need different budgets and only the caller knows which it
    is sending. The values travel WITH the page rather than with the engine load,
    because a pool is built once and then serves every caller, and they are
    applied thread-locally in the worker, because several pages are in flight in
    one process and the hooks they feed are module-level — a per-request value in
    a module cell would be overwritten by the next page to arrive. Carried on a
    ContextVar rather than widening `recognize_image`, which is the one thing
    every engine has, duck-typed ones included.
    
    Omitting a parameter means "use the configured value", not zero: 0 is
    meaningful for both (no cap override, and resumption off).
    
    surya_max_continuations 2 -> 8. It is PER PAGE, not per document: a fifty-page
    document is fifty independent pages each with its own budget, so what the
    number covers is the densest SINGLE page. The real ceiling is not the number
    but the model's context — each resumption re-sends the text so far as INPUT,
    so with surya-ocr-2's 32,768-token window and ~4,000 of it on the page image,
    about thirteen 2,048-token segments exhaust it whatever is configured, and
    raising surya_max_tokens lowers that count in proportion.
    
    A generous budget is only safe if a page that stops making progress stops
    costing, so resumption now ends when a segment comes back repeating itself or
    returns the text already in hand. The second case is also how a server that
    IGNORES continue_final_message is caught: a model that starts the page over
    returns what we already have, and without the check a budget of 8 would buy
    eight full generations of the same page.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    88add444
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...