• Stefy Lanza (nextime / spora )'s avatar
    ocr: finish a page that ran out of room; say when a page is simply blank (v0.3.48) · bba914d5
    Stefy Lanza (nextime / spora ) authored
    Two things, from one sample document.
    
    RESUMING A CUT-OFF PAGE. Raising surya_max_tokens only moves the cliff: a
    dense enough page overflows any fixed number, and incomplete block markup
    parses to a fragment or to nothing. `ocr.surya_max_continuations` (default 2)
    resumes instead -- the text so far goes back as a prefilled assistant turn
    with vLLM's `continue_final_message`, so the model carries on the exact string
    rather than answering afresh about the same image. A server rejecting the flag
    (llama.cpp) gets a plain "continue" instruction; a server rejecting both
    leaves the page partial and honestly marked, since a page that cannot be
    finished is still a page. Logprobs are concatenated in step with the text,
    because surya_repair_accents rebuilds characters from the model's own
    per-token bytes and a later segment paired with earlier bytes would corrupt
    exactly the accented characters the repair exists to fix; if they cannot be
    aligned the repair is given up on rather than fed a mismatch. Temperature 0 --
    the opposite of the retry path, which raises it to break a repetition. A page
    still unfinished after its budget carries "complete": false so a caller can
    refuse a partial rather than store it as read. On by default: returning a page
    KNOWN to be incomplete is a correctness failure, not a cost trade-off.
    
    IS THERE ANYTHING ON THE PAGE AT ALL. A four-page court PDF came back with 47
    characters and was reported as an OCR failure. It was not. Every page was a
    ~900-byte CCITT Group 4 image -- what an EMPTY page costs, a couple of bytes
    per all-white row -- carrying 0.13% ink, all of it a diagonal scan edge and a
    diagonal "copia non ufficiale" watermark. The 47 characters WERE the
    watermark. The engine had read the page perfectly.
    
    "The model could not read it", "the reply was cut off" and "the page is blank"
    were indistinguishable in a short result, and they lead to completely
    different places -- a better engine, a bigger budget, or a document fixed
    upstream. Any page whose text is shorter than ocr.blank_page_max_chars (200)
    now carries the fraction of its pixels that are not background, from an image
    already in memory, with blank_pages listed at the top of the response. The
    opposite verdict is as useful as the blank one: ink above the floor says there
    IS something there, so the engine's failure is real and worth chasing. It
    applies to every engine, because the measurement is of the image rather than
    of the model, and it is a diagnostic only -- nothing is skipped, refused or
    altered because of it.
    
    Verified against the sample: all four pages 0.13-0.33% ink, correctly called
    blank against the 0.5% floor.
    
    Two bugs found while testing: `rounds` counted resumption ATTEMPTS rather than
    successes, so a failed resume would have reported the page as whole; and three
    assertions matched a source window that moved, one of them for a sentence that
    is wrapped across f-string lines and never appears contiguously.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    bba914d5
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...