• Stefy Lanza (nextime / spora )'s avatar
    ocr: a ruined page is not an error, and nothing was catching it (v0.3.79) · 731ae0e4
    Stefy Lanza (nextime / spora ) authored
    From last night's run: ~97% of a 1,000-document batch came out truncated,
    with no error anywhere. The engine answered 200 every time. Its decode had
    fallen into a loop, run to the token cap emitting the same fragment, and
    the resumer correctly refused to continue a reply that was already
    repeating itself -- so the page was flagged, returned, and indexed.
    
    `fallback_engines` never fired, because it only fires on an exception.
    The same pages read cleanly on another engine (38,055 characters against a
    truncated stub), so the recovery was there the whole time and nothing
    reached for it.
    
    `ocr.retry_degraded` (default "auto") re-reads ONLY the degraded pages on
    another engine and keeps the replacement only if it is not degraded in
    turn. Bounded by construction: a clean document costs nothing, a ruined
    one costs at most a second pass over its ruined pages, and
    `retry_degraded_max_pages` caps even that. The page keeps its own index
    and carries `degraded_retry` saying who re-read it and why; the response
    reports `recovered_pages`.
    
    Degraded means looping or truncated -- the two the engine itself attests.
    NOT a short page, a blank page or an unreadable one: those have their own
    meaning and re-reading them spends a card to be told the same thing.
    
    Which leaves the other half of the report: olmOCR returned 4,046
    characters where chandra read 30,983 on the same page, ending cleanly well
    short of the cap, flagging nothing. The only evidence an early stop leaves
    is the company it keeps, so `retry_short_pages_below` compares a page with
    the DOCUMENT'S OWN median rather than a character count -- a threshold in
    characters would need re-tuning for every language, DPI and layout.
    Default 0, OFF, and deliberately: this signal is inferred where the others
    are attested, and a genuine title or signature page looks identical. A
    short page is also only replaced by a LONGER one.
    
    Off on pods: only the orchestrator can reach another engine's pods, so a
    pod would re-read on the engine that just produced the flag.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    731ae0e4
Name
Last commit
Last update
..
static Loading commit data...
templates Loading commit data...
__init__.py Loading commit data...
auth.py Loading commit data...
download_worker.py Loading commit data...
routes.py Loading commit data...