• Stefy Lanza (nextime / spora )'s avatar
    ocr: surya says when a page was still looping, and its detector is tunable (v0.3.64) · 0e334bae
    Stefy Lanza (nextime / spora ) authored
    Raised from the Digesta side: the one high-leverage way to make surya fall
    through less often is to make its repeat-detector and retry tunable. Half right,
    and the other half turned out to be the bigger lever.
    
    Checking the installed surya rather than the description: `_should_retry` stops
    at `max_retries` and RETURNS the looping text — surya does not return empty. The
    empty page comes later, because looping markup parses to no blocks. So the
    symptom is real and the mechanism is not what it looked like.
    
    **The lever is that surya never said it looped.** chandra has reported
    `repeat_detected` since it shipped, with the text returned anyway because a
    looping transcription usually has a correct beginning. surya reported nothing,
    so a short or empty page was indistinguishable from an unreadable scan — and
    those need opposite handling: one is a cascade candidate, the other means the
    document is bad. surya now flags it at the moment the verdict is made (the
    retry hook's last call for the page), with `repeat_chars`, the length of the
    reply that was still looping: "looped and produced four thousand characters" has
    a usable beginning, "looped and produced nothing" does not.
    
    **And the detector is now retunable** — `ocr.surya_repeat_max_repeats`,
    `surya_repeat_window`, `surya_repeat_scaling`, 0 on each leaving surya's own (4,
    500, 3.0). Hooked like `_should_retry` already was, since the vLLM backend
    passes neither. A false positive is expensive: the page is re-run several times
    at rising temperature and then parsed from a reply nobody wanted, usually
    yielding less text than the first one had. Operator settings, deliberately not
    per-request — they describe the corpus (tables, dotted leaders and form fields
    repeat short strings legitimately), not the page. An explicit argument from
    surya's own code still wins, so its `cut_from_end` call keeps working.
    
    11 tests in tests/test_surya_loop_reported.py, including that a loop surya then
    rescues is not recorded as a verdict, and that the thresholds must be in place
    BEFORE the hook is installed (it patches once per process).
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    0e334bae
surya.py 16.6 KB