• Stefy Lanza (nextime / spora )'s avatar
    surya reports a page confidence, and whether anything measured it (v0.4.15) · 6053e8b1
    Stefy Lanza (nextime / spora ) authored
    chandra and olmOCR each set `meta.confidence` and `meta.confidence_measured`.
    surya set neither: its predictor reports confidence per LINE, so a caller
    switching engines was handed a different kind of answer, and nothing could tell
    "no line reported a confidence" from "the model was certain it was unsure".
    
    Now it reports both. Weighted by CHARACTERS rather than per line, because the
    other two average over tokens -- that is, over text -- and a plain line-mean
    lets a two-word line count as much as a full one: on
    [(100 chars, 0.95), (2 chars, 0.1)] it answers 0.9333 instead of 0.525.
    
    `confidence_measured` is the half that earns its place, and the reason the
    aggregate reads the RAW value rather than the per-line field: that field has to
    coerce a missing confidence to 0.0 for its own sake, which would make
    
        nothing reported  -> 0.0
        a real zero       -> 0.0
    
    indistinguishable. One such 0.0 has already produced a confidently wrong
    verdict in this subsystem. So:
    
        nothing reported  -> (0.0, False)
        a real zero       -> (0.0, True)
        mixed             -> unreported lines skipped, never counted as zero
    
    Both keys reach the caller with no new plumbing: `_page_from_dict` forwards
    unknown keys by EXCLUSION (a choice made after that reassembly silently dropped
    `accents_repaired`, and then `model_attempts`), and both are already in the
    pod->orchestrator whitelist, so a page read on a rented pod reports what a
    local one does. Tested, including that `False` is not mistaken for the empty
    value that exclusion filter skips.
    
    The aggregate is extracted as `_page_confidence()` so it is testable without
    surya installed.
    
    Worth recording what this is NOT: on the page where the accent bug was found,
    surya reported ~96 % per line while every accent was a replacement character --
    confident, and right to be, having emitted the correct tokens. Confidence is
    not a corruption detector; the byte trace is.
    
    13 new tests. 3447 passed, 13 pre-existing failures (broker, studio, manual
    client) untouched.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    6053e8b1
test_ocr_surya_confidence.py 4.82 KB