• Stefy Lanza (nextime / spora )'s avatar
    ocr: surya's accents were lost to its own decode table, not to anything we do (v0.3.33) · ccc0ac89
    Stefy Lanza (nextime / spora ) authored
    ROOT CAUSE, measured in surya-ocr-2's own tokenizer:
    
        id 2160   decode -> U+FFFD    vocabulary piece -> 'a-grave'
        id 2168   decode -> U+FFFD    vocabulary piece -> 'e-grave'
        id 2169   decode -> U+FFFD    vocabulary piece -> 'e-acute'
        id 2172   decode -> U+FFFD    vocabulary piece -> 'i-grave'
        id 2178   decode -> U+FFFD    vocabulary piece -> 'o-grave'
        id 2185   decode -> U+FFFD    vocabulary piece -> 'u-grave'
    
    The model emits the RIGHT token -- 2160 really is a-grave -- and the decode table
    cannot render it. Every decoder in the chain inherits that: surya's own, and
    vLLM's, which reports `decoded_token: None` and therefore `bytes: []` and writes
    U+FFFD into the text.
    
    So the accents were never lost in transport. Eliminated by measurement, each in
    turn: DPI (a crisp synthetic page lost them identically), the document (chandra
    reads the same pixels perfectly), the vLLM version (0.20.1 cannot load the model
    at all), --trust-remote-code (applied, `trust_remote_code=True` in the engine
    config, no change), and the byte channel (vLLM reports `bytes: []` precisely
    because the decoder failed).
    
    THE FIX: when the decoder answers with the replacement character for an id, use
    the VOCABULARY's piece for that id. The vocabulary is the authority on what a
    token means. Decoding is per id, because a bulk decode cannot be corrected after
    the fact -- once flattened to text, the id that failed is unidentifiable.
    
    Verified end to end against the live pod:
    
        decoder alone : 'nt ? pi? facot? perci? cos?'
        with vocab    : 'nt a-grave piu-grave facota-grave percie-acute cosi-grave'
    
    This needs the ids, which vLLM only reports when its server is launched with
    --return-tokens-as-token-ids (the per-REQUEST field is ignored on the chat path:
    chat_completion/serving.py:1179 falls back to the server setting only when the
    request field is None, and the id form is reached through _get_decoded_token).
    That flag is now in Aruba's global vllm.extra_args, which is what a pod receives
    -- the per-model `vllm` block in models.json does NOT reach a pod, which cost
    two test cycles to discover.
    
    An id the vocabulary also cannot name keeps its marker, and the ASCII-skeleton
    guard still decides whether the repair as a whole is safe to apply.
    
    tests: +3 (45 in test_ocr_accent_repair.py) pinning the vocabulary fallback, the
    unknown-id case, and that decoding stays per id.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    ccc0ac89
__init__.py 4.21 KB