• Stefy Lanza (nextime / spora )'s avatar
    ocr: surya's accents were lost to its own decode table, not to anything we do (v0.3.33) · ccc0ac89
    Stefy Lanza (nextime / spora ) authored
    ROOT CAUSE, measured in surya-ocr-2's own tokenizer:
    
        id 2160   decode -> U+FFFD    vocabulary piece -> 'a-grave'
        id 2168   decode -> U+FFFD    vocabulary piece -> 'e-grave'
        id 2169   decode -> U+FFFD    vocabulary piece -> 'e-acute'
        id 2172   decode -> U+FFFD    vocabulary piece -> 'i-grave'
        id 2178   decode -> U+FFFD    vocabulary piece -> 'o-grave'
        id 2185   decode -> U+FFFD    vocabulary piece -> 'u-grave'
    
    The model emits the RIGHT token -- 2160 really is a-grave -- and the decode table
    cannot render it. Every decoder in the chain inherits that: surya's own, and
    vLLM's, which reports `decoded_token: None` and therefore `bytes: []` and writes
    U+FFFD into the text.
    
    So the accents were never lost in transport. Eliminated by measurement, each in
    turn: DPI (a crisp synthetic page lost them identically), the document (chandra
    reads the same pixels perfectly), the vLLM version (0.20.1 cannot load the model
    at all), --trust-remote-code (applied, `trust_remote_code=True` in the engine
    config, no change), and the byte channel (vLLM reports `bytes: []` precisely
    because the decoder failed).
    
    THE FIX: when the decoder answers with the replacement character for an id, use
    the VOCABULARY's piece for that id. The vocabulary is the authority on what a
    token means. Decoding is per id, because a bulk decode cannot be corrected after
    the fact -- once flattened to text, the id that failed is unidentifiable.
    
    Verified end to end against the live pod:
    
        decoder alone : 'nt ? pi? facot? perci? cos?'
        with vocab    : 'nt a-grave piu-grave facota-grave percie-acute cosi-grave'
    
    This needs the ids, which vLLM only reports when its server is launched with
    --return-tokens-as-token-ids (the per-REQUEST field is ignored on the chat path:
    chat_completion/serving.py:1179 falls back to the server setting only when the
    request field is None, and the id form is reached through _get_decoded_token).
    That flag is now in Aruba's global vllm.extra_args, which is what a pod receives
    -- the per-model `vllm` block in models.json does NOT reach a pod, which cost
    two test cycles to discover.
    
    An id the vocabulary also cannot name keeps its marker, and the ASCII-skeleton
    guard still decides whether the repair as a whole is safe to apply.
    
    tests: +3 (45 in test_ocr_accent_repair.py) pinning the vocabulary fallback, the
    unknown-id case, and that decoding stays per id.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    ccc0ac89
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
compose Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
util Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...