-
Stefy Lanza (nextime / spora ) authored
ROOT CAUSE, measured in surya-ocr-2's own tokenizer: id 2160 decode -> U+FFFD vocabulary piece -> 'a-grave' id 2168 decode -> U+FFFD vocabulary piece -> 'e-grave' id 2169 decode -> U+FFFD vocabulary piece -> 'e-acute' id 2172 decode -> U+FFFD vocabulary piece -> 'i-grave' id 2178 decode -> U+FFFD vocabulary piece -> 'o-grave' id 2185 decode -> U+FFFD vocabulary piece -> 'u-grave' The model emits the RIGHT token -- 2160 really is a-grave -- and the decode table cannot render it. Every decoder in the chain inherits that: surya's own, and vLLM's, which reports `decoded_token: None` and therefore `bytes: []` and writes U+FFFD into the text. So the accents were never lost in transport. Eliminated by measurement, each in turn: DPI (a crisp synthetic page lost them identically), the document (chandra reads the same pixels perfectly), the vLLM version (0.20.1 cannot load the model at all), --trust-remote-code (applied, `trust_remote_code=True` in the engine config, no change), and the byte channel (vLLM reports `bytes: []` precisely because the decoder failed). THE FIX: when the decoder answers with the replacement character for an id, use the VOCABULARY's piece for that id. The vocabulary is the authority on what a token means. Decoding is per id, because a bulk decode cannot be corrected after the fact -- once flattened to text, the id that failed is unidentifiable. Verified end to end against the live pod: decoder alone : 'nt ? pi? facot? perci? cos?' with vocab : 'nt a-grave piu-grave facota-grave percie-acute cosi-grave' This needs the ids, which vLLM only reports when its server is launched with --return-tokens-as-token-ids (the per-REQUEST field is ignored on the chat path: chat_completion/serving.py:1179 falls back to the server setting only when the request field is None, and the id form is reached through _get_decoded_token). That flag is now in Aruba's global vllm.extra_args, which is what a pod receives -- the per-model `vllm` block in models.json does NOT reach a pod, which cost two test cycles to discover. An id the vocabulary also cannot name keeps its marker, and the ASCII-skeleton guard still decides whether the repair as a whole is safe to apply. tests: +3 (45 in test_ocr_accent_repair.py) pinning the vocabulary fallback, the unknown-id case, and that decoding stays per id. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
ccc0ac89