ocr: surya's accents were lost to its own decode table, not to anything we do (v0.3.33)

ROOT CAUSE, measured in surya-ocr-2's own tokenizer:

    id 2160   decode -> U+FFFD    vocabulary piece -> 'a-grave'
    id 2168   decode -> U+FFFD    vocabulary piece -> 'e-grave'
    id 2169   decode -> U+FFFD    vocabulary piece -> 'e-acute'
    id 2172   decode -> U+FFFD    vocabulary piece -> 'i-grave'
    id 2178   decode -> U+FFFD    vocabulary piece -> 'o-grave'
    id 2185   decode -> U+FFFD    vocabulary piece -> 'u-grave'

The model emits the RIGHT token -- 2160 really is a-grave -- and the decode table
cannot render it. Every decoder in the chain inherits that: surya's own, and
vLLM's, which reports `decoded_token: None` and therefore `bytes: []` and writes
U+FFFD into the text.

So the accents were never lost in transport. Eliminated by measurement, each in
turn: DPI (a crisp synthetic page lost them identically), the document (chandra
reads the same pixels perfectly), the vLLM version (0.20.1 cannot load the model
at all), --trust-remote-code (applied, `trust_remote_code=True` in the engine
config, no change), and the byte channel (vLLM reports `bytes: []` precisely
because the decoder failed).

THE FIX: when the decoder answers with the replacement character for an id, use
the VOCABULARY's piece for that id. The vocabulary is the authority on what a
token means. Decoding is per id, because a bulk decode cannot be corrected after
the fact -- once flattened to text, the id that failed is unidentifiable.

Verified end to end against the live pod:

    decoder alone : 'nt ? pi? facot? perci? cos?'
    with vocab    : 'nt a-grave piu-grave facota-grave percie-acute cosi-grave'

This needs the ids, which vLLM only reports when its server is launched with
--return-tokens-as-token-ids (the per-REQUEST field is ignored on the chat path:
chat_completion/serving.py:1179 falls back to the server setting only when the
request field is None, and the id form is reached through _get_decoded_token).
That flag is now in Aruba's global vllm.extra_args, which is what a pod receives
-- the per-model `vllm` block in models.json does NOT reach a pod, which cost
two test cycles to discover.

An id the vocabulary also cannot name keeps its marker, and the ASCII-skeleton
guard still decides whether the repair as a whole is safe to apply.

tests: +3 (45 in test_ocr_accent_repair.py) pinning the vocabulary fallback, the
unknown-id case, and that decoding stays per id.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.32"
+__version__ = "0.3.33"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/ocr/workers/ocr_worker.py

@@ -389,11 +389,29 @@ def _token_ids(tokens):
 
 
 def _decode_ids(ids):
-    """``ids`` through surya's OWN tokenizer, which round-trips accents.
+    """``ids`` to text, using the VOCABULARY where the decoder fails.
 
-    Imported here rather than at module scope: this worker must start even when
-    the surya venv is mid-build, and a page that needs no repair must not pay for
-    loading a tokenizer. Cached on the function because a document is many pages.
+    This is the whole fix, and it rests on a measured contradiction inside
+    surya-ocr-2's own tokenizer:
+
+        id 2160  decode -> U+FFFD      vocabulary piece -> 'a-grave'
+        id 2168  decode -> U+FFFD      vocabulary piece -> 'e-grave'
+        id 2185  decode -> U+FFFD      vocabulary piece -> 'u-grave'
+
+    The model emits the RIGHT token -- 2160 really is a-grave -- and the decode
+    table cannot render it. Every decoder in the chain inherits that: surya's own,
+    and vLLM's, which reports ``decoded_token = None`` and therefore ``bytes: []``
+    and writes U+FFFD into the text. So the accents were never lost in transport,
+    at any resolution, on any vLLM version, with or without --trust-remote-code.
+    They were lost in a decode table that disagrees with its own vocabulary.
+
+    The vocabulary is the authority on what a token means, so when the decoder
+    answers with the replacement character, the piece is used instead. Per id,
+    because a bulk decode cannot be corrected after the fact.
+
+    Imported lazily: the worker must start while the surya venv is still building,
+    and a clean page must not pay for loading a tokenizer. Cached, because a
+    document is many pages.
     """
     if not ids:
         return None
@@ -403,6 +421,7 @@ def _decode_ids(ids):
             from transformers import AutoTokenizer
             ckpt = os.environ.get("SURYA_MODEL_CHECKPOINT") or "datalab-to/surya-ocr-2"
             tok = AutoTokenizer.from_pretrained(ckpt, trust_remote_code=True)
+            _decode_ids._vocab = {v: k for k, v in tok.get_vocab().items()}
         except Exception as e:
             print(f"[surya] cannot load the tokenizer for id decoding: {e}",
                   file=sys.stderr)
@@ -411,11 +430,19 @@ def _decode_ids(ids):
         _decode_ids._tok = tok
     if tok is False:
         return None
+    inv = getattr(_decode_ids, "_vocab", None) or {}
+    out = []
     try:
-        return tok.decode(ids, skip_special_tokens=True)
+        for i in ids:
+            piece = tok.decode([i], skip_special_tokens=True)
+            if UNREADABLE in piece:
+                # The decoder could not render this id. The vocabulary can.
+                piece = inv.get(i, piece)
+            out.append(piece)
     except Exception as e:
         print(f"[surya] id decoding failed: {e}", file=sys.stderr)
         return None
+    return "".join(out)
 
 
 def repair_content(raw: str, tokens):

tests/test_ocr_accent_repair.py

@@ -420,3 +420,63 @@ def test_forwarding_it_cannot_affect_an_install_that_did_not_ask():
     from codai.api import runpod_worker
     src = inspect.getsource(runpod_worker.engine_config_for_pod)
     assert "if v in (None, \"\", 0, False):" in src
+
+
+# ----------- the actual root cause: a decode table that contradicts the vocab #
+def test_the_vocabulary_wins_where_the_decoder_fails(monkeypatch):
+    """MEASURED in surya-ocr-2's own tokenizer:
+
+        id 2160  decode -> U+FFFD    vocabulary piece -> 'à'
+        id 2168  decode -> U+FFFD    vocabulary piece -> 'è'
+        id 2185  decode -> U+FFFD    vocabulary piece -> 'ù'
+
+    The model emits the RIGHT token and the decode table cannot render it, which
+    is why every decoder in the chain produced U+FFFD — surya's own, and vLLM's,
+    which then reports `decoded_token: None` and `bytes: []`. The vocabulary is
+    the authority on what a token means, so it is consulted when the decoder
+    answers with the replacement character."""
+    UN = "�"
+
+    class _Tok:
+        def decode(self, ids, skip_special_tokens=True):
+            return {2046: "n", 1979: " ", 2048: "p", 2041: "i",
+                    2160: UN, 2185: UN}.get(ids[0], "?")
+
+        def get_vocab(self):
+            return {"n": 2046, " ": 1979, "p": 2048, "i": 2041,
+                    "à": 2160, "ù": 2185}
+
+    monkeypatch.setattr(W._decode_ids, "_tok", _Tok(), raising=False)
+    monkeypatch.setattr(W._decode_ids, "_vocab",
+                        {2046: "n", 1979: " ", 2048: "p", 2041: "i",
+                         2160: "à", 2185: "ù"}, raising=False)
+    got = W._decode_ids([2048, 2041, 2185, 1979, 2046, 2160])
+    assert got == "più nà"
+    assert UN not in got
+
+
+def test_an_id_the_vocabulary_also_lacks_keeps_its_marker(monkeypatch):
+    """Not every unrenderable id is a known character. One the vocabulary cannot
+    name keeps the marker, and the ASCII-skeleton guard then decides whether the
+    whole repair is safe."""
+    UN = "�"
+
+    class _Tok:
+        def decode(self, ids, skip_special_tokens=True):
+            return UN
+
+        def get_vocab(self):
+            return {}
+
+    monkeypatch.setattr(W._decode_ids, "_tok", _Tok(), raising=False)
+    monkeypatch.setattr(W._decode_ids, "_vocab", {}, raising=False)
+    assert W._decode_ids([9999]) == UN
+
+
+def test_decoding_is_per_id_not_bulk():
+    """A bulk decode cannot be corrected afterwards — once the decoder has
+    flattened the sequence to text, the id that failed is unidentifiable."""
+    import inspect
+    src = inspect.getsource(W._decode_ids)
+    assert "for i in ids:" in src
+    assert "tok.decode([i]" in src