ocr: recover surya's accents on vLLM 0.27.1 from the token IDS (v0.3.31)

The byte channel is a dead end on 0.27.1 -- measured: `content` comes back with
one U+FFFD per accented letter AND the per-token `bytes` come back EMPTY (257 of
them on one page, no byte >= 0x80 anywhere), so neither output carries the
character. But the token IDS are not lossy, and surya's OWN tokenizer decodes
them correctly: decode([2131, 2104]) == 'e-grave', measured in its venv.

So the repair now asks for ids and decodes them itself, bypassing vLLM's
detokenizer entirely. Both halves, because each alone does nothing:

1. THE WORKER reads vLLM's `token_id:{id}` form, decodes with surya's tokenizer
   (lazily loaded and cached -- the worker must start while the venv is building,
   and a clean page must not pay for a tokenizer), and keeps the byte route as the
   fallback for any stack that reports bytes honestly. Ids are tried FIRST because
   on 0.27.1 the bytes lead nowhere. All-or-nothing on the id list: a partial one
   would decode to a partial page. The ASCII-skeleton guard still applies, so a
   decode of the wrong call cannot be substituted.

2. THE PROXY stops dropping the request. `return_tokens_as_token_ids` is a
   per-request field on vLLM's ChatCompletionRequest, but sending it to a pod
   changed nothing -- because codai/backends/vllm.py::_chat_payload builds an
   explicit whitelist and silently discarded it. That is exactly the failure its
   own neighbouring comment warns about ("a proxy that edits the request decides
   for it"), five lines further up. Now forwarded, gated on
   `vllm.return_token_ids`, and only alongside logprobs since it describes how
   logprob tokens are rendered.

`vllm.return_token_ids` is off by default (it changes what the `token` field
looks like, and only this one consumer reads it) and is in VLLM_FIELDS, so the
single checkpoint with a byte-level vocabulary can ask for ids without changing
what every other vLLM model reports. A config field rather than relaunching the
server with --return-tokens-as-token-ids: same effect, no pod rebuild.

This is a fix ON 0.27.1. It makes the coderai-vllm-ocr image (vLLM 0.20.1) a
CONTROL that confirms the diagnosis rather than the only way out.

tests: +11 in test_ocr_accent_repair.py (36 -> 47) covering the id form, the
all-or-nothing rule, ids-before-bytes, the fallback when a decode fails, the
skeleton guard on the id route, and the proxy forwarding with its logprobs
gating. 912 passed across the ocr/front/config/admin/runpod/vllm/backend
selection; the 9 failures are pre-existing (broker x6, studio, whisper x2) plus
test_cluster's RPC check, which depends on llama_cpp being installed in this venv
and touches nothing changed here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.30"
+__version__ = "0.3.31"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -4776,6 +4776,7 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
         if "gpu_memory_utilization" in d:
             try: v.gpu_memory_utilization = float(d.get("gpu_memory_utilization") or v.gpu_memory_utilization)
             except (TypeError, ValueError): pass
+        if "return_token_ids" in d: v.return_token_ids = bool(d["return_token_ids"])
         if "tensor_parallel_size" in d:
             try: v.tensor_parallel_size = max(1, int(d.get("tensor_parallel_size") or 1))
             except (TypeError, ValueError): pass

codai/backends/overrides.py

@@ -20,7 +20,11 @@ import dataclasses
 VLLM_FIELDS = ("ctx", "gpu", "gpu_memory_utilization", "tensor_parallel_size",
                "pipeline_parallel_size", "distributed_executor_backend", "ray_address",
                "ray_port", "nodes", "nodes_ready_timeout_s", "max_num_seqs", "dtype",
-               "quantization", "extra_args", "extra_env")
+               "quantization", "extra_args", "extra_env",
+               # Per model, so the one checkpoint with a byte-level vocabulary can
+               # ask for token ids without changing what every other vLLM model
+               # reports (see VllmConfig.return_token_ids).
+               "return_token_ids")
 
 KT_FIELDS = ("ctx", "kt_weight_path", "nnodes", "dist_init_addr", "tp_size", "nodes",
              "extra_args", "extra_env")

codai/backends/vllm.py

@@ -213,6 +213,22 @@ class VllmBackend(ModelBackend):
             payload["logprobs"] = True
             if top_logprobs is not None:
                 payload["top_logprobs"] = int(top_logprobs)
+            # Ask for the RAW VOCABULARY IDS rather than rendered strings, when the
+            # operator has turned it on. This is the only lossless channel for a
+            # model whose vocabulary is byte-level: measured on vLLM 0.27.1 serving
+            # surya-ocr-2, `content` comes back with one U+FFFD per accented letter
+            # and the per-token `bytes` come back EMPTY, so neither output carries
+            # the character -- while the ids are intact and surya's own tokenizer
+            # decodes them correctly. vLLM's own words for the field: "so that
+            # tokens that are not JSON-encodable can be identified".
+            #
+            # Set here rather than relaunching the server with
+            # --return-tokens-as-token-ids, because this is a per-request field and
+            # a config change beats a pod rebuild. It only changes the `token`
+            # STRING inside logprobs; nothing in coderai reads that except the OCR
+            # worker's repair.
+            if getattr(self._cfg, "return_token_ids", False):
+                payload["return_tokens_as_token_ids"] = True
         return payload
 
     # ------------------------------------------------------------------ #

codai/config.py

@@ -709,6 +709,19 @@ class VllmConfig:
     the OCR subsystem passes its own model. Leave ``model_id`` blank for the model-list flow.
     Also reused by the OCR subsystem to serve Surya2 (a VLM) with continuous batching.
     """
+    # Ask vLLM to report logprob tokens as `token_id:{id}` instead of rendered
+    # strings. Off by default because it changes what that field looks like, and
+    # only one consumer reads it.
+    #
+    # It exists for a measured dead end: on vLLM 0.27.1 serving
+    # `datalab-to/surya-ocr-2`, whose vocabulary is BYTE-level, an accented letter
+    # comes back as U+FFFD in `content` AND as an empty `bytes` list in the
+    # logprobs -- 257 such tokens on one page, with no byte >= 0x80 anywhere. So
+    # there is nothing to recover from either output. The token IDS are intact,
+    # and surya's own tokenizer decodes them correctly, so with this on the OCR
+    # worker can rebuild the text itself (`ocr.surya_repair_accents`) instead of
+    # waiting for an upstream fix or pinning vLLM back to 0.20.1.
+    return_token_ids: bool = False
     enabled: bool = False
     # Point this at an engine already running somewhere else (another host, a
     # container, a rented pod) and coderai proxies to it instead of building,
@@ -1690,6 +1703,7 @@ class ConfigManager:
                 "port": self.config.vllm.port,
                 "ctx": self.config.vllm.ctx,
                 "gpu_memory_utilization": self.config.vllm.gpu_memory_utilization,
+                "return_token_ids": self.config.vllm.return_token_ids,
                 "tensor_parallel_size": self.config.vllm.tensor_parallel_size,
                 "max_num_seqs": self.config.vllm.max_num_seqs,
                 "dtype": self.config.vllm.dtype,

codai/ocr/workers/ocr_worker.py

@@ -16,6 +16,7 @@
 # argv[1] = engine name ("paddle" | "surya").
 
 import base64
+import os
 import io
 import json
 import sys
@@ -353,6 +354,70 @@ def _reassemble(tokens) -> str:
     return None if UNREADABLE in text else text
 
 
+#: vLLM renders a token it cannot JSON-encode as this, when it is launched with
+#: ``--return-tokens-as-token-ids``. Those are exactly the tokens that lose
+#: surya's accents: byte-level pieces that are not characters on their own.
+_TOKEN_ID_PREFIX = "token_id:"
+
+
+def _token_ids(tokens):
+    """The raw vocabulary ids behind a reply, or None if they were not reported.
+
+    This is the way out of the dead end the ``bytes`` field leads to. vLLM 0.27.1
+    reports ``bytes: []`` for surya-ocr-2's byte-level tokens and renders them as
+    U+FFFD in the text, so neither output carries the character -- but the IDS are
+    not lossy, and surya's own tokenizer decodes them correctly
+    (``decode([2131, 2104]) == 'e-grave'``, measured). vLLM will report them as
+    ``token_id:2131`` when its server is launched with
+    ``--return-tokens-as-token-ids``.
+
+    All or nothing: a partial id list would decode to a partial page.
+    """
+    if not tokens:
+        return None
+    ids = []
+    for t in tokens:
+        tok = (t.get("token") if isinstance(t, dict)
+               else getattr(t, "token", None))
+        if not isinstance(tok, str) or not tok.startswith(_TOKEN_ID_PREFIX):
+            return None
+        try:
+            ids.append(int(tok[len(_TOKEN_ID_PREFIX):]))
+        except ValueError:
+            return None
+    return ids or None
+
+
+def _decode_ids(ids):
+    """``ids`` through surya's OWN tokenizer, which round-trips accents.
+
+    Imported here rather than at module scope: this worker must start even when
+    the surya venv is mid-build, and a page that needs no repair must not pay for
+    loading a tokenizer. Cached on the function because a document is many pages.
+    """
+    if not ids:
+        return None
+    tok = getattr(_decode_ids, "_tok", None)
+    if tok is None:
+        try:
+            from transformers import AutoTokenizer
+            ckpt = os.environ.get("SURYA_MODEL_CHECKPOINT") or "datalab-to/surya-ocr-2"
+            tok = AutoTokenizer.from_pretrained(ckpt, trust_remote_code=True)
+        except Exception as e:
+            print(f"[surya] cannot load the tokenizer for id decoding: {e}",
+                  file=sys.stderr)
+            _decode_ids._tok = False
+            return None
+        _decode_ids._tok = tok
+    if tok is False:
+        return None
+    try:
+        return tok.decode(ids, skip_special_tokens=True)
+    except Exception as e:
+        print(f"[surya] id decoding failed: {e}", file=sys.stderr)
+        return None
+
+
 def repair_content(raw: str, tokens):
     """``(repaired_text, characters_recovered)`` for a reply whose text lost
     characters, or ``(None, 0)`` to leave it alone.
@@ -371,7 +436,13 @@ def repair_content(raw: str, tokens):
     """
     if not raw or UNREADABLE not in raw:
         return None, 0
-    fixed = _reassemble(tokens)
+    # IDS FIRST. They are the only lossless channel on vLLM 0.27.1, which reports
+    # `bytes: []` for precisely the tokens whose characters went missing. The byte
+    # route is kept because it needs no server flag and works wherever the bytes
+    # are honestly reported.
+    fixed = _decode_ids(_token_ids(tokens))
+    if fixed is None or UNREADABLE in fixed:
+        fixed = _reassemble(tokens)
     if fixed is None:
         return None, 0
     if _ascii_skeleton(fixed) != _ascii_skeleton(raw):

tests/test_ocr_accent_repair.py

@@ -297,3 +297,110 @@ def test_the_existing_vllm_image_is_unchanged():
     """A model on `coderai-vllm` must not be quietly moved to the pinned image."""
     from codai.api.runpod_worker import default_capability_image
     assert default_capability_image("vllm") == "ghcr.io/nextime/coderai-vllm:latest"
+
+
+# ------------------- token ids: the lossless channel on vLLM 0.27.1 #
+def test_token_ids_are_read_from_the_vllm_form():
+    """vLLM renders a token it cannot JSON-encode as `token_id:N` when launched
+    with --return-tokens-as-token-ids. Those are exactly surya's byte-level
+    tokens — the ones it otherwise reports as `bytes: []` and U+FFFD."""
+    toks = [{"token": "token_id:2131"}, {"token": "token_id:2104"}]
+    assert W._token_ids(toks) == [2131, 2104]
+
+
+def test_rendered_tokens_are_not_mistaken_for_ids():
+    assert W._token_ids([{"token": "a"}, {"token": "token_id:5"}]) is None
+    assert W._token_ids([{"token": "token_id:x"}]) is None
+    assert W._token_ids([]) is None
+    assert W._token_ids(None) is None
+
+
+def test_ids_are_all_or_nothing():
+    """A partial id list would decode to a partial page, which is worse than no
+    repair at all."""
+    assert W._token_ids([{"token": "token_id:1"}, {"token": "b"}]) is None
+
+
+def test_the_repair_prefers_ids_over_bytes(monkeypatch):
+    """On 0.27.1 the bytes are empty for the very tokens that matter, so the id
+    route has to be tried first or the repair falls back to a dead end."""
+    UN = "�"
+    monkeypatch.setattr(W, "_decode_ids", lambda ids: "società" if ids else None)
+    toks = [{"token": "token_id:1", "bytes": []}]
+    fixed, n = W.repair_content("societ" + UN, toks)
+    assert fixed == "società"
+    assert n == 1
+
+
+def test_a_bad_id_decode_falls_back_to_the_byte_route(monkeypatch):
+    """The byte route needs no server flag, so it stays as the fallback for any
+    stack that reports bytes honestly."""
+    UN = "�"
+    monkeypatch.setattr(W, "_decode_ids", lambda ids: None)
+    good = "società"
+    toks = [{"token": "x", "bytes": [b]} for b in good.encode()]
+    fixed, _ = W.repair_content("societ" + UN, toks)
+    assert fixed == good
+
+
+def test_an_id_decode_that_still_contains_a_marker_is_rejected(monkeypatch):
+    UN = "�"
+    monkeypatch.setattr(W, "_decode_ids", lambda ids: "societ" + UN)
+    assert W.repair_content("societ" + UN, [{"token": "token_id:1"}]) == (None, 0)
+
+
+def test_the_skeleton_guard_still_applies_to_the_id_route(monkeypatch):
+    """Ids are lossless but the decode could still be of the wrong call."""
+    UN = "�"
+    monkeypatch.setattr(W, "_decode_ids", lambda ids: "qualcosa di diverso")
+    assert W.repair_content("societ" + UN, [{"token": "token_id:1"}]) == (None, 0)
+
+
+# ------------- the other half: the proxy must stop dropping the field #
+def test_the_vllm_backend_forwards_the_token_ids_request():
+    """MEASURED: sending `return_tokens_as_token_ids` to a pod changed nothing,
+    because `_chat_payload` builds an explicit whitelist and silently dropped it —
+    the exact failure its own neighbouring comment warns about ("a proxy that
+    edits the request decides for it")."""
+    from codai.backends.vllm import VllmBackend
+    from codai.config import VllmConfig
+
+    be = VllmBackend.__new__(VllmBackend)
+    be._cfg = VllmConfig()
+    be._model_id = "m"
+    be._served_name = "m"
+
+    off = be._chat_payload([], None, 0.0, 1.0, None, False, logprobs=True)
+    assert "return_tokens_as_token_ids" not in off
+
+    be._cfg.return_token_ids = True
+    on = be._chat_payload([], None, 0.0, 1.0, None, False, logprobs=True)
+    assert on["return_tokens_as_token_ids"] is True
+
+
+def test_the_field_is_only_sent_with_logprobs():
+    """It describes how logprob tokens are rendered, so without logprobs it is
+    noise on the wire."""
+    from codai.backends.vllm import VllmBackend
+    from codai.config import VllmConfig
+
+    be = VllmBackend.__new__(VllmBackend)
+    be._cfg = VllmConfig()
+    be._cfg.return_token_ids = True
+    be._model_id = be._served_name = "m"
+    assert "return_tokens_as_token_ids" not in be._chat_payload(
+        [], None, 0.0, 1.0, None, False)
+
+
+def test_it_is_off_by_default():
+    """It changes what the `token` field looks like, and only one consumer reads
+    it."""
+    from codai.config import VllmConfig
+    assert VllmConfig().return_token_ids is False
+
+
+def test_it_can_be_set_per_model():
+    """So the one checkpoint with a byte-level vocabulary can ask for ids without
+    changing what every other vLLM model reports."""
+    from codai.backends.overrides import VLLM_FIELDS
+    assert "return_token_ids" in VLLM_FIELDS