The byte channel is a dead end on 0.27.1 -- measured: `content` comes back with
one U+FFFD per accented letter AND the per-token `bytes` come back EMPTY (257 of
them on one page, no byte >= 0x80 anywhere), so neither output carries the
character. But the token IDS are not lossy, and surya's OWN tokenizer decodes
them correctly: decode([2131, 2104]) == 'e-grave', measured in its venv.

So the repair now asks for ids and decodes them itself, bypassing vLLM's
detokenizer entirely. Both halves, because each alone does nothing:

1. THE WORKER reads vLLM's `token_id:{id}` form, decodes with surya's tokenizer
   (lazily loaded and cached -- the worker must start while the venv is building,
   and a clean page must not pay for a tokenizer), and keeps the byte route as the
   fallback for any stack that reports bytes honestly. Ids are tried FIRST because
   on 0.27.1 the bytes lead nowhere. All-or-nothing on the id list: a partial one
   would decode to a partial page. The ASCII-skeleton guard still applies, so a
   decode of the wrong call cannot be substituted.

2. THE PROXY stops dropping the request. `return_tokens_as_token_ids` is a
   per-request field on vLLM's ChatCompletionRequest, but sending it to a pod
   changed nothing -- because codai/backends/vllm.py::_chat_payload builds an
   explicit whitelist and silently discarded it. That is exactly the failure its
   own neighbouring comment warns about ("a proxy that edits the request decides
   for it"), five lines further up. Now forwarded, gated on
   `vllm.return_token_ids`, and only alongside logprobs since it describes how
   logprob tokens are rendered.

`vllm.return_token_ids` is off by default (it changes what the `token` field
looks like, and only this one consumer reads it) and is in VLLM_FIELDS, so the
single checkpoint with a byte-level vocabulary can ask for ids without changing
what every other vLLM model reports. A config field rather than relaunching the
server with --return-tokens-as-token-ids: same effect, no pod rebuild.

This is a fix ON 0.27.1. It makes the coderai-vllm-ocr image (vLLM 0.20.1) a
CONTROL that confirms the diagnosis rather than the only way out.

tests: +11 in test_ocr_accent_repair.py (36 -> 47) covering the id form, the
all-or-nothing rule, ids-before-bytes, the fallback when a decode fails, the
skeleton guard on the id route, and the proxy forwarding with its logprobs
gating. 912 passed across the ocr/front/config/admin/runpod/vllm/backend
selection; the 9 failures are pre-existing (broker x6, studio, whisper x2) plus
test_cluster's RPC check, which depends on llama_cpp being installed in this venv
and touches nothing changed here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW