Italian accents were lost again on surya: a measured page came back with 25
U+FFFD, **zero** 0xC3 lead bytes and a surviving 3-byte em-dash, while chandra
read the identical pixels with 40 0xC3 bytes and every accent intact. So the
2-byte UTF-8 sequences were lost in DETOKENISATION, not misread from the page --
which is what `ocr.surya_repair_accents` exists to undo, and it had been shipped
and verified months earlier.

It had no input. The repair rebuilds the text from the model's own token IDS,
the only lossless channel: vLLM 0.27.1 reports `bytes: []` for exactly the
byte-level tokens that lose their characters (257 on one page, not one byte
>= 0x80), so the byte fallback is dead there, and ids arrive only when vLLM is
launched with `--return-tokens-as-token-ids`.

That flag WAS configured -- `vllm.extra_args` on the renting install -- and the
pods never got it. A pod launches its own vLLM from its own seeded config, the
seeder sent only CODERAI_VLLM_ENABLE_LORA and _VENV, and there was no pod-side
reader for `extra_args` or `return_token_ids` either. The pod's launch line
proves it: `--trust-remote-code` present (the launcher adds that itself), the
flag absent. The repair therefore had no working channel at all, and said
nothing, because its diagnostics could not escape the pod either (below).

Fixed at both doors, because the operator had already done their part:

* `--return-tokens-as-token-ids` is FORCED for a byte-level OCR model
  (surya/chandra/olmOCR, matched on served name or path), deduplicated against
  `extra_args` so the original fix cannot pass it twice. A plain LLM is
  untouched unless asked. A guard that depends on every install remembering a
  flag is not a guard.
* the operator's own `vllm.return_token_ids` now travels:
  CODERAI_VLLM_RETURN_TOKEN_IDS is seeded and read on the pod.

And the reason a verified fix could regress in silence:

* **the worker could not be heard.** `stderr=None` let it inherit the engine's
  raw fd 2, while a pod's log ring tees the Python `sys.stdout`/`sys.stderr`
  OBJECTS -- a subprocess holding the descriptor writes straight past the tee.
  `grep -r "\[surya\]"` across 343 drained pod logs returned nothing. It now
  gets its own pipe with a daemon pump that re-prints through the teed stream.
  NOT merged into stdout: that pipe carries the newline-JSON protocol.
* **the diagnosis was opt-in.** The byte trace -- the only thing that separates
  "the model emitted broken bytes" from "our serving path mis-assembled correct
  ones" -- was gated on `surya_token_trace`, so a page full of markers came
  back unexplained unless someone had set a flag BEFORE the run. It now runs
  whenever the markers are present, which costs nothing on a clean page.
* **nothing announced it.** A corrupt page now prints one line naming what was
  lost, whether the repair ran, recovered nothing, or is off, and the byte
  trace's verdict. A missing hook target is loud and on stdout.

21 new tests. 3432 passed, 13 pre-existing failures (broker, studio, manual
client) untouched.

NOT yet observed on a real page: the chain is proven in tests and the missing
input restored, but the flag is applied at POD LAUNCH, so it takes this image
plus a freshly rented pod.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM