the accent repair had no input on a pod (v0.4.14)

Italian accents were lost again on surya: a measured page came back with 25
U+FFFD, **zero** 0xC3 lead bytes and a surviving 3-byte em-dash, while chandra
read the identical pixels with 40 0xC3 bytes and every accent intact. So the
2-byte UTF-8 sequences were lost in DETOKENISATION, not misread from the page --
which is what `ocr.surya_repair_accents` exists to undo, and it had been shipped
and verified months earlier.

It had no input. The repair rebuilds the text from the model's own token IDS,
the only lossless channel: vLLM 0.27.1 reports `bytes: []` for exactly the
byte-level tokens that lose their characters (257 on one page, not one byte
>= 0x80), so the byte fallback is dead there, and ids arrive only when vLLM is
launched with `--return-tokens-as-token-ids`.

That flag WAS configured -- `vllm.extra_args` on the renting install -- and the
pods never got it. A pod launches its own vLLM from its own seeded config, the
seeder sent only CODERAI_VLLM_ENABLE_LORA and _VENV, and there was no pod-side
reader for `extra_args` or `return_token_ids` either. The pod's launch line
proves it: `--trust-remote-code` present (the launcher adds that itself), the
flag absent. The repair therefore had no working channel at all, and said
nothing, because its diagnostics could not escape the pod either (below).

Fixed at both doors, because the operator had already done their part:

* `--return-tokens-as-token-ids` is FORCED for a byte-level OCR model
  (surya/chandra/olmOCR, matched on served name or path), deduplicated against
  `extra_args` so the original fix cannot pass it twice. A plain LLM is
  untouched unless asked. A guard that depends on every install remembering a
  flag is not a guard.
* the operator's own `vllm.return_token_ids` now travels:
  CODERAI_VLLM_RETURN_TOKEN_IDS is seeded and read on the pod.

And the reason a verified fix could regress in silence:

* **the worker could not be heard.** `stderr=None` let it inherit the engine's
  raw fd 2, while a pod's log ring tees the Python `sys.stdout`/`sys.stderr`
  OBJECTS -- a subprocess holding the descriptor writes straight past the tee.
  `grep -r "\[surya\]"` across 343 drained pod logs returned nothing. It now
  gets its own pipe with a daemon pump that re-prints through the teed stream.
  NOT merged into stdout: that pipe carries the newline-JSON protocol.
* **the diagnosis was opt-in.** The byte trace -- the only thing that separates
  "the model emitted broken bytes" from "our serving path mis-assembled correct
  ones" -- was gated on `surya_token_trace`, so a page full of markers came
  back unexplained unless someone had set a flag BEFORE the run. It now runs
  whenever the markers are present, which costs nothing on a clean page.
* **nothing announced it.** A corrupt page now prints one line naming what was
  lost, whether the repair ran, recovered nothing, or is off, and the byte
  trace's verdict. A missing hook target is loud and on stdout.

21 new tests. 3432 passed, 13 pre-existing failures (broker, studio, manual
client) untouched.

NOT yet observed on a real page: the chain is proven in tests and the missing
input restored, but the flag is applied at POD LAUNCH, so it takes this image
plus a freshly rented pod.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.13"
+__version__ = "0.4.14"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/runpod_worker.py

@@ -1475,6 +1475,14 @@ def _plan(engine, image, args, mcfg, api_key, entry, served,
         # This image carries NO docTR -- `auto` would route a failed page to an
         # engine that is not there. This engine or nothing, said out loud.
         env["CODERAI_OCR_FALLBACK_ENGINES"] = "off"
+        # The pod launches its OWN vLLM from its OWN config, so a vLLM setting
+        # made on the renting install does not reach it. That is not a detail:
+        # `--return-tokens-as-token-ids` is what the accent repair rebuilds
+        # from, an operator had set it in `vllm.extra_args` here, and the pods
+        # silently ran without it -- a fixed bug, back, invisibly. The flag is
+        # now forced for byte-level OCR models in vllm_worker, and the
+        # operator's own choice travels too, so neither door is the only one.
+        env["CODERAI_VLLM_RETURN_TOKEN_IDS"] = "1"
         # Same reasoning for the degraded-page re-read, and one more: only the
         # ORCHESTRATOR can reach another engine's pods. A pod re-reading its own
         # degraded page would re-read it on the one engine that just produced

codai/api/vllm_worker.py

@@ -261,10 +261,59 @@ def _launch_cmd(py, cfg, host: str, port: int, model_path: str,
     # was launched for it, and the message names flags the caller cannot set.
     cmd += _tool_args(cfg, model_config, model_path)
 
+    # TOKEN IDS IN LOGPROBS. `ocr.surya_repair_accents` rebuilds a reply whose
+    # accents were lost in detokenisation from the model's own token ids, and
+    # vLLM only reports ids (`token_id:2131`) when launched with this flag --
+    # `bytes` is empty for exactly the tokens that lost their characters, so
+    # without it there is nothing to rebuild from and the repair silently does
+    # nothing.
+    #
+    # Why it is forced rather than configured: the operator DID configure it
+    # (`vllm.extra_args` on the renting install) and it still regressed, because
+    # a rented pod launches its own vLLM from its own seeded config and that
+    # setting never travelled. A page then came back with 25 replacement
+    # characters, zero 0xC3 bytes and a surviving 3-byte em-dash -- the repair
+    # unable to run, on a fix that had been shipped and verified months
+    # earlier. A guard that depends on remembering to set it on every install
+    # is not a guard.
+    #
+    # Safe to force: it changes only the `token` STRING inside logprobs, which
+    # nothing in coderai reads except that repair.
+    if _wants_token_ids(cfg, served_name, model_path):
+        if "--return-tokens-as-token-ids" not in cmd:
+            cmd += ["--return-tokens-as-token-ids"]
+
     extra = (getattr(cfg, "extra_args", "") or "").strip()
     if extra:
         cmd += shlex.split(extra)
-    return cmd
+    # Deduplicate: an operator who also put the flag in `extra_args` (as the
+    # original fix did) must not launch vLLM with it twice.
+    out, seen = [], set()
+    for tok in cmd:
+        if tok == "--return-tokens-as-token-ids":
+            if tok in seen:
+                continue
+            seen.add(tok)
+        out.append(tok)
+    return out
+
+
+#: Served names (and path fragments) whose vocabulary is BYTE-level, so a lost
+#: character can only be recovered from token ids. Matched loosely because a
+#: model arrives here as a repo id, a served name or a local path.
+_BYTE_LEVEL_OCR = ("surya", "chandra", "olmocr")
+
+
+def _wants_token_ids(cfg, served_name: str = "", model_path: str = "") -> bool:
+    """Whether this engine must report token ids in its logprobs.
+
+    True when the operator asked (`vllm.return_token_ids`), and ALWAYS for a
+    byte-level OCR model, whose accent repair cannot work without them.
+    """
+    if bool(getattr(cfg, "return_token_ids", False)):
+        return True
+    hay = f"{served_name or ''} {model_path or ''}".lower()
+    return any(name in hay for name in _BYTE_LEVEL_OCR)
 
 
 def _base_of(entry: dict) -> str:

codai/config.py

@@ -2385,6 +2385,13 @@ class ConfigManager:
                     ("CODERAI_OCR_SURYA_USE_GPU", "surya_use_gpu")):
                 if _os.environ.get(_env) is not None:
                     setattr(self.config.ocr, _attr, _flag(_env))
+            # vLLM settings a POD needs but cannot inherit: it launches its own
+            # engine from its own config. `return_token_ids` is the one the
+            # accent repair depends on -- see vllm_worker._wants_token_ids for
+            # what its absence cost.
+            if _os.environ.get("CODERAI_VLLM_RETURN_TOKEN_IDS") is not None:
+                self.config.vllm.return_token_ids = _flag(
+                    "CODERAI_VLLM_RETURN_TOKEN_IDS")
             if _flag("CODERAI_OCR_SURYA_ACCEPT_LICENSE"):
                 # Surya is GPL and gated on an explicit acceptance. The pod
                 # inherits the decision made here; it cannot make it itself.

codai/ocr/subprocess_engine.py

@@ -145,6 +145,38 @@ class SubprocessOcrEngine(OcrEngine):
             status=503,
         )
 
+    def _pump_stderr(self, proc) -> None:
+        """Re-print the worker's stderr through THIS process's stdout.
+
+        Through the Python stream on purpose: that is the one the pod's log ring
+        tees, so the line reaches /logs, the drained pod log, and the
+        orchestrator's own output in one step. Prefixed with the engine so a
+        mixed log stays readable.
+
+        A daemon thread, and it never raises: this is a diagnostic path, and a
+        diagnostic that can break a page is worse than no diagnostic. It also
+        has to keep draining, because a full stderr pipe would eventually block
+        the worker mid-page.
+        """
+        import threading
+
+        def _run():
+            try:
+                for line in iter(proc.stderr.readline, ""):
+                    line = (line or "").rstrip()
+                    if line:
+                        print(f"[{self.name}:worker] {line}", flush=True)
+            except Exception:
+                pass
+            finally:
+                try:
+                    proc.stderr.close()
+                except Exception:
+                    pass
+
+        threading.Thread(target=_run, name=f"ocr-{self.name}-stderr",
+                         daemon=True).start()
+
     def load(self) -> None:
         py = self._ensure_venv()
         env = dict(os.environ)
@@ -153,8 +185,23 @@ class SubprocessOcrEngine(OcrEngine):
             self._proc = subprocess.Popen(
                 [py, _WORKER, self._worker_engine],
                 stdin=subprocess.PIPE, stdout=subprocess.PIPE,
-                stderr=None, text=True, bufsize=1, env=env,
+                # ITS OWN PIPE, pumped below. `stderr=None` let the worker
+                # inherit this process's fd 2 -- which on a rented POD writes
+                # straight past the log ring: `podlog.install()` tees the
+                # Python `sys.stdout`/`sys.stderr` OBJECTS, and a subprocess
+                # holding the raw descriptor never passes through them. So
+                # every diagnostic the worker printed was invisible to /logs
+                # and therefore absent from the drained pod log -- including
+                # "token trace: no _generate_one to hook", which is exactly the
+                # line that would have said why a solved accent bug was back.
+                # `grep -r "\[surya\]"` across 343 pod logs returned nothing.
+                #
+                # NOT merged into stdout: that pipe carries the newline-JSON
+                # protocol, and a log line where a reply belongs would
+                # desynchronise it.
+                stderr=subprocess.PIPE, text=True, bufsize=1, env=env,
             )
+            self._pump_stderr(self._proc)
         except Exception as e:
             raise OcrError(f"{self.name}: failed to launch worker: {e}", status=500)
         # Handshake: wait for {"ready": true}

codai/ocr/workers/ocr_worker.py

@@ -458,6 +458,15 @@ class _LastMime(threading.local):
 _LAST_MIME = _LastMime()
 
 
+def _oc_repair_on() -> bool:
+    """Whether the accent repair is active in this process, for reporting."""
+    try:
+        from surya.inference.backends import openai_client as _oc
+        return bool(getattr(_oc, "_coderai_repair", False))
+    except Exception:
+        return False
+
+
 def _note_loop(raw: str) -> None:
     """Record that this page's generation ended still repeating itself.
 
@@ -1413,7 +1422,9 @@ class SuryaWorker:
         try:
             from surya.inference.backends import openai_client as _oc
         except Exception as e:
-            print(f"[surya] token trace unavailable: {e}", file=sys.stderr)
+            print(f"[surya] ACCENT REPAIR AND TOKEN TRACE UNAVAILABLE: {e} — "
+                  f"a reply whose accents are lost in detokenisation cannot be "
+                  f"rebuilt from its own bytes", flush=True)
             return
         _oc._coderai_repair = self._repair
         if getattr(_oc, "_coderai_traced", False):
@@ -1422,7 +1433,12 @@ class SuryaWorker:
             return
         inner = getattr(_oc, "_generate_one", None)
         if inner is None:
-            print("[surya] token trace: no _generate_one to hook", file=sys.stderr)
+            print("[surya] ACCENT REPAIR AND TOKEN TRACE ARE OFF: surya's "
+                  "openai_client has no `_generate_one` to hook in this "
+                  "release, so a reply whose accents are lost in "
+                  "detokenisation cannot be rebuilt from its own bytes. This "
+                  "is the one line that explains an accent regression.",
+                  flush=True)
             return
 
         def _wrapped(*args, **kwargs):
@@ -1787,7 +1803,15 @@ class SuryaWorker:
                                "repeated itself; each re-run is a full "
                                "generation at a higher temperature")
             page["model_attempts"] = info
-        if self._trace and UNREADABLE in page["text"]:
+        # NOT gated on `self._trace` any more. The trace used to require an
+        # opt-in flag, so a page full of replacement characters came back with
+        # no explanation unless somebody had remembered to turn the flag on
+        # BEFORE the run -- which is how a fixed accent bug came back and went
+        # unnoticed: the symptom was in the text, the diagnosis was switched
+        # off, and nothing in between said so. The cost is paid only on a page
+        # that already has markers in it, which is exactly when it is worth
+        # paying.
+        if UNREADABLE in page["text"]:
             try:
                 page["unreadable_trace"] = _trace_report(calls)
             except Exception as e:
@@ -1798,6 +1822,21 @@ class SuryaWorker:
                     "why": f"trace analysis failed: {e}",
                     "findings": [],
                 }
+            # On STDOUT, because that is the stream a rented pod's log ring
+            # tees and the renting install drains. One line per affected page,
+            # naming what was lost, whether the repair ran and what the bytes
+            # say -- the line whose absence let this regress unseen.
+            _tr = page.get("unreadable_trace") or {}
+            _n = page["text"].count(UNREADABLE)
+            _rep = _trace_repairs()
+            _fixed = sum(int(r.get("recovered") or 0) for r in (_rep or []))
+            print(f"[surya] page kept {_n} replacement character(s) after "
+                  f"reading; accent repair "
+                  + (f"recovered {_fixed}" if _fixed else
+                     ("ran and recovered nothing" if _oc_repair_on() else "is OFF"))
+                  + f"; byte trace: conclusive={_tr.get('conclusive')} "
+                  f"blame={_tr.get('blame') or _tr.get('why') or 'unreported'}",
+                  flush=True)
         return page
 
 

tests/test_ocr_accent_token_ids.py

+"""The accent repair's input: token ids in vLLM's logprobs.
+
+`ocr.surya_repair_accents` rebuilds a reply whose accents were lost in
+detokenisation from the model's own token IDS -- the only lossless channel,
+because vLLM 0.27.1 reports `bytes: []` for exactly the byte-level tokens that
+lost their characters. vLLM reports ids (`token_id:2131`) only when launched
+with `--return-tokens-as-token-ids`.
+
+That flag was configured on the renting install (`vllm.extra_args`) and the fix
+was verified. Then the OCR pipeline moved onto rented pods, and a pod launches
+its OWN vLLM from its OWN seeded config -- which never carried the setting. The
+pod's launch line read:
+
+    --model datalab-to/surya-ocr-2 --max-model-len 32768
+    --gpu-memory-utilization 0.9 --tensor-parallel-size 1 --trust-remote-code
+
+no flag, so no ids, so nothing to repair from, so a page came back with 25
+replacement characters, zero 0xC3 lead bytes and a surviving 3-byte em-dash --
+while chandra read the same page with 40 0xC3 bytes and every accent intact.
+
+A guard that depends on each install remembering a flag is not a guard, so the
+flag is now forced for byte-level OCR models AND the operator's own setting
+travels to pods. Either alone would have prevented this.
+"""
+
+import types
+
+import pytest
+
+from codai.api.vllm_worker import _launch_cmd, _wants_token_ids
+
+FLAG = "--return-tokens-as-token-ids"
+
+
+def _cfg(**kw):
+    base = dict(extra_args="", return_token_ids=False, model_id="", dtype="",
+                quantization="", max_num_seqs=0, distributed_executor_backend="",
+                gpu_memory_utilization=0.9, ctx=32768, tensor_parallel_size=1)
+    base.update(kw)
+    return types.SimpleNamespace(**base)
+
+
+# ------------------------------------------------------- forced where needed #
+
+@pytest.mark.parametrize("served", [
+    "datalab-to/surya-ocr-2", "surya-ocr-2", "datalab-to/chandra-ocr-2",
+    "allenai/olmOCR-2-7B-1025-FP8", "SURYA-OCR-2",
+])
+def test_a_byte_level_ocr_model_always_reports_token_ids(served):
+    assert _wants_token_ids(_cfg(), served, "") is True
+    assert FLAG in _launch_cmd("py", _cfg(), "127.0.0.1", 8000, served, served, {})
+
+
+def test_it_is_recognised_from_the_path_too():
+    assert _wants_token_ids(_cfg(), "", "/cache/models/surya-ocr-2") is True
+
+
+def test_a_plain_llm_is_left_alone():
+    """The flag changes the `token` string in logprobs; nothing else reads it,
+    but a change nobody needs is still a change nobody asked for."""
+    assert _wants_token_ids(_cfg(), "qwen38-awq", "") is False
+    assert FLAG not in _launch_cmd("py", _cfg(), "127.0.0.1", 8000,
+                                   "qwen38-awq", "qwen38-awq", {})
+
+
+def test_the_operator_can_still_ask_for_any_model():
+    assert _wants_token_ids(_cfg(return_token_ids=True), "qwen38-awq", "") is True
+    cmd = _launch_cmd("py", _cfg(return_token_ids=True), "127.0.0.1", 8000,
+                      "qwen38-awq", "qwen38-awq", {})
+    assert FLAG in cmd
+
+
+def test_it_is_never_passed_twice():
+    """The original fix put it in extra_args; launching vLLM with a duplicate
+    flag is at best noise."""
+    cmd = _launch_cmd("py", _cfg(extra_args=f"--trust-remote-code {FLAG}"),
+                      "127.0.0.1", 8000, "datalab-to/surya-ocr-2",
+                      "datalab-to/surya-ocr-2", {})
+    assert cmd.count(FLAG) == 1
+    assert "--trust-remote-code" in cmd
+
+
+# --------------------------------------------- and the setting reaches a pod #
+
+def test_the_knob_is_seeded_to_pods():
+    import inspect
+
+    from codai.api import runpod_worker as rw
+    src = inspect.getsource(rw)
+    assert "CODERAI_VLLM_RETURN_TOKEN_IDS" in src
+
+
+def test_a_pod_reads_the_knob(monkeypatch, tmp_path):
+    monkeypatch.setenv("CODERAI_OCR_ENABLED", "1")
+    monkeypatch.setenv("CODERAI_OCR_SURYA_ENABLED", "1")
+    monkeypatch.setenv("CODERAI_VLLM_RETURN_TOKEN_IDS", "1")
+    from codai.config import ConfigManager
+    cm = ConfigManager(str(tmp_path))
+    cm.load()
+    assert cm.config.vllm.return_token_ids is True
+
+
+def test_the_repair_still_needs_the_ids_it_is_given():
+    """The repair's own contract, so this test file stays honest about what the
+    flag buys: ids in, repaired text out; no ids, no repair."""
+    from codai.ocr.workers.ocr_worker import repair_content, UNREADABLE
+    # Nothing to work from -> left alone, never guessed at.
+    assert repair_content(f"societ{UNREADABLE}", None) == (None, 0)
+    assert repair_content("nothing wrong here", [{"token": "token_id:1"}]) == (None, 0)

tests/test_ocr_accent_visibility.py

+"""Why a fixed accent bug could come back unseen, and what now prevents it.
+
+The repair exists and works (see test_ocr_accent_repair.py). On 2026-10-17 it
+was not running on rented pods, and NOTHING said so:
+
+* the worker's diagnostics went to STDERR, and `stderr=None` let the worker
+  inherit the engine's raw fd 2 -- while the pod's log ring tees the Python
+  `sys.stdout`/`sys.stderr` OBJECTS. A subprocess holding the descriptor writes
+  straight past the tee, so `grep -r "\\[surya\\]"` across 343 drained pod logs
+  returned nothing at all;
+* the byte trace, the one thing that says whether the model or the serving path
+  lost the characters, was gated on an opt-in flag -- so a page full of
+  replacement characters came back with no explanation unless someone had
+  turned the flag on before the run.
+
+A measured page read `societ<FFFD>` with 25 markers, zero 0xC3 lead bytes and a
+surviving 3-byte em-dash: 2-byte UTF-8 sequences lost in detokenisation, which
+is precisely what the repair is for.
+"""
+
+import inspect
+
+from codai.ocr import subprocess_engine
+from codai.ocr.workers import ocr_worker
+
+
+# --------------------------------------------------- the worker can be heard #
+
+def test_the_workers_stderr_is_captured_not_inherited():
+    """Asserted on CODE lines, not on prose: the comment that explains the bug
+    quotes `stderr=None`, and a test that cannot tell the two apart fails for
+    the wrong reason (it did)."""
+    src = inspect.getsource(subprocess_engine.SubprocessOcrEngine.load)
+    code = [ln.split("#", 1)[0] for ln in src.splitlines()]
+    code = "\n".join(ln for ln in code if ln.strip())
+    assert "stderr=subprocess.PIPE" in code
+    assert "stderr=None" not in code
+
+
+def test_the_stderr_pump_exists_and_is_a_daemon():
+    src = inspect.getsource(subprocess_engine.SubprocessOcrEngine._pump_stderr)
+    assert "daemon=True" in src
+    # Through print(), i.e. the Python stream the ring tees -- not os.write.
+    assert "print(" in src
+    assert "flush=True" in src
+
+
+def test_stderr_is_not_merged_into_the_protocol_pipe():
+    """stdout carries newline-JSON; a log line there would desynchronise it."""
+    src = inspect.getsource(subprocess_engine.SubprocessOcrEngine.load)
+    assert "stderr=subprocess.STDOUT" not in src
+
+
+def test_the_pump_keeps_draining(monkeypatch):
+    """A full stderr pipe would eventually block the worker mid-page."""
+    import io
+    import threading
+
+    class _Proc:
+        def __init__(self):
+            self.stderr = io.StringIO("[surya] one\n[surya] two\n")
+
+    eng = subprocess_engine.SubprocessOcrEngine.__new__(
+        subprocess_engine.SubprocessOcrEngine)
+    eng.name = "surya"
+    seen = []
+    monkeypatch.setattr("builtins.print", lambda *a, **k: seen.append(a[0] if a else ""))
+    eng._pump_stderr(_Proc())
+    for _ in range(50):
+        if len(seen) >= 2:
+            break
+        threading.Event().wait(0.02)
+    assert any("one" in s for s in seen) and any("two" in s for s in seen)
+    assert all(s.startswith("[surya:worker]") for s in seen)
+
+
+# ------------------------------------------- the diagnosis is not opt-in now #
+
+def test_the_byte_trace_runs_whenever_markers_are_present():
+    """It used to require `token_trace`, so the diagnosis was missing exactly
+    when it was needed."""
+    src = inspect.getsource(ocr_worker.SuryaWorker.ocr)
+    assert "if UNREADABLE in page[\"text\"]:" in src
+    assert "if self._trace and UNREADABLE" not in src
+
+
+def test_a_corrupt_page_reports_itself_on_stdout():
+    src = inspect.getsource(ocr_worker.SuryaWorker.ocr)
+    i = src.index("replacement character(s) after")
+    line = src[i - 400:i + 500]
+    assert "flush=True" in line
+    assert "file=sys.stderr" not in line
+    # It must name all three things a reader needs.
+    assert "accent repair" in line
+    assert "byte trace" in line
+    assert "blame" in line
+
+
+def test_the_report_distinguishes_repaired_from_off_from_ineffective():
+    src = inspect.getsource(ocr_worker.SuryaWorker.ocr)
+    for phrase in ("recovered", "ran and recovered nothing", "is OFF"):
+        assert phrase in src, phrase
+
+
+def test_a_missing_hook_target_is_loud_and_on_stdout():
+    """The single line that explains an accent regression; it used to go to a
+    stream a pod throws away."""
+    src = inspect.getsource(ocr_worker.SuryaWorker._install_trace_hook)
+    assert "ACCENT REPAIR AND TOKEN TRACE ARE OFF" in src
+    i = src.index("ACCENT REPAIR AND TOKEN TRACE ARE OFF")
+    assert "file=sys.stderr" not in src[i:i + 400]
+
+
+def test_repair_state_is_reportable():
+    assert ocr_worker._oc_repair_on() in (True, False)