ocr: surface the model's own bytes where a page came back unreadable (v0.3.25)

`ocr.surya_token_trace` (off by default) reports the per-token bytes at every
place a surya page lost a character. It exists to answer one question that the
finished text cannot: whose fault is it?

  invalid-utf8   the reassembled stream does not decode -- a lone UTF-8 lead
                 byte with nothing to complete it. The MODEL emitted a broken
                 sequence; the text faithfully renders a wrong answer.
  model-emitted  the stream decodes and a token IS the replacement character.
                 The model chose it.
  no findings    the bytes were sound and the markers appeared anyway, so the
                 loss happened AFTER them -- OURS, in the serving stack.

That last case is why an empty trace is a result, not a failure.

CORRECTION, and the reason this was needed. v0.3.21 claimed U+FFFD could not be
a byte-level artefact, on the grounds that `a`-grave was a single token so one
marker had to mean the model chose it. That was wrong. Measured:

  'a-grave' utf8=[195, 160]  ids=[2131, 2258]  pieces=['A-tilde', 'l-bar']
  decode([2131])       = U+FFFD    <- a lone lead byte, exactly ONE marker
  decode([2131, 2104]) = 'e-grave'

surya-ocr-2's vocabulary is BYTE-level, so one marker per accented letter is
precisely the signature of a byte-level failure -- the thing it was claimed to
rule out. The vocab lookup was misread too: `vocab['a-grave'] = 2160` does not
mean the letter is token 2160; in a byte-mapped vocabulary that key is the glyph
standing for byte 0xA0, which alone also decodes to one marker. The
"one marker vs two" discriminator does not exist. Digesta's charset instinct was
closer to the mark than the rebuttal. base.UNREADABLE and docs/ocr.md now carry
the correction rather than the refuted argument.

What IS established, measured on the Aruba install at the same DPI through the
same transport on perfect synthetic Italian (crisp vector glyphs, 34pt, 250 DPI):

    surya    29 U+FFFD markers, zero accents surviving
    chandra   0 markers, all 12 accents intact

So the document is exonerated. Also notable: the em dash survived surya intact
while every two-byte accent died, and surya emitted U+1EEA (Vietnamese) in place
of one -- no decoding bug can fabricate a character that was never in the input.

Implementation notes:

- surya DROPS this data. Its recognition predictor does pass it on
  (`BlockOCRResult(..., raw_logprobs=out.logprobs)`) but the model never declares
  the field and pydantic defaults to extra="ignore", so it is constructed and
  silently discarded -- their own comment calls it "phase 2 use". Verified by
  construction: the attribute does not exist on the result. So the value is taken
  one layer earlier, wrapping `_generate_one`, which still has it. The wrapper
  only reads and swallows every error: a diagnostic that could break a page would
  be worse than no diagnostic.
- The scan reassembles the WHOLE byte stream and walks it as UTF-8 rather than
  judging tokens pairwise. Per token is meaningless here -- one character
  legitimately arrives as two or three tokens, each "invalid" alone -- and the
  first version of this did judge them pairwise and flagged every accent and
  every em dash, the exact false positive that would make it worthless.
- Capped at 40 findings per page, two tokens of context each side, because a lead
  byte is only wrong in the light of what followed it.
- Clean pages are byte-identical to before: the key appears only when the text
  actually contains a marker, and it is reported ONCE document-wide under
  `unreadable.tokens` rather than repeated in every page's meta.
- The flag is in _pool_params, because the hook patches surya's backend at LOAD
  -- without that, toggling it would appear to save and take effect never.

A third discriminator needing no code: serve the same weights through llama.cpp
(`datalab-to/surya-ocr-2-gguf`, `surya_serve: "llamacpp"`). A different
detokenizer on identical weights would convict vLLM outright.

tests/test_ocr_token_trace.py: 30 tests. 818 passed across the OCR/config/admin/
runpod selection with the same 10 pre-existing failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.24"
+__version__ = "0.3.25"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -4139,6 +4139,7 @@ def build_settings_dict(c, gpu_cards):
             "surya_accept_license": c.ocr.surya_accept_license,
             "surya_instances": c.ocr.surya_instances,
             "surya_langs": c.ocr.surya_langs,
+            "surya_token_trace": c.ocr.surya_token_trace,
             "surya_venv": c.ocr.surya_venv,
             "surya_auto_build": c.ocr.surya_auto_build,
             "surya_serve": c.ocr.surya_serve,
@@ -4956,6 +4957,7 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
         if "surya_enabled" in d: o.surya_enabled = bool(d["surya_enabled"])
         if "surya_accept_license" in d: o.surya_accept_license = bool(d["surya_accept_license"])
         if "surya_instances" in d: o.surya_instances = _i("surya_instances", o.surya_instances, 1)
+        if "surya_token_trace" in d: o.surya_token_trace = bool(d["surya_token_trace"])
         if "surya_langs" in d: o.surya_langs = (d.get("surya_langs") or "it").strip()
         if "surya_venv" in d: o.surya_venv = (d.get("surya_venv") or "").strip()
         if "surya_auto_build" in d: o.surya_auto_build = bool(d["surya_auto_build"])

codai/admin/templates/settings.html

@@ -1206,6 +1206,7 @@
       <div class="form-group" style="flex:1;min-width:110px"><label class="form-label">Instances</label><input type="number" id="s-ocr-surya-instances" class="form-input" min="1" placeholder="1"></div>
       <label style="display:flex;align-items:center;gap:.4rem;cursor:pointer" title="Local mode only. On, Surya runs on this machine's GPU or refuses to load — it will not quietly fall back to the CPU, where a page takes minutes. Off, the CPU is used deliberately."><input type="checkbox" id="s-ocr-surya-use-gpu" checked><span>Require GPU (Local)</span></label>
       <div class="form-group" style="flex:1;min-width:120px"><label class="form-label">Langs</label><input type="text" id="s-ocr-surya-langs" class="form-input" placeholder="it"></div>
+      <label style="display:flex;align-items:center;gap:.4rem;cursor:pointer" title="Return the model's own per-token bytes wherever a page came back unreadable. A diagnostic — leave off unless you are chasing lost characters."><input type="checkbox" id="s-ocr-surya-token-trace"><span>Trace unreadable tokens</span></label>
     </div>
     <p class="form-hint" style="margin:0 0 .3rem">Runs in an <b>isolated venv</b> (Surya caps pillow&lt;11, incompatible with the main venv).</p>
     <div class="form-row" style="display:flex;gap:1rem;flex-wrap:wrap;align-items:center">
@@ -2202,6 +2203,7 @@ async function loadSettings(){
     document.getElementById('s-ocr-surya-instances').value = ocr.surya_instances ?? 1;
     document.getElementById('s-ocr-surya-use-gpu').checked = ocr.surya_use_gpu !== false;
     document.getElementById('s-ocr-surya-langs').value = ocr.surya_langs ?? 'it';
+    document.getElementById('s-ocr-surya-token-trace').checked = !!ocr.surya_token_trace;
     document.getElementById('s-ocr-surya-venv').value = ocr.surya_venv ?? '';
     document.getElementById('s-ocr-surya-auto-build').checked = !!ocr.surya_auto_build;
     document.getElementById('s-ocr-surya-serve').value = ocr.surya_serve || 'model';
@@ -2484,6 +2486,7 @@ async function saveSettings(silent){
       surya_accept_license: document.getElementById('s-ocr-surya-license').checked,
       surya_instances: parseInt(document.getElementById('s-ocr-surya-instances').value) || 1,
       surya_langs: document.getElementById('s-ocr-surya-langs').value.trim() || 'it',
+      surya_token_trace: document.getElementById('s-ocr-surya-token-trace').checked,
       surya_venv: document.getElementById('s-ocr-surya-venv').value.trim(),
       surya_auto_build: document.getElementById('s-ocr-surya-auto-build').checked,
       surya_serve: document.getElementById('s-ocr-surya-serve').value,

codai/config.py

@@ -988,6 +988,16 @@ class OcrConfig:
     surya_accept_license: bool = False   # MUST be explicitly set (GPL — compatible with coderai GPLv3)
     surya_instances: int = 1
     surya_langs: str = "it"
+    # Return the model's own per-token bytes for the places a page came back
+    # unreadable (see base.UNREADABLE). Off by default: it is a diagnostic, it
+    # costs response weight, and it reaches into surya's internals to get data
+    # surya itself constructs and then drops. Turn it on when a document is
+    # losing characters and you need to know WHOSE fault it is -- the model
+    # emitting a broken byte sequence, or our serving stack mis-assembling a
+    # correct one. The two are indistinguishable in the finished text, because
+    # surya-ocr-2's vocabulary is byte-level and decoding either byte of an
+    # accented letter alone yields exactly one replacement character.
+    surya_token_trace: bool = False
     surya_venv: str = ""               # isolated venv dir; blank = ~/.coderai/surya_venv
     surya_auto_build: bool = False     # create the venv + pip install requirements-surya.txt on first use
     # Surya serving mode: "local" = classic det+recognition on torch in the isolated venv
@@ -1702,6 +1712,7 @@ class ConfigManager:
                 "surya_accept_license": self.config.ocr.surya_accept_license,
                 "surya_instances": self.config.ocr.surya_instances,
                 "surya_langs": self.config.ocr.surya_langs,
+                "surya_token_trace": self.config.ocr.surya_token_trace,
                 "surya_venv": self.config.ocr.surya_venv,
                 "surya_auto_build": self.config.ocr.surya_auto_build,
                 "surya_use_gpu": self.config.ocr.surya_use_gpu,

codai/ocr/base.py

@@ -44,24 +44,32 @@ class OcrError(RuntimeError):
 
 #: What a VLM OCR engine writes where it cannot identify a character.
 #:
-#: This is NOT a decoding or transport artefact, which is the natural assumption
-#: when accented letters come back as U+FFFD. In surya-ocr-2 it is a real entry in
-#: the vocabulary — id 65421 of 65425, the only id that decodes to it, and
-#: ``<unk>`` is a different token (65424). The tokenizer is ``WordLevel`` and
-#: character-level for Latin text, so ``à`` is a single token and an accent
-#: *cannot* be split across two: no partial-UTF-8 artefact is possible, and
-#: per-token decoding is byte-identical to decoding the whole string. Surya also
-#: reads its text only from ``message.content``, never from the per-token
-#: logprobs, and coderai's OCR gateway forwards the upstream body as raw bytes.
+#: It may be the model's own answer OR a byte-level accident, and the text alone
+#: cannot tell you which. An earlier version of this comment claimed otherwise —
+#: that ``à`` was a single token, so one marker had to mean the model chose it.
+#: That was wrong, and measurement settled it: surya-ocr-2's vocabulary is
+#: BYTE-level, and an accented letter is two tokens.
 #:
-#: So a U+FFFD in the output is the MODEL saying "there is a character here I
-#: cannot read". It is honest, and it is also silent: ``più`` becomes ``pi?`` and
-#: indexes, searches and reads wrong ever after. For Italian judicial documents
-#: that is a real loss, so it gets counted and located rather than passed through
-#: unremarked — a known unknown beats a quiet corruption.
+#:     'à' utf8=[195, 160]  ids=[2131, 2258]  pieces=['Ã', 'ł']
+#:     decode([2131])       = U+FFFD          <- a lone lead byte, ONE marker
+#:     decode([2131, 2104]) = 'è'
 #:
-#: The usual cause is too few pixels: a diacritic sits above the x-height and is
-#: ~3 pixels tall at 200 DPI. See ``OcrConfig.surya_dpi``.
+#: So one marker per accented letter is exactly what a byte-level failure looks
+#: like, and ``vocab['à'] = 2160`` does not mean the letter is token 2160 — in a
+#: byte-mapped vocabulary that key is the glyph standing for byte 0xA0, which
+#: alone also decodes to one marker.
+#:
+#: Either way the loss is silent: ``più`` becomes ``pi?`` and indexes, searches
+#: and reads wrong ever after. For Italian judicial documents that is real, so it
+#: gets counted and located rather than passed through unremarked — a known
+#: unknown beats a quiet corruption. To find out WHOSE fault a given document's
+#: markers are, turn on ``OcrConfig.surya_token_trace``, which reports the model's
+#: own bytes at each break (see docs/ocr.md).
+#:
+#: Too few pixels can also cause it — a diacritic is ~3 pixels tall at 200 DPI
+#: (see ``OcrConfig.surya_dpi``) — but that has been RULED OUT for the Italian
+#: case: a crisp synthetic page at 250 DPI lost every accent under surya and none
+#: under chandra.
 UNREADABLE = "\ufffd"
 
 
@@ -136,7 +144,12 @@ class OcrPage:
             "tables": self.tables,
         }
         if self.meta:
-            out["meta"] = self.meta
+            # The byte-level trace is reported ONCE, document-wide, by
+            # unreadable_report(). Repeating it on every page would put the same
+            # findings in the response twice.
+            meta = {k: v for k, v in self.meta.items() if k != "unreadable_tokens"}
+            if meta:
+                out["meta"] = meta
         n = count_unreadable(self.text)
         if n:
             out["unreadable"] = n
@@ -356,4 +369,19 @@ def unreadable_report(pages: "List[OcrPage]") -> dict:
     report = {"count": total, "pages": per_page, "spots": spots}
     if total > len(spots):
         report["truncated"] = total - len(spots)
+    # The byte-level diagnosis, when the engine produced one (surya with
+    # ``ocr.surya_token_trace`` on). This is the part that says WHOSE fault the
+    # loss is: a marker in the text is equally consistent with the model emitting
+    # a broken byte sequence and with our serving stack mis-assembling a correct
+    # one, because surya-ocr-2's vocabulary is byte-level and either byte of an
+    # accented letter decodes alone to exactly one marker. The tokens tell them
+    # apart; the text never can.
+    tokens = []
+    for page in pages or []:
+        for finding in (getattr(page, "meta", {}) or {}).get("unreadable_tokens", []):
+            entry = dict(finding)
+            entry["page"] = page.index
+            tokens.append(entry)
+    if tokens:
+        report["tokens"] = tokens
     return report

codai/ocr/manager.py

@@ -539,6 +539,9 @@ class OcrManager:
             cfg.doctr_det_arch, cfg.doctr_reco_arch,
             cfg.surya_enabled, cfg.surya_accept_license, cfg.surya_instances,
             cfg.surya_langs, cfg.surya_venv, cfg.surya_auto_build,
+            # The token trace patches surya's backend at load, so toggling it has
+            # to rebuild the pool rather than quietly taking effect never.
+            getattr(cfg, "surya_token_trace", False),
             getattr(cfg, "olmocr_enabled", False), getattr(cfg, "olmocr_instances", 1),
             getattr(cfg, "olmocr_serve", "model"), getattr(cfg, "olmocr_model", ""),
             getattr(cfg, "olmocr_model_id", ""), getattr(cfg, "olmocr_server_url", ""),

codai/ocr/subprocess_engine.py

@@ -204,4 +204,10 @@ class SubprocessOcrEngine(OcrEngine):
         page.regions = [OcrRegion(label=r.get("label", ""), bbox=list(r.get("bbox", [0, 0, 0, 0])),
                                   conf=float(r.get("conf", 0.0))) for r in d.get("regions", [])]
         page.tables = list(d.get("tables", []))
+        # The worker's byte-level diagnosis of an unreadable page, when it made one.
+        # Carried on meta rather than a field of its own so engines that cannot
+        # produce it change nothing about the response shape.
+        trace = d.get("unreadable_tokens")
+        if trace:
+            page.meta["unreadable_tokens"] = trace
         return page

codai/ocr/surya.py

@@ -66,7 +66,11 @@ class SuryaEngine(SubprocessOcrEngine):
         return bool(getattr(self.cfg, "surya_auto_build", False))
 
     def _worker_opts(self) -> dict:
-        return {"langs": self.cfg.surya_langs or "it"}
+        return {"langs": self.cfg.surya_langs or "it",
+                # Sent at LOAD, like everything else here: the hook it switches on
+                # patches surya's backend, which has to happen before the first page
+                # rather than per request.
+                "token_trace": bool(getattr(self.cfg, "surya_token_trace", False))}
 
     #: Modes where the recognition VLM lives in a model server we only talk to.
     SERVED_MODES = ("model", "vllm", "llamacpp", "server")

codai/ocr/workers/ocr_worker.py

@@ -233,6 +233,154 @@ def _construct_tolerant(cls, kwargs):
 # Surya
 # ---------------------------------------------------------------------------
 
+#: What a model writes where it could not read a character.
+UNREADABLE = "\ufffd"
+
+#: At most this many anomalies are reported per page. A page that has gone
+#: completely wrong would otherwise return its entire token stream.
+TRACE_MAX = 40
+
+#: Tokens of surrounding stream kept with each anomaly, each side.
+TRACE_CONTEXT = 2
+
+
+def _utf8_bad_offsets(buf) -> list:
+    """Byte offsets in ``buf`` where the stream is not valid UTF-8.
+
+    A real scan rather than a per-token guess, because per-token is meaningless
+    here: surya-ocr-2's vocabulary is byte-level, so ONE character arrives as two
+    or three tokens and every one of them is "invalid" in isolation. The first
+    version of this checked each token against its immediate successor and duly
+    flagged every accent and every em dash — the exact false positive that would
+    make the diagnostic worthless.
+
+    So the tokens are reassembled into the byte stream the server decoded, and the
+    stream is walked as UTF-8: a lead byte declares its length, the continuations
+    must be 0x80-0xBF, and anything else is a genuine break. Both the bytes that
+    can never begin a character (0x80-0xC1, 0xF5+) and the sequences that run off
+    the end are breaks.
+    """
+    bad = []
+    i, n = 0, len(buf)
+    while i < n:
+        b = buf[i]
+        if b < 0x80:                       # plain ASCII
+            i += 1
+            continue
+        if 0xC2 <= b <= 0xDF:
+            need = 1
+        elif 0xE0 <= b <= 0xEF:
+            need = 2
+        elif 0xF0 <= b <= 0xF4:
+            need = 3
+        else:
+            # 0x80-0xC1 is a continuation or an overlong lead; 0xF5+ is out of
+            # range. Neither can start a character.
+            bad.append(i)
+            i += 1
+            continue
+        if i + need >= n:                  # runs off the end of the stream
+            bad.append(i)
+            i += 1
+            continue
+        if not all(0x80 <= buf[i + k] <= 0xBF for k in range(1, need + 1)):
+            bad.append(i)
+            i += 1
+            continue
+        i += need + 1
+    return bad
+
+
+def _tok_entry(entry) -> dict:
+    """One OpenAI-style logprob entry reduced to what diagnoses a byte problem."""
+    if not isinstance(entry, dict):
+        entry = {"token": getattr(entry, "token", None),
+                 "bytes": getattr(entry, "bytes", None),
+                 "logprob": getattr(entry, "logprob", None)}
+    raw = entry.get("bytes")
+    out = {"token": entry.get("token")}
+    if isinstance(raw, (list, tuple)):
+        out["bytes"] = [int(b) for b in raw]
+    lp = entry.get("logprob")
+    if isinstance(lp, (int, float)):
+        out["logprob"] = round(float(lp), 4)
+    return out
+
+
+def _trace_findings(calls) -> list:
+    """Where in the model's own token stream the text stopped being readable.
+
+    This is the measurement that separates two faults which look identical in the
+    finished text:
+
+      * the MODEL emitted a broken byte sequence -- a lone UTF-8 lead byte with no
+        continuation, say -- in which case the text faithfully renders a wrong
+        answer and nothing on our side can fix it;
+      * the model emitted a CORRECT sequence and the serving stack mis-assembled
+        it, in which case the model was right and the bug is ours.
+
+    A single marker per accented letter is equally the signature of both, because
+    the vocabulary is byte-level and either byte of an accented letter decodes
+    alone to exactly one replacement character. Only the bytes tell them apart.
+
+    Two kinds of finding, and the difference matters:
+
+      ``invalid-utf8``  the reassembled stream does not decode. The server had
+                        nothing good to work with, so the model is at fault.
+      ``model-emitted`` the stream decodes cleanly and a token IS the replacement
+                        character -- the model chose it, or something upstream
+                        had already substituted it before we saw a byte.
+
+    What is NOT a finding: a stream that decodes. If the bytes are sound and the
+    text still shows markers, the loss happened after this point, which is itself
+    the answer -- an empty trace on an unreadable page indicts the stack.
+    """
+    findings = []
+    for call_idx, tokens in enumerate(calls or []):
+        if not tokens:
+            continue
+        entries = [_tok_entry(t) for t in tokens]
+
+        # Reassemble the stream, remembering which token each byte came from.
+        buf, owner = bytearray(), []
+        for ti, e in enumerate(entries):
+            for b in (e.get("bytes") or []):
+                buf.append(b & 0xFF)
+                owner.append(ti)
+
+        def _add(ti, why, kind):
+            if len(findings) >= TRACE_MAX:
+                return False
+            lo = max(0, ti - TRACE_CONTEXT)
+            hi = min(len(entries), ti + TRACE_CONTEXT + 1)
+            findings.append({
+                "call": call_idx, "index": ti, "kind": kind,
+                "token": entries[ti], "why": why, "context": entries[lo:hi],
+            })
+            return True
+
+        seen = set()
+        for off in _utf8_bad_offsets(buf):
+            ti = owner[off] if off < len(owner) else len(entries) - 1
+            if ti in seen:
+                continue
+            seen.add(ti)
+            if not _add(ti, "byte 0x%02X does not form valid UTF-8 here" % buf[off],
+                        "invalid-utf8"):
+                return findings
+
+        # Markers the model produced outright. EF BF BD is perfectly valid UTF-8,
+        # so the scan above cannot see these.
+        for ti, e in enumerate(entries):
+            tok = e.get("token")
+            if isinstance(tok, str) and UNREADABLE in tok and ti not in seen:
+                seen.add(ti)
+                if not _add(ti, "the token itself is the replacement character",
+                            "model-emitted"):
+                    return findings
+    return findings
+
+
 class SuryaWorker:
     def __init__(self, opts):
         self.opts = opts
@@ -243,6 +391,9 @@ class SuryaWorker:
         self._rec_proc = None
         self._run_ocr = None
         self._langs = [l.strip() for l in str(opts.get("langs", "it")).split(",") if l.strip()] or ["it"]
+        #: Per-call token logprobs, collected by the hook installed in load().
+        self._calls = []
+        self._trace = bool(opts.get("token_trace", False))
 
     def load(self):
         """Load surya, matching the API of whichever release is installed.
@@ -271,8 +422,53 @@ class SuryaWorker:
             self._rec = RecognitionPredictor()
         self._det = DetectionPredictor()
         self._mode = "predictor"
+        self._install_trace_hook()
         return
 
+    def _install_trace_hook(self):
+        """Keep the per-token logprobs surya asks for but then throws away.
+
+        surya's recognition predictor DOES pass them on
+        (``BlockOCRResult(..., raw_logprobs=out.logprobs)``) but the model never
+        declares the field, and pydantic's default is ``extra="ignore"`` -- so the
+        data is constructed and silently dropped. Their own comment calls it
+        "phase 2 use", which is presumably why nobody noticed. Verified by
+        construction: the attribute does not exist on the result.
+
+        So the value is taken one layer earlier, where it still exists:
+        ``_generate_one`` returns the ``GenerationResult`` that carries it. The
+        wrapper only reads, never alters, and swallows everything -- a diagnostic
+        that can break OCR would be far worse than no diagnostic.
+        """
+        if not self._trace:
+            return
+        try:
+            from surya.inference.backends import openai_client as _oc
+        except Exception as e:
+            print(f"[surya] token trace unavailable: {e}", file=sys.stderr)
+            return
+        if getattr(_oc, "_coderai_traced", False):
+            _oc._coderai_sink = self._calls          # re-point at this worker
+            return
+        inner = getattr(_oc, "_generate_one", None)
+        if inner is None:
+            print("[surya] token trace: no _generate_one to hook", file=sys.stderr)
+            return
+        _oc._coderai_sink = self._calls
+
+        def _wrapped(*args, **kwargs):
+            result = inner(*args, **kwargs)
+            try:
+                sink = getattr(_oc, "_coderai_sink", None)
+                if sink is not None:
+                    sink.append(getattr(result, "logprobs", None))
+            except Exception:
+                pass
+            return result
+
+        _oc._generate_one = _wrapped
+        _oc._coderai_traced = True
+
     def _load_legacy(self):
         # very old (<0.6): the functional run_ocr API. Kept reachable but no
         # longer a silent fallback — a modern install never gets here.
@@ -307,6 +503,10 @@ class SuryaWorker:
 
     def ocr(self, img):
         img = img.convert("RGB")
+        # One page per call, so the trace is cleared here rather than accumulated:
+        # a long document would otherwise report the first page's anomalies on the
+        # hundredth.
+        del self._calls[:]
         if self._mode == "predictor":
             preds = self._predict(img)
         else:
@@ -348,8 +548,19 @@ class SuryaWorker:
                     conf = b.get("confidence", conf)
                 lines.append({"text": text, "bbox": _poly_to_xyxy(poly or [0, 0, 0, 0]),
                               "conf": float(conf or 0.0)})
-        return {"index": 0, "width": img.width, "height": img.height,
-                "lines": lines, "regions": [], "tables": [], "text": "\n".join(l["text"] for l in lines)}
+        page = {"index": 0, "width": img.width, "height": img.height,
+                "lines": lines, "regions": [], "tables": [],
+                "text": "\n".join(l["text"] for l in lines)}
+        # Only when something actually went unread: a clean page stays byte-identical
+        # to what it has always returned.
+        if self._trace and UNREADABLE in page["text"]:
+            try:
+                findings = _trace_findings(self._calls)
+            except Exception as e:
+                findings = [{"error": f"trace analysis failed: {e}"}]
+            if findings:
+                page["unreadable_tokens"] = findings
+        return page
 
 
 # ---------------------------------------------------------------------------

docs/ocr.md

@@ -496,6 +496,65 @@ Note that `ocr.surya_langs` is a **legacy** setting. It reaches only the ≤0.17
 recognition API (`rec(images, langs, det_predictor)`); surya-2 has no language input,
 so on the pod path it does nothing.
 
+### `surya_token_trace` — whose fault is an unreadable character?
+
+"Surya's path" is two things, and only one of them is ours:
+
+1. the **model** emits a broken byte sequence — then the text faithfully renders a wrong
+   answer and nothing on our side can fix it;
+2. the model emits a **correct** sequence and the serving stack mis-assembles it — then
+   the model was right and the bug is ours.
+
+The finished text cannot tell them apart, for the reason above. The *bytes* can, so
+`ocr.surya_token_trace` surfaces them. It is **off by default**: it is a diagnostic, it
+adds response weight, and it reaches into surya's internals.
+
+It has to, because surya drops the data on the floor. Its recognition predictor really
+does pass the per-token logprobs on — `BlockOCRResult(..., raw_logprobs=out.logprobs)` —
+but the model never declares that field and pydantic's default is `extra="ignore"`, so
+the value is constructed and silently discarded (their own comment calls it "phase 2
+use"). Verified by construction: the attribute does not exist on the result. coderai
+therefore takes it one layer earlier, wrapping `_generate_one`, which still has it. The
+wrapper only reads, and swallows every error — a diagnostic that could break a page would
+be far worse than no diagnostic.
+
+With it on, a document that lost characters carries its own diagnosis under
+`unreadable.tokens`:
+
+```json
+{ "unreadable": {
+    "count": 29,
+    "tokens": [
+      { "page": 0, "call": 3, "index": 41, "kind": "invalid-utf8",
+        "token":   {"token": "Ã", "bytes": [195], "logprob": -0.0021},
+        "why":     "byte 0xC3 does not form valid UTF-8 here",
+        "context": [ {"token": "p", "bytes": [112]}, {"token": "i", "bytes": [105]},
+                     {"token": "Ã", "bytes": [195]}, {"token": " ", "bytes": [32]} ] }
+    ] } }
+```
+
+How to read it:
+
+| what you see | verdict |
+|---|---|
+| `kind: "invalid-utf8"`, a lone `[195]` with nothing to complete it | the **model** emitted a broken sequence — report it to Datalab |
+| `kind: "model-emitted"`, the token *is* the replacement character | the model chose the marker, or something upstream substituted it before we saw a byte |
+| **no `tokens` at all** on a page full of markers | the bytes were sound and the loss happened after them — **ours**, in the serving stack |
+
+That last row is why an empty trace is a result and not a failure. The scan reassembles
+the whole byte stream and walks it as UTF-8 rather than judging tokens one at a time: per
+token is meaningless here, since one character legitimately arrives as two or three
+tokens and each is "invalid" alone. The first version of this did judge them pairwise and
+duly flagged every accent *and* every em dash — exactly the false positive that would
+make the diagnostic worthless.
+
+Findings are capped (`TRACE_MAX`, 40 per page) with two tokens of context each side,
+because a lead byte is only wrong in the light of what followed it.
+
+A third, independent discriminator needs no code at all: serve the **same weights through
+llama.cpp** (`datalab-to/surya-ocr-2-gguf`, with `surya_serve: "llamacpp"`). A different
+detokenizer on identical weights — accents surviving there would convict vLLM outright.
+
 ### Where `conf` comes from on a served engine
 
 A VLM pipeline has no detector score to report, so surya asks the model server for
@@ -577,7 +636,26 @@ the document's own glyph quality and the serving stack (coderai's pods run vLLM
 them apart is to OCR **perfect synthetic Italian** through the same path — clean
 output indicts the document, U+FFFD on crisp vector glyphs indicts the stack.
 
-There is now a second, cheaper discriminator: **run the page through `chandra`**.
+**Correction, measured 2026-10-12.** An earlier version of this section argued that
+U+FFFD could not be a byte-level artefact, because `à` was a single token and one marker
+therefore meant the model had chosen it. **That was wrong.** surya-ocr-2's vocabulary is
+**byte-level**, and an accented letter is two tokens:
+
+```
+'à' utf8=[195, 160]  ids=[2131, 2258]  pieces=['Ã', 'ł']
+'è' utf8=[195, 168]  ids=[2131, 2104]  pieces=['Ã', '¨']
+
+decode([2131])       = '?'     <- a lone lead byte gives exactly ONE marker
+decode([2131, 2104]) = 'è'
+```
+
+So **one marker per accented letter is exactly the signature of a byte-level failure** —
+the thing it was claimed to rule out. The vocabulary lookup was misread too: `vocab['à']
+= 2160` does not mean the letter is token 2160; in a byte-mapped vocabulary that key is
+the glyph standing for byte `0xA0`, and `decode([2160])` is a single marker as well. The
+"one marker vs two" discriminator does not exist.
+
+There is a second, cheaper discriminator that does: **run the page through `chandra`**.
 It is a different model family with a different tokenizer (Qwen3-VL, against
 surya-2's own 65k `WordLevel` vocabulary), reached over the same transport, at the
 same pixel ceiling, from the same rasterised page. Accents intact under chandra

tests/test_ocr_token_trace.py

+"""The byte-level trace: whose fault is an unreadable character?
+
+A marker in the finished text says nothing about the cause, and I got this wrong once
+by arguing that it did. surya-ocr-2's vocabulary is BYTE-level: `a`-grave is two tokens
+(UTF-8 C3 A0), and decoding EITHER byte alone yields exactly one replacement character.
+So one marker per accented letter is equally the signature of
+
+  * the model emitting a broken byte sequence  -> a faithful rendering of a wrong
+    answer, nothing we can fix;
+  * the model emitting a correct sequence that our serving stack mis-assembled ->
+    the model was right and the bug is ours.
+
+Only the tokens tell them apart. These tests pin the discriminator, and the two
+properties that make it safe to ship: it is silent on a clean page, and it can never
+break OCR to produce a diagnostic.
+"""
+
+import importlib.util
+import pathlib
+import sys
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+import pytest                                                      # noqa: E402
+
+
+def _worker_module():
+    """Load the worker the way it actually runs: as a standalone file.
+
+    It is deliberately self-contained (stdlib + PIL + one OCR library, never
+    coderai) because it executes inside an isolated venv, so it is not importable
+    as ``codai.ocr.workers.ocr_worker`` in that venv at all. Loading it by path is
+    what keeps this test honest about the thing being shipped.
+    """
+    path = ROOT / "codai" / "ocr" / "workers" / "ocr_worker.py"
+    spec = importlib.util.spec_from_file_location("ocr_worker_under_test", path)
+    mod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(mod)
+    return mod
+
+
+W = _worker_module()
+
+
+def _tok(token, byts, logprob=-0.01):
+    return {"token": token, "bytes": byts, "logprob": logprob}
+
+
+# ------------------------------------------------------------- the two verdicts #
+def test_a_lone_lead_byte_is_reported_as_broken():
+    """The model-is-wrong case: C3 with no continuation. `a`-grave never arrives."""
+    calls = [[_tok("p", [112]), _tok("i", [105]), _tok("Ã", [195]), _tok(" ", [32])]]
+    found = W._trace_findings(calls)
+    assert len(found) == 1
+    assert found[0]["token"]["bytes"] == [195]
+    assert found[0]["kind"] == "invalid-utf8"
+
+
+def test_a_complete_byte_pair_is_not_reported():
+    """The stack-is-wrong case must NOT be confused with it: C3 A0 is a correct
+    `a`-grave, and a trace that flagged it would cry wolf on every accent."""
+    calls = [[_tok("Ã", [195]), _tok("ł", [160]), _tok("a", [97])]]
+    assert W._trace_findings(calls) == []
+
+
+def test_a_three_byte_character_split_across_tokens_is_not_reported():
+    """An em dash is E2 80 94 over three tokens. It survived in production while
+    every two-byte accent died, so it must read as healthy here."""
+    calls = [[_tok("â", [226]), _tok("Ģ", [128]), _tok("Ķ", [148])]]
+    assert W._trace_findings(calls) == []
+
+
+def test_a_token_that_is_itself_the_marker_is_reported():
+    """The third possibility: the SERVER already decoded to U+FFFD before we saw
+    it, so the bytes arrive as the replacement character's own encoding."""
+    calls = [[_tok("�", [239, 191, 189])]]
+    found = W._trace_findings(calls)
+    assert len(found) == 1
+    assert "replacement character" in found[0]["why"]
+
+
+def test_a_trailing_lead_byte_at_the_very_end_is_reported():
+    """Nothing follows, so nothing can complete it — the commonest shape of a
+    truncated generation."""
+    calls = [[_tok("a", [97]), _tok("Ã", [195])]]
+    assert len(W._trace_findings(calls)) == 1
+
+
+def test_a_lead_byte_followed_by_ascii_is_reported():
+    """C3 then 'a': the model started an accent and finished with something that
+    cannot continue one."""
+    found = W._trace_findings([[_tok("Ã", [195]), _tok("a", [97])]])
+    assert len(found) == 1
+    assert found[0]["kind"] == "invalid-utf8"
+
+
+def test_a_four_byte_character_is_healthy():
+    """F0 9F 93 84 — outside Italian, but the scan must not invent a fault."""
+    assert W._trace_findings([[_tok("a", [240]), _tok("b", [159]),
+                               _tok("c", [147]), _tok("d", [132])]]) == []
+
+
+def test_a_sound_stream_with_no_markers_yields_nothing():
+    """The decisive negative: if the bytes are good and the text still lost
+    characters, the loss happened AFTER this point — and an empty trace on an
+    unreadable page is what convicts the serving stack."""
+    assert W._trace_findings([[_tok("p", [112]), _tok("Ã", [195]),
+                               _tok("ł", [160])]]) == []
+
+
+def test_one_finding_per_token_not_per_byte():
+    """A three-byte token that is entirely wrong must not report three times."""
+    found = W._trace_findings([[_tok("?", [195, 195, 195])]])
+    assert len(found) == 1
+
+
+def test_a_continuation_byte_with_no_lead_is_reported():
+    calls = [[_tok("ł", [160]), _tok("a", [97])]]
+    found = W._trace_findings(calls)
+    assert len(found) == 1
+    assert found[0]["token"]["bytes"] == [160]
+
+
+# ------------------------------------------------------------------- the shape #
+def test_each_finding_carries_its_neighbours():
+    """A lead byte is only wrong in the light of what followed it, so a finding
+    without context cannot be judged."""
+    calls = [[_tok("a", [97]), _tok("b", [98]), _tok("Ã", [195]),
+              _tok("c", [99]), _tok("d", [100])]]
+    found = W._trace_findings(calls)
+    ctx = found[0]["context"]
+    assert [c["token"] for c in ctx] == ["a", "b", "Ã", "c", "d"]
+
+
+def test_context_is_clipped_at_the_edges():
+    calls = [[_tok("Ã", [195])]]
+    assert len(W._trace_findings(calls)[0]["context"]) == 1
+
+
+def test_findings_say_which_call_and_which_token():
+    """A page is several blocks, each its own request; "token 4" is meaningless
+    without which block it was token 4 of."""
+    calls = [[_tok("a", [97])], [_tok("a", [97]), _tok("Ã", [195])]]
+    found = W._trace_findings(calls)
+    assert found[0]["call"] == 1
+    assert found[0]["index"] == 1
+
+
+def test_the_logprob_comes_through_so_confidence_is_visible():
+    """A broken byte the model was SURE about is a different problem from one it
+    guessed at."""
+    calls = [[_tok("Ã", [195], logprob=-0.0001)]]
+    assert W._trace_findings(calls)[0]["token"]["logprob"] == -0.0001
+
+
+def test_findings_are_capped():
+    """A page of pure noise must not return its entire token stream."""
+    calls = [[_tok("Ã", [195]) for _ in range(W.TRACE_MAX + 25)]]
+    assert len(W._trace_findings(calls)) == W.TRACE_MAX
+
+
+def test_pydantic_style_objects_are_accepted_too():
+    """The OpenAI client hands back model objects, not dicts, depending on version."""
+    class _Entry:
+        token, bytes, logprob = "Ã", [195], -0.2
+    assert len(W._trace_findings([[_Entry()]])) == 1
+
+
+def test_missing_bytes_are_not_an_anomaly():
+    """A server that reports no bytes is not evidence of a broken byte."""
+    calls = [[{"token": "a", "logprob": -0.1}]]
+    assert W._trace_findings(calls) == []
+
+
+def test_no_calls_is_no_findings():
+    assert W._trace_findings([]) == []
+    assert W._trace_findings(None) == []
+    assert W._trace_findings([None, []]) == []
+
+
+# ------------------------------------------------- it must never break the OCR #
+def test_the_trace_is_off_unless_asked_for():
+    w = W.SuryaWorker({})
+    assert w._trace is False
+    assert W.SuryaWorker({"token_trace": True})._trace is True
+
+
+def test_installing_the_hook_without_surya_does_not_raise():
+    """The worker runs in an isolated venv; a surya too old to have the backend
+    module must degrade to "no diagnostic", never to "no OCR"."""
+    w = W.SuryaWorker({"token_trace": True})
+    w._install_trace_hook()          # surya is not installed in the test venv
+
+
+def test_a_clean_page_reports_nothing(monkeypatch):
+    """The whole response shape has to stay byte-identical when nothing is wrong."""
+    w = W.SuryaWorker({"token_trace": True})
+    w._mode = "predictor"
+    w._calls.append([_tok("Ã", [195])])          # an anomaly is present...
+    monkeypatch.setattr(w, "_predict", lambda img: [
+        {"blocks": [{"html": "<p>tutto bene</p>", "polygon": [[0, 0], [1, 0], [1, 1], [0, 1]],
+                     "confidence": 0.9, "reading_order": 0}]}])
+    from PIL import Image
+    page = w.ocr(Image.new("RGB", (10, 10)))
+    # ...but the text has no marker, so there is nothing to diagnose.
+    assert "unreadable_tokens" not in page
+
+
+def test_an_unreadable_page_carries_the_diagnosis(monkeypatch):
+    w = W.SuryaWorker({"token_trace": True})
+    w._mode = "predictor"
+    from PIL import Image
+
+    def _predict(img):
+        # The hook would have filled this during the real call.
+        w._calls.append([_tok("p", [112]), _tok("i", [105]), _tok("Ã", [195])])
+        return [{"blocks": [{"html": "<p>pi�</p>",
+                             "polygon": [[0, 0], [1, 0], [1, 1], [0, 1]],
+                             "confidence": 0.4, "reading_order": 0}]}]
+    monkeypatch.setattr(w, "_predict", _predict)
+    page = w.ocr(Image.new("RGB", (10, 10)))
+    assert page["unreadable_tokens"][0]["token"]["bytes"] == [195]
+
+
+def test_the_trace_is_cleared_between_pages(monkeypatch):
+    """Otherwise page 100 of a document reports page 1's anomalies."""
+    w = W.SuryaWorker({"token_trace": True})
+    w._mode = "predictor"
+    from PIL import Image
+    monkeypatch.setattr(w, "_predict", lambda img: [
+        {"blocks": [{"html": "<p>ok</p>", "polygon": [[0, 0], [1, 0], [1, 1], [0, 1]],
+                     "confidence": 0.9, "reading_order": 0}]}])
+    w._calls.append([_tok("Ã", [195])])
+    w.ocr(Image.new("RGB", (10, 10)))
+    assert w._calls == []
+
+
+def test_analysis_failure_is_reported_not_raised(monkeypatch):
+    """A diagnostic that can break a page would be worse than no diagnostic."""
+    w = W.SuryaWorker({"token_trace": True})
+    w._mode = "predictor"
+    from PIL import Image
+
+    def _boom(calls):
+        raise RuntimeError("bad trace")
+    monkeypatch.setattr(W, "_trace_findings", _boom)
+
+    def _predict(img):
+        return [{"blocks": [{"html": "<p>pi�</p>",
+                             "polygon": [[0, 0], [1, 0], [1, 1], [0, 1]],
+                             "confidence": 0.4, "reading_order": 0}]}]
+    monkeypatch.setattr(w, "_predict", _predict)
+    page = w.ocr(Image.new("RGB", (10, 10)))
+    assert "bad trace" in page["unreadable_tokens"][0]["error"]
+
+
+# ------------------------------------------------------ and out through the API #
+def test_the_report_carries_the_tokens():
+    from codai.ocr.base import OcrPage, OcrLine, unreadable_report
+    page = OcrPage(index=0, text="pi�")
+    page.lines = [OcrLine(text="pi�", bbox=[0, 0, 1, 1], conf=0.4)]
+    page.meta["unreadable_tokens"] = [{"index": 2, "token": {"bytes": [195]},
+                                       "why": "x", "context": []}]
+    rep = unreadable_report([page])
+    assert rep["count"] == 1
+    assert rep["tokens"][0]["token"]["bytes"] == [195]
+    assert rep["tokens"][0]["page"] == 0
+
+
+def test_a_clean_document_has_no_report_at_all():
+    from codai.ocr.base import OcrPage, unreadable_report
+    page = OcrPage(index=0, text="tutto bene")
+    page.meta["unreadable_tokens"] = [{"anything": 1}]
+    assert unreadable_report([page]) == {}
+
+
+def test_the_trace_is_not_repeated_on_every_page():
+    """It is reported once, document-wide. Repeating it per page would put the
+    same findings in the response twice."""
+    from codai.ocr.base import OcrPage
+    page = OcrPage(index=0, text="pi�")
+    page.meta["unreadable_tokens"] = [{"index": 2}]
+    page.meta["primary_language"] = "it"
+    d = page.to_dict()
+    assert "unreadable_tokens" not in d.get("meta", {})
+    assert d["meta"]["primary_language"] == "it"      # other metadata survives
+
+
+def test_a_page_whose_only_meta_was_the_trace_reports_no_meta():
+    from codai.ocr.base import OcrPage
+    page = OcrPage(index=0, text="x")
+    page.meta["unreadable_tokens"] = [{"index": 1}]
+    assert "meta" not in page.to_dict()
+
+
+def test_toggling_the_flag_rebuilds_the_pool():
+    """The hook patches surya's backend at LOAD, so a change that did not
+    invalidate the pool would appear to save and do nothing."""
+    from codai.config import OcrConfig
+    from codai.ocr.manager import OcrManager
+    off, on = OcrConfig(), OcrConfig()
+    on.surya_token_trace = True
+    assert OcrManager._pool_params(off) != OcrManager._pool_params(on)
+
+
+def test_the_engine_tells_the_worker_at_load():
+    from codai.config import OcrConfig
+    from codai.ocr.surya import SuryaEngine
+    cfg = OcrConfig()
+    cfg.surya_token_trace = True
+    assert SuryaEngine(cfg)._worker_opts()["token_trace"] is True
+    assert SuryaEngine(OcrConfig())._worker_opts()["token_trace"] is False