ocr: surface the model's own bytes where a page came back unreadable (v0.3.25)

`ocr.surya_token_trace` (off by default) reports the per-token bytes at every
place a surya page lost a character. It exists to answer one question that the
finished text cannot: whose fault is it?

  invalid-utf8   the reassembled stream does not decode -- a lone UTF-8 lead
                 byte with nothing to complete it. The MODEL emitted a broken
                 sequence; the text faithfully renders a wrong answer.
  model-emitted  the stream decodes and a token IS the replacement character.
                 The model chose it.
  no findings    the bytes were sound and the markers appeared anyway, so the
                 loss happened AFTER them -- OURS, in the serving stack.

That last case is why an empty trace is a result, not a failure.

CORRECTION, and the reason this was needed. v0.3.21 claimed U+FFFD could not be
a byte-level artefact, on the grounds that `a`-grave was a single token so one
marker had to mean the model chose it. That was wrong. Measured:

  'a-grave' utf8=[195, 160]  ids=[2131, 2258]  pieces=['A-tilde', 'l-bar']
  decode([2131])       = U+FFFD    <- a lone lead byte, exactly ONE marker
  decode([2131, 2104]) = 'e-grave'

surya-ocr-2's vocabulary is BYTE-level, so one marker per accented letter is
precisely the signature of a byte-level failure -- the thing it was claimed to
rule out. The vocab lookup was misread too: `vocab['a-grave'] = 2160` does not
mean the letter is token 2160; in a byte-mapped vocabulary that key is the glyph
standing for byte 0xA0, which alone also decodes to one marker. The
"one marker vs two" discriminator does not exist. Digesta's charset instinct was
closer to the mark than the rebuttal. base.UNREADABLE and docs/ocr.md now carry
the correction rather than the refuted argument.

What IS established, measured on the Aruba install at the same DPI through the
same transport on perfect synthetic Italian (crisp vector glyphs, 34pt, 250 DPI):

    surya    29 U+FFFD markers, zero accents surviving
    chandra   0 markers, all 12 accents intact

So the document is exonerated. Also notable: the em dash survived surya intact
while every two-byte accent died, and surya emitted U+1EEA (Vietnamese) in place
of one -- no decoding bug can fabricate a character that was never in the input.

Implementation notes:

- surya DROPS this data. Its recognition predictor does pass it on
  (`BlockOCRResult(..., raw_logprobs=out.logprobs)`) but the model never declares
  the field and pydantic defaults to extra="ignore", so it is constructed and
  silently discarded -- their own comment calls it "phase 2 use". Verified by
  construction: the attribute does not exist on the result. So the value is taken
  one layer earlier, wrapping `_generate_one`, which still has it. The wrapper
  only reads and swallows every error: a diagnostic that could break a page would
  be worse than no diagnostic.
- The scan reassembles the WHOLE byte stream and walks it as UTF-8 rather than
  judging tokens pairwise. Per token is meaningless here -- one character
  legitimately arrives as two or three tokens, each "invalid" alone -- and the
  first version of this did judge them pairwise and flagged every accent and
  every em dash, the exact false positive that would make it worthless.
- Capped at 40 findings per page, two tokens of context each side, because a lead
  byte is only wrong in the light of what followed it.
- Clean pages are byte-identical to before: the key appears only when the text
  actually contains a marker, and it is reported ONCE document-wide under
  `unreadable.tokens` rather than repeated in every page's meta.
- The flag is in _pool_params, because the hook patches surya's backend at LOAD
  -- without that, toggling it would appear to save and take effect never.

A third discriminator needing no code: serve the same weights through llama.cpp
(`datalab-to/surya-ocr-2-gguf`, `surya_serve: "llamacpp"`). A different
detokenizer on identical weights would convict vLLM outright.

tests/test_ocr_token_trace.py: 30 tests. 818 passed across the OCR/config/admin/
runpod selection with the same 10 pre-existing failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.24"
+__version__ = "0.3.25"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -4139,6 +4139,7 @@ def build_settings_dict(c, gpu_cards):
             "surya_accept_license": c.ocr.surya_accept_license,
             "surya_instances": c.ocr.surya_instances,
             "surya_langs": c.ocr.surya_langs,
+            "surya_token_trace": c.ocr.surya_token_trace,
             "surya_venv": c.ocr.surya_venv,
             "surya_auto_build": c.ocr.surya_auto_build,
             "surya_serve": c.ocr.surya_serve,
@@ -4956,6 +4957,7 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
         if "surya_enabled" in d: o.surya_enabled = bool(d["surya_enabled"])
         if "surya_accept_license" in d: o.surya_accept_license = bool(d["surya_accept_license"])
         if "surya_instances" in d: o.surya_instances = _i("surya_instances", o.surya_instances, 1)
+        if "surya_token_trace" in d: o.surya_token_trace = bool(d["surya_token_trace"])
         if "surya_langs" in d: o.surya_langs = (d.get("surya_langs") or "it").strip()
         if "surya_venv" in d: o.surya_venv = (d.get("surya_venv") or "").strip()
         if "surya_auto_build" in d: o.surya_auto_build = bool(d["surya_auto_build"])

codai/admin/templates/settings.html

@@ -1206,6 +1206,7 @@
       <div class="form-group" style="flex:1;min-width:110px"><label class="form-label">Instances</label><input type="number" id="s-ocr-surya-instances" class="form-input" min="1" placeholder="1"></div>
       <label style="display:flex;align-items:center;gap:.4rem;cursor:pointer" title="Local mode only. On, Surya runs on this machine's GPU or refuses to load — it will not quietly fall back to the CPU, where a page takes minutes. Off, the CPU is used deliberately."><input type="checkbox" id="s-ocr-surya-use-gpu" checked><span>Require GPU (Local)</span></label>
       <div class="form-group" style="flex:1;min-width:120px"><label class="form-label">Langs</label><input type="text" id="s-ocr-surya-langs" class="form-input" placeholder="it"></div>
+      <label style="display:flex;align-items:center;gap:.4rem;cursor:pointer" title="Return the model's own per-token bytes wherever a page came back unreadable. A diagnostic — leave off unless you are chasing lost characters."><input type="checkbox" id="s-ocr-surya-token-trace"><span>Trace unreadable tokens</span></label>
     </div>
     <p class="form-hint" style="margin:0 0 .3rem">Runs in an <b>isolated venv</b> (Surya caps pillow&lt;11, incompatible with the main venv).</p>
     <div class="form-row" style="display:flex;gap:1rem;flex-wrap:wrap;align-items:center">
@@ -2202,6 +2203,7 @@ async function loadSettings(){
     document.getElementById('s-ocr-surya-instances').value = ocr.surya_instances ?? 1;
     document.getElementById('s-ocr-surya-use-gpu').checked = ocr.surya_use_gpu !== false;
     document.getElementById('s-ocr-surya-langs').value = ocr.surya_langs ?? 'it';
+    document.getElementById('s-ocr-surya-token-trace').checked = !!ocr.surya_token_trace;
     document.getElementById('s-ocr-surya-venv').value = ocr.surya_venv ?? '';
     document.getElementById('s-ocr-surya-auto-build').checked = !!ocr.surya_auto_build;
     document.getElementById('s-ocr-surya-serve').value = ocr.surya_serve || 'model';
@@ -2484,6 +2486,7 @@ async function saveSettings(silent){
       surya_accept_license: document.getElementById('s-ocr-surya-license').checked,
       surya_instances: parseInt(document.getElementById('s-ocr-surya-instances').value) || 1,
       surya_langs: document.getElementById('s-ocr-surya-langs').value.trim() || 'it',
+      surya_token_trace: document.getElementById('s-ocr-surya-token-trace').checked,
       surya_venv: document.getElementById('s-ocr-surya-venv').value.trim(),
       surya_auto_build: document.getElementById('s-ocr-surya-auto-build').checked,
       surya_serve: document.getElementById('s-ocr-surya-serve').value,

codai/config.py

@@ -988,6 +988,16 @@ class OcrConfig:
     surya_accept_license: bool = False   # MUST be explicitly set (GPL — compatible with coderai GPLv3)
     surya_instances: int = 1
     surya_langs: str = "it"
+    # Return the model's own per-token bytes for the places a page came back
+    # unreadable (see base.UNREADABLE). Off by default: it is a diagnostic, it
+    # costs response weight, and it reaches into surya's internals to get data
+    # surya itself constructs and then drops. Turn it on when a document is
+    # losing characters and you need to know WHOSE fault it is -- the model
+    # emitting a broken byte sequence, or our serving stack mis-assembling a
+    # correct one. The two are indistinguishable in the finished text, because
+    # surya-ocr-2's vocabulary is byte-level and decoding either byte of an
+    # accented letter alone yields exactly one replacement character.
+    surya_token_trace: bool = False
     surya_venv: str = ""               # isolated venv dir; blank = ~/.coderai/surya_venv
     surya_auto_build: bool = False     # create the venv + pip install requirements-surya.txt on first use
     # Surya serving mode: "local" = classic det+recognition on torch in the isolated venv
@@ -1702,6 +1712,7 @@ class ConfigManager:
                 "surya_accept_license": self.config.ocr.surya_accept_license,
                 "surya_instances": self.config.ocr.surya_instances,
                 "surya_langs": self.config.ocr.surya_langs,
+                "surya_token_trace": self.config.ocr.surya_token_trace,
                 "surya_venv": self.config.ocr.surya_venv,
                 "surya_auto_build": self.config.ocr.surya_auto_build,
                 "surya_use_gpu": self.config.ocr.surya_use_gpu,

codai/ocr/base.py

@@ -44,24 +44,32 @@ class OcrError(RuntimeError):
 
 #: What a VLM OCR engine writes where it cannot identify a character.
 #:
-#: This is NOT a decoding or transport artefact, which is the natural assumption
-#: when accented letters come back as U+FFFD. In surya-ocr-2 it is a real entry in
-#: the vocabulary — id 65421 of 65425, the only id that decodes to it, and
-#: ``<unk>`` is a different token (65424). The tokenizer is ``WordLevel`` and
-#: character-level for Latin text, so ``à`` is a single token and an accent
-#: *cannot* be split across two: no partial-UTF-8 artefact is possible, and
-#: per-token decoding is byte-identical to decoding the whole string. Surya also
-#: reads its text only from ``message.content``, never from the per-token
-#: logprobs, and coderai's OCR gateway forwards the upstream body as raw bytes.
+#: It may be the model's own answer OR a byte-level accident, and the text alone
+#: cannot tell you which. An earlier version of this comment claimed otherwise —
+#: that ``à`` was a single token, so one marker had to mean the model chose it.
+#: That was wrong, and measurement settled it: surya-ocr-2's vocabulary is
+#: BYTE-level, and an accented letter is two tokens.
 #:
-#: So a U+FFFD in the output is the MODEL saying "there is a character here I
-#: cannot read". It is honest, and it is also silent: ``più`` becomes ``pi?`` and
-#: indexes, searches and reads wrong ever after. For Italian judicial documents
-#: that is a real loss, so it gets counted and located rather than passed through
-#: unremarked — a known unknown beats a quiet corruption.
+#:     'à' utf8=[195, 160]  ids=[2131, 2258]  pieces=['Ã', 'ł']
+#:     decode([2131])       = U+FFFD          <- a lone lead byte, ONE marker
+#:     decode([2131, 2104]) = 'è'
 #:
-#: The usual cause is too few pixels: a diacritic sits above the x-height and is
-#: ~3 pixels tall at 200 DPI. See ``OcrConfig.surya_dpi``.
+#: So one marker per accented letter is exactly what a byte-level failure looks
+#: like, and ``vocab['à'] = 2160`` does not mean the letter is token 2160 — in a
+#: byte-mapped vocabulary that key is the glyph standing for byte 0xA0, which
+#: alone also decodes to one marker.
+#:
+#: Either way the loss is silent: ``più`` becomes ``pi?`` and indexes, searches
+#: and reads wrong ever after. For Italian judicial documents that is real, so it
+#: gets counted and located rather than passed through unremarked — a known
+#: unknown beats a quiet corruption. To find out WHOSE fault a given document's
+#: markers are, turn on ``OcrConfig.surya_token_trace``, which reports the model's
+#: own bytes at each break (see docs/ocr.md).
+#:
+#: Too few pixels can also cause it — a diacritic is ~3 pixels tall at 200 DPI
+#: (see ``OcrConfig.surya_dpi``) — but that has been RULED OUT for the Italian
+#: case: a crisp synthetic page at 250 DPI lost every accent under surya and none
+#: under chandra.
 UNREADABLE = "\ufffd"
 
 
@@ -136,7 +144,12 @@ class OcrPage:
             "tables": self.tables,
         }
         if self.meta:
-            out["meta"] = self.meta
+            # The byte-level trace is reported ONCE, document-wide, by
+            # unreadable_report(). Repeating it on every page would put the same
+            # findings in the response twice.
+            meta = {k: v for k, v in self.meta.items() if k != "unreadable_tokens"}
+            if meta:
+                out["meta"] = meta
         n = count_unreadable(self.text)
         if n:
             out["unreadable"] = n
@@ -356,4 +369,19 @@ def unreadable_report(pages: "List[OcrPage]") -> dict:
     report = {"count": total, "pages": per_page, "spots": spots}
     if total > len(spots):
         report["truncated"] = total - len(spots)
+    # The byte-level diagnosis, when the engine produced one (surya with
+    # ``ocr.surya_token_trace`` on). This is the part that says WHOSE fault the
+    # loss is: a marker in the text is equally consistent with the model emitting
+    # a broken byte sequence and with our serving stack mis-assembling a correct
+    # one, because surya-ocr-2's vocabulary is byte-level and either byte of an
+    # accented letter decodes alone to exactly one marker. The tokens tell them
+    # apart; the text never can.
+    tokens = []
+    for page in pages or []:
+        for finding in (getattr(page, "meta", {}) or {}).get("unreadable_tokens", []):
+            entry = dict(finding)
+            entry["page"] = page.index
+            tokens.append(entry)
+    if tokens:
+        report["tokens"] = tokens
     return report

codai/ocr/manager.py

@@ -539,6 +539,9 @@ class OcrManager:
             cfg.doctr_det_arch, cfg.doctr_reco_arch,
             cfg.surya_enabled, cfg.surya_accept_license, cfg.surya_instances,
             cfg.surya_langs, cfg.surya_venv, cfg.surya_auto_build,
+            # The token trace patches surya's backend at load, so toggling it has
+            # to rebuild the pool rather than quietly taking effect never.
+            getattr(cfg, "surya_token_trace", False),
             getattr(cfg, "olmocr_enabled", False), getattr(cfg, "olmocr_instances", 1),
             getattr(cfg, "olmocr_serve", "model"), getattr(cfg, "olmocr_model", ""),
             getattr(cfg, "olmocr_model_id", ""), getattr(cfg, "olmocr_server_url", ""),

codai/ocr/subprocess_engine.py

@@ -204,4 +204,10 @@ class SubprocessOcrEngine(OcrEngine):
         page.regions = [OcrRegion(label=r.get("label", ""), bbox=list(r.get("bbox", [0, 0, 0, 0])),
                                   conf=float(r.get("conf", 0.0))) for r in d.get("regions", [])]
         page.tables = list(d.get("tables", []))
+        # The worker's byte-level diagnosis of an unreadable page, when it made one.
+        # Carried on meta rather than a field of its own so engines that cannot
+        # produce it change nothing about the response shape.
+        trace = d.get("unreadable_tokens")
+        if trace:
+            page.meta["unreadable_tokens"] = trace
         return page

codai/ocr/surya.py

@@ -66,7 +66,11 @@ class SuryaEngine(SubprocessOcrEngine):
         return bool(getattr(self.cfg, "surya_auto_build", False))
 
     def _worker_opts(self) -> dict:
-        return {"langs": self.cfg.surya_langs or "it"}
+        return {"langs": self.cfg.surya_langs or "it",
+                # Sent at LOAD, like everything else here: the hook it switches on
+                # patches surya's backend, which has to happen before the first page
+                # rather than per request.
+                "token_trace": bool(getattr(self.cfg, "surya_token_trace", False))}
 
     #: Modes where the recognition VLM lives in a model server we only talk to.
     SERVED_MODES = ("model", "vllm", "llamacpp", "server")

docs/ocr.md

@@ -496,6 +496,65 @@ Note that `ocr.surya_langs` is a **legacy** setting. It reaches only the ≤0.17
 recognition API (`rec(images, langs, det_predictor)`); surya-2 has no language input,
 so on the pod path it does nothing.
 
+### `surya_token_trace` — whose fault is an unreadable character?
+
+"Surya's path" is two things, and only one of them is ours:
+
+1. the **model** emits a broken byte sequence — then the text faithfully renders a wrong
+   answer and nothing on our side can fix it;
+2. the model emits a **correct** sequence and the serving stack mis-assembles it — then
+   the model was right and the bug is ours.
+
+The finished text cannot tell them apart, for the reason above. The *bytes* can, so
+`ocr.surya_token_trace` surfaces them. It is **off by default**: it is a diagnostic, it
+adds response weight, and it reaches into surya's internals.
+
+It has to, because surya drops the data on the floor. Its recognition predictor really
+does pass the per-token logprobs on — `BlockOCRResult(..., raw_logprobs=out.logprobs)` —
+but the model never declares that field and pydantic's default is `extra="ignore"`, so
+the value is constructed and silently discarded (their own comment calls it "phase 2
+use"). Verified by construction: the attribute does not exist on the result. coderai
+therefore takes it one layer earlier, wrapping `_generate_one`, which still has it. The
+wrapper only reads, and swallows every error — a diagnostic that could break a page would
+be far worse than no diagnostic.
+
+With it on, a document that lost characters carries its own diagnosis under
+`unreadable.tokens`:
+
+```json
+{ "unreadable": {
+    "count": 29,
+    "tokens": [
+      { "page": 0, "call": 3, "index": 41, "kind": "invalid-utf8",
+        "token":   {"token": "Ã", "bytes": [195], "logprob": -0.0021},
+        "why":     "byte 0xC3 does not form valid UTF-8 here",
+        "context": [ {"token": "p", "bytes": [112]}, {"token": "i", "bytes": [105]},
+                     {"token": "Ã", "bytes": [195]}, {"token": " ", "bytes": [32]} ] }
+    ] } }
+```
+
+How to read it:
+
+| what you see | verdict |
+|---|---|
+| `kind: "invalid-utf8"`, a lone `[195]` with nothing to complete it | the **model** emitted a broken sequence — report it to Datalab |
+| `kind: "model-emitted"`, the token *is* the replacement character | the model chose the marker, or something upstream substituted it before we saw a byte |
+| **no `tokens` at all** on a page full of markers | the bytes were sound and the loss happened after them — **ours**, in the serving stack |
+
+That last row is why an empty trace is a result and not a failure. The scan reassembles
+the whole byte stream and walks it as UTF-8 rather than judging tokens one at a time: per
+token is meaningless here, since one character legitimately arrives as two or three
+tokens and each is "invalid" alone. The first version of this did judge them pairwise and
+duly flagged every accent *and* every em dash — exactly the false positive that would
+make the diagnostic worthless.
+
+Findings are capped (`TRACE_MAX`, 40 per page) with two tokens of context each side,
+because a lead byte is only wrong in the light of what followed it.
+
+A third, independent discriminator needs no code at all: serve the **same weights through
+llama.cpp** (`datalab-to/surya-ocr-2-gguf`, with `surya_serve: "llamacpp"`). A different
+detokenizer on identical weights — accents surviving there would convict vLLM outright.
+
 ### Where `conf` comes from on a served engine
 
 A VLM pipeline has no detector score to report, so surya asks the model server for
@@ -577,7 +636,26 @@ the document's own glyph quality and the serving stack (coderai's pods run vLLM
 them apart is to OCR **perfect synthetic Italian** through the same path — clean
 output indicts the document, U+FFFD on crisp vector glyphs indicts the stack.
 
-There is now a second, cheaper discriminator: **run the page through `chandra`**.
+**Correction, measured 2026-10-12.** An earlier version of this section argued that
+U+FFFD could not be a byte-level artefact, because `à` was a single token and one marker
+therefore meant the model had chosen it. **That was wrong.** surya-ocr-2's vocabulary is
+**byte-level**, and an accented letter is two tokens:
+
+```
+'à' utf8=[195, 160]  ids=[2131, 2258]  pieces=['Ã', 'ł']
+'è' utf8=[195, 168]  ids=[2131, 2104]  pieces=['Ã', '¨']
+
+decode([2131])       = '?'     <- a lone lead byte gives exactly ONE marker
+decode([2131, 2104]) = 'è'
+```
+
+So **one marker per accented letter is exactly the signature of a byte-level failure** —
+the thing it was claimed to rule out. The vocabulary lookup was misread too: `vocab['à']
+= 2160` does not mean the letter is token 2160; in a byte-mapped vocabulary that key is
+the glyph standing for byte `0xA0`, and `decode([2160])` is a single marker as well. The
+"one marker vs two" discriminator does not exist.
+
+There is a second, cheaper discriminator that does: **run the page through `chandra`**.
 It is a different model family with a different tokenizer (Qwen3-VL, against
 surya-2's own 65k `WordLevel` vocabulary), reached over the same transport, at the
 same pixel ceiling, from the same rasterised page. Accents intact under chandra