ocr: empty bytes are absence, not cleanliness (v0.3.29)

CORRECTION. v0.3.26-0.3.28 reported `blame: "the serving path"` with
`conclusive: true` and "the byte stream decoded cleanly", on data that was not
there. Measured through the live gateway today:

    result.raw (what surya reads) : 3 markers  'nt ? propriet? della propriet? d'
    reassembled from logprobs     : 0 markers  'nt  propriet della propriet ...'
    entries with bytes=[]         : 257
    any byte >= 0x80 in stream    : False

vLLM 0.27.1 reports `bytes: []` for the byte-level tokens of surya-ocr-2's custom
vocabulary, and not one byte >= 0x80 appears anywhere in the stream. The accent
bytes are not corrupted in the logprobs, they are ABSENT from them.

_reassemble accepted an empty byte list as valid data, so it built a string with
the accents silently missing, found no invalid UTF-8 in it, and called the result
clean. That is reading absence as cleanliness -- the exact failure the envelope
in v0.3.26 was built to prevent for MISSING logprobs, walked into one level
further down.

Fixed: an empty byte list counts as missing. The trace now counts
`tokens_without_bytes` and, when any are present, reports `conclusive: false`
with a `why` that says the loss is inside the serving stack (it produced neither
usable bytes nor usable text) while stating plainly that this CANNOT say whether
the model's own output was sound. `unreadable.blame` is then "unknown" rather
than an accusation the data does not support.

Consequence for surya_repair_accents: it CANNOT work on vLLM 0.27.1 -- there is
nothing to rebuild from, and that is not a tuning problem. It is kept because its
guards refuse correctly: the ASCII-skeleton check catches the mismatch, so a page
comes back with its markers rather than with the accents quietly deleted, and the
repair will work on any stack that does report the bytes.

So on vLLM 0.27.1 the real options are `chandra` (reads the same pixels
perfectly) or the `coderai-vllm-ocr` image pinned to the vLLM surya validates
against.

tests: +5 in test_ocr_token_trace.py (42 total) pinning that empty bytes are
refused by both the reassembly and the repair, that the blame goes to "unknown"
rather than the serving path, and that a FULLY reported clean stream still
produces the genuine verdict. 643 passed across the ocr/front/config/admin
selection with the same 8 pre-existing failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.28"
+__version__ = "0.3.29"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/ocr/workers/ocr_worker.py

@@ -333,8 +333,15 @@ def _reassemble(tokens) -> str:
     for t in tokens:
         e = _tok_entry(t)
         raw = e.get("bytes")
-        if not isinstance(raw, list):
-            return None                 # a server that reports no bytes: no repair
+        if not isinstance(raw, list) or not raw:
+            # EMPTY counts as missing, and getting this wrong once already
+            # produced a confidently wrong answer. vLLM 0.27.1 reports
+            # ``bytes: []`` for the byte-level tokens of surya-ocr-2's custom
+            # vocabulary -- 257 of them on one page, with not a single byte
+            # >= 0x80 anywhere in the stream. Accepting that as data built a
+            # string with the accents silently absent, found no invalid UTF-8
+            # in it, and called the result clean. Absence is not cleanliness.
+            return None
         for b in raw:
             buf.append(int(b) & 0xFF)
     if not buf:
@@ -394,9 +401,21 @@ def _trace_report(calls) -> dict:
     """
     calls = list(calls or [])
     with_lp = sum(1 for c in calls if c)
+    # How many tokens the server declined to give bytes for. A token with
+    # ``bytes: []`` carries no information, and a stream full of them decodes
+    # "cleanly" only because there is nothing in it to be wrong -- so this has to
+    # be counted and reported, not quietly skipped.
+    no_bytes = 0
+    for c in calls:
+        for t in (c or []):
+            e = _tok_entry(t)
+            raw = e.get("bytes")
+            if not isinstance(raw, list) or not raw:
+                no_bytes += 1
     report = {
         "calls": len(calls),
         "with_logprobs": with_lp,
+        "tokens_without_bytes": no_bytes,
         "findings": _trace_findings(calls) if with_lp else [],
     }
     if not calls:
@@ -407,12 +426,27 @@ def _trace_report(calls) -> dict:
         report["conclusive"] = False
         report["why"] = ("the model server returned no logprobs, so the bytes "
                          "could not be read -- this says nothing about the cause")
+    elif no_bytes:
+        # The decisive case on vLLM 0.27.1 + surya-ocr-2: the server answers with
+        # logprobs but gives ``bytes: []`` for the very tokens whose characters
+        # went missing, so there is nothing to check and nothing to rebuild from.
+        # That is still an ANSWER -- it localises the loss inside the server,
+        # which produced neither usable bytes nor usable text -- but it is not
+        # the same statement as "the model's bytes were sound".
+        report["conclusive"] = False
+        report["why"] = (
+            f"the server reported no bytes for {no_bytes} token(s), including the "
+            f"ones whose characters are missing, so the byte stream could not be "
+            f"checked. The loss is inside the serving stack -- it produced neither "
+            f"usable bytes nor usable text -- but this cannot say whether the "
+            f"MODEL's own output was sound, and nothing can be rebuilt from it")
     else:
         report["conclusive"] = True
         if not report["findings"]:
-            report["why"] = ("the byte stream decoded cleanly, so the characters "
-                             "were lost AFTER the model: the fault is in the "
-                             "serving path, not the model")
+            report["why"] = ("the byte stream decoded cleanly and every token "
+                             "reported its bytes, so the characters were lost "
+                             "AFTER the model: the fault is in the serving path, "
+                             "not the model")
         else:
             kinds = {}
             for f in report["findings"]:

docs/ocr.md

@@ -570,7 +570,44 @@ A third, independent discriminator needs no code at all: serve the **same weight
 llama.cpp** (`datalab-to/surya-ocr-2-gguf`, with `surya_serve: "llamacpp"`). A different
 detokenizer on identical weights — accents surviving there would convict vLLM outright.
 
-### The verdict, and the repair (`surya_repair_accents`)
+### The verdict — measured, and it corrects an earlier one here
+
+**`surya_repair_accents` cannot work on vLLM 0.27.1, and an earlier version of this
+section claimed a verdict it had not earned.** Both facts come from the same
+measurement, taken through the live gateway on 2026-10-12:
+
+```
+result.raw (what surya reads) : 3 markers   'nt ? propriet? della propriet? d'
+reassembled from logprobs     : 0 markers   'nt  propriet della propriet ...'
+entries with bytes=[]         : 257
+any byte >= 0x80 in stream    : False
+```
+
+vLLM reports **`bytes: []`** for the byte-level tokens of surya-ocr-2's custom
+vocabulary, and **not one byte ≥ 0x80 appears anywhere in the stream**. The accent
+bytes are not corrupted in the logprobs — they are *absent* from them. So:
+
+* **the repair has nothing to rebuild from.** It is not a tuning problem; the data
+  does not exist. Its guards do refuse correctly (the ASCII-skeleton check catches
+  the mismatch), so a page comes back with its markers rather than with the accents
+  silently deleted — but it cannot fix anything on this stack.
+* **the trace's first verdict was vacuous.** `_reassemble` accepted an empty byte
+  list as data, built a string with the accents simply missing, found no invalid
+  UTF-8 in it, and reported `conclusive: true — the byte stream decoded cleanly, the
+  fault is in the serving path`. It was reading absence as cleanliness: the same
+  failure the envelope was built to prevent for *missing logprobs*, one level
+  further down. It now counts `tokens_without_bytes` and reports
+  `conclusive: false` with `blame: "unknown"` when the server declines to give
+  bytes.
+
+What the measurement *does* establish: the loss is **inside the serving stack**,
+which produced neither usable bytes nor usable text. It cannot say whether the
+model's own output was sound, and the honest trace now says exactly that.
+
+So on vLLM 0.27.1 the options are **`chandra`** (which reads the same pixels
+perfectly) or **`coderai-vllm-ocr`** below.
+
+### The repair, and why it is still shipped (`surya_repair_accents`)
 
 The trace was run. On a page of crisp synthetic Italian, served through the Aruba
 install on vLLM 0.27.1:

tests/test_ocr_token_trace.py

@@ -376,3 +376,58 @@ def test_the_engine_tells_the_worker_at_load():
     cfg.surya_token_trace = True
     assert SuryaEngine(cfg)._worker_opts()["token_trace"] is True
     assert SuryaEngine(OcrConfig())._worker_opts()["token_trace"] is False
+
+
+# ----------------------------- empty bytes are absence, not cleanliness #
+def test_empty_bytes_are_not_treated_as_data():
+    """MEASURED on vLLM 0.27.1 + surya-ocr-2: the server answers with logprobs
+    but gives `bytes: []` for the byte-level tokens of surya's custom vocabulary
+    — 257 of them on one page, with not a single byte >= 0x80 anywhere in the
+    stream. Accepting that as data built a string with the accents silently
+    absent, found no invalid UTF-8 in it, and reported `conclusive: true, the
+    byte stream decoded cleanly`. That is reading absence as cleanliness."""
+    assert W._reassemble([{"token": "a", "bytes": [97]},
+                          {"token": "�", "bytes": []}]) is None
+
+
+def test_a_stream_full_of_empty_bytes_is_inconclusive():
+    calls = [[{"token": "p", "bytes": [112]},
+              {"token": "i", "bytes": [105]},
+              {"token": "�", "bytes": []}]]
+    rep = W._trace_report(calls)
+    assert rep["tokens_without_bytes"] == 1
+    assert rep["conclusive"] is False
+    assert "no bytes" in rep["why"]
+    assert "nothing can be rebuilt" in rep["why"]
+
+
+def test_the_blame_is_unknown_when_the_bytes_were_not_reported():
+    """It must NOT claim the model's output was sound — that is exactly the
+    overclaim this case produced before."""
+    from codai.ocr.base import OcrPage, unreadable_report
+    page = OcrPage(index=0, text="pi�")
+    page.meta["unreadable_trace"] = {"calls": 1, "with_logprobs": 1,
+                                     "tokens_without_bytes": 257,
+                                     "conclusive": False, "findings": []}
+    assert unreadable_report([page])["blame"] == "unknown"
+
+
+def test_a_fully_reported_clean_stream_still_blames_the_serving_path():
+    """The genuine version of the verdict: every token reported its bytes AND
+    they decoded."""
+    rep = W._trace_report([[{"token": "p", "bytes": [112]},
+                            {"token": "Ã", "bytes": [195]},
+                            {"token": "ł", "bytes": [160]}]])
+    assert rep["tokens_without_bytes"] == 0
+    assert rep["conclusive"] is True
+    assert "every token reported its bytes" in rep["why"]
+
+
+def test_the_repair_refuses_when_bytes_are_empty():
+    """And so the 29 markers reached the caller untouched rather than being
+    replaced by a string with the accents quietly missing."""
+    UN = "�"
+    broken = "propriet" + UN
+    tokens = [{"token": c, "bytes": [ord(c)]} for c in "propriet"] + \
+             [{"token": UN, "bytes": []}]
+    assert W.repair_content(broken, tokens) == (None, 0)