ocr: U+FFFD is the model saying it cannot read a glyph, not a broken encoding (v0.3.21)

Accented letters came back from surya as U+FFFD -- "piu`" as "pi?", "facolta`"
as "facolt?", nine of them in one page, as real EF BF BD bytes. That is exactly
what a byte-boundary decoding bug looks like: a` is C3 A0, and byte-level BPE
splitting it across two tokens is the classic cause.

It is not that. U+FFFD is a REAL ENTRY in surya-ocr-2's vocabulary -- id 65421
of 65425, the only id that decodes to it, with <unk> a separate token at 65424.
The tokenizer is WordLevel and character-level for Latin text, so a` is a single
token (2160) and an accent CANNOT be split across two: no partial-UTF-8
artefact is possible, and per-token decoding is byte-identical to decoding the
whole string. Surya reads text only from message.content, never from the
per-token logprobs, and the fan-out gateway forwards the upstream body as raw
bytes. So the model predicted "there is a character here I cannot identify" and
every hop delivered that faithfully. Two things follow.

It is a RESOLUTION problem. Surya caps its own input at 3072x2048 = 6,291,456
pixels in scale_to_fit, which is called with DEFAULTS ONLY -- no surya setting
raises it, IMAGE_DPI_HIGHRES included. Measured, for A4: 200 DPI reaches the
model as 3.89 Mpixels, 250 as 6.03, and 255 saturates it; 300 and 400 are
downscaled back to the same 6.23. The shared ocr.dpi = 200 was therefore handing
the model 62% of the pixels it accepts, and a diacritic sits above the x-height
at ~3 pixels tall. So DPI is now chosen per engine (ocr.surya_dpi, default 250)
and resolved BEFORE rasterising; paddle and docTR have their own optima and keep
ocr.dpi. The lookup is generic (f"{engine}_dpi"), so another engine only needs
the field. Resolving the engine first also refuses an unknown one before a large
PDF is rasterised rather than after.

It must not be SILENT. "pi?" indexes, searches and reads wrong for ever, and for
judicial documents a known unknown beats a quiet corruption. Lines, pages and
the document now count their unreadable glyphs and locate each one with the
confidence of the line it sits in -- the finest granularity that exists, since
surya reports one mean token probability per block, not per character, and that
is also the signal worth acting on: a low-confidence line with an unreadable
glyph is a line to re-read at a higher DPI. The key is OMITTED when there are
none, so a clean response is byte-for-byte what it always was, spots are capped
with a truncated count, and the text itself is never rewritten -- stripping the
marker would destroy the only record of where the loss is.

20 tests, including the arithmetic that chose 250, kept so a later edit has to
face it. The 8 unrelated failures in test_broker_config /
test_studio_composed_surfaces / test_whisper_server_local_models are
pre-existing -- verified identical in a clean worktree at HEAD.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.20"
+__version__ = "0.3.21"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -4117,6 +4117,7 @@ def build_settings_dict(c, gpu_cards):
             "enabled": c.ocr.enabled,
             "default_engine": c.ocr.default_engine,
             "dpi": c.ocr.dpi,
+            "surya_dpi": c.ocr.surya_dpi,
             "max_concurrency": c.ocr.max_concurrency,
             "lang": c.ocr.lang,
             "paddle_enabled": c.ocr.paddle_enabled,
@@ -4914,6 +4915,7 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
         if "enabled" in d: o.enabled = bool(d["enabled"])
         if "default_engine" in d: o.default_engine = (d.get("default_engine") or "paddle").strip()
         if "dpi" in d: o.dpi = _i("dpi", o.dpi, 36)
+        if "surya_dpi" in d: o.surya_dpi = _i("surya_dpi", o.surya_dpi, 0)
         if "max_concurrency" in d: o.max_concurrency = _i("max_concurrency", o.max_concurrency, 1)
         if "lang" in d: o.lang = (d.get("lang") or "it").strip()
         # PaddleOCR

codai/admin/templates/settings.html

@@ -1102,6 +1102,15 @@
         <label class="form-label">PDF DPI</label>
         <input type="number" id="s-ocr-dpi" class="form-input" min="72" placeholder="200">
       </div>
+      <div class="form-group" style="flex:1;min-width:110px">
+        <label class="form-label">Surya PDF DPI</label>
+        <input type="number" id="s-ocr-surya-dpi" class="form-input" min="0" placeholder="250">
+        <span class="form-hint">Surya-2 only (0 = use PDF DPI). Its input is capped at
+        3072&times;2048 = 6.29&nbsp;Mpixels and that cap is not configurable, so an A4 page
+        saturates it at about 255&nbsp;DPI &mdash; at 200 the model was seeing 62% of the
+        pixels it accepts, and a diacritic is ~3 pixels tall. Higher than 250 still works
+        (it supersamples) but buys no extra pixels at the model.</span>
+      </div>
       <div class="form-group" style="flex:1;min-width:110px">
         <label class="form-label">Max concurrency</label>
         <input type="number" id="s-ocr-max-concurrency" class="form-input" min="1" placeholder="4">
@@ -2111,6 +2120,7 @@ async function loadSettings(){
     document.getElementById('s-ocr-enabled').checked = !!ocr.enabled;
     document.getElementById('s-ocr-default-engine').value = ocr.default_engine ?? 'paddle';
     document.getElementById('s-ocr-dpi').value = ocr.dpi ?? 200;
+    document.getElementById('s-ocr-surya-dpi').value = ocr.surya_dpi ?? 250;
     document.getElementById('s-ocr-max-concurrency').value = ocr.max_concurrency ?? 4;
     document.getElementById('s-ocr-lang').value = ocr.lang ?? 'it';
     document.getElementById('s-ocr-retry-cooldown').value = ocr.build_retry_cooldown_s ?? 60;
@@ -2365,6 +2375,7 @@ async function saveSettings(silent){
       enabled: document.getElementById('s-ocr-enabled').checked,
       default_engine: document.getElementById('s-ocr-default-engine').value || 'paddle',
       dpi: parseInt(document.getElementById('s-ocr-dpi').value) || 200,
+      surya_dpi: parseInt(document.getElementById('s-ocr-surya-dpi').value) || 0,
       max_concurrency: parseInt(document.getElementById('s-ocr-max-concurrency').value) || 4,
       lang: document.getElementById('s-ocr-lang').value.trim() || 'it',
       build_retry_cooldown_s: parseFloat(document.getElementById('s-ocr-retry-cooldown').value) || 0,

codai/config.py

@@ -899,6 +899,17 @@ class OcrConfig:
     enabled: bool = False
     default_engine: str = "paddle"        # paddle|doctr|surya|olmocr
     dpi: int = 200                        # PDF rasterisation DPI
+    # Per-engine PDF rasterisation DPI (0 = use `dpi` above). Surya-2 gets its own
+    # because its input is capped at 3072x2048 = 6.29 Mpixels: `scale_to_fit` in
+    # surya/inference/util.py is called with DEFAULTS ONLY, so no surya setting can
+    # raise it, and anything larger is LANCZOS-downscaled to fit. An A4 page needs
+    # ~255 DPI to reach that ceiling, so the shared 200 was handing the model 3.89 of
+    # the 6.29 Mpixels it will accept -- 62% -- and an Italian accent is ~3 pixels
+    # tall at 200 DPI. That is why accented letters came back as the unreadable-glyph
+    # marker (see base.UNREADABLE). 250 saturates A4 at 1.5x the pixels of 200;
+    # higher still works (it supersamples, which tends to help glyphs) but costs
+    # rasterisation time and memory for no extra pixels at the model.
+    surya_dpi: int = 250
     max_concurrency: int = 4             # global cap on in-flight OCR pages
     lang: str = "it"                     # default document language
     # After an engine fails to come up, refuse that engine FAST for this long instead of

codai/ocr/base.py

@@ -38,6 +38,43 @@ class OcrError(RuntimeError):
         self.status = status
 
 
+# ---------------------------------------------------------------------------
+# Unreadable glyphs
+# ---------------------------------------------------------------------------
+
+#: What a VLM OCR engine writes where it cannot identify a character.
+#:
+#: This is NOT a decoding or transport artefact, which is the natural assumption
+#: when accented letters come back as U+FFFD. In surya-ocr-2 it is a real entry in
+#: the vocabulary — id 65421 of 65425, the only id that decodes to it, and
+#: ``<unk>`` is a different token (65424). The tokenizer is ``WordLevel`` and
+#: character-level for Latin text, so ``à`` is a single token and an accent
+#: *cannot* be split across two: no partial-UTF-8 artefact is possible, and
+#: per-token decoding is byte-identical to decoding the whole string. Surya also
+#: reads its text only from ``message.content``, never from the per-token
+#: logprobs, and coderai's OCR gateway forwards the upstream body as raw bytes.
+#:
+#: So a U+FFFD in the output is the MODEL saying "there is a character here I
+#: cannot read". It is honest, and it is also silent: ``più`` becomes ``pi?`` and
+#: indexes, searches and reads wrong ever after. For Italian judicial documents
+#: that is a real loss, so it gets counted and located rather than passed through
+#: unremarked — a known unknown beats a quiet corruption.
+#:
+#: The usual cause is too few pixels: a diacritic sits above the x-height and is
+#: ~3 pixels tall at 200 DPI. See ``OcrConfig.surya_dpi``.
+UNREADABLE = "\ufffd"
+
+
+def unreadable_offsets(text: str) -> List[int]:
+    """Character offsets of every unreadable-glyph marker in ``text``."""
+    return [i for i, ch in enumerate(text or "") if ch == UNREADABLE]
+
+
+def count_unreadable(text: str) -> int:
+    """How many characters the engine could not identify."""
+    return (text or "").count(UNREADABLE)
+
+
 # ---------------------------------------------------------------------------
 # Result types
 # ---------------------------------------------------------------------------
@@ -50,7 +87,12 @@ class OcrLine:
     conf: float = 0.0
 
     def to_dict(self) -> dict:
-        return {"text": self.text, "bbox": self.bbox, "conf": round(self.conf, 4)}
+        out = {"text": self.text, "bbox": self.bbox, "conf": round(self.conf, 4)}
+        n = count_unreadable(self.text)
+        if n:
+            # Omitted when zero: a clean result stays exactly what it always was.
+            out["unreadable"] = n
+        return out
 
 
 @dataclass
@@ -95,6 +137,9 @@ class OcrPage:
         }
         if self.meta:
             out["meta"] = self.meta
+        n = count_unreadable(self.text)
+        if n:
+            out["unreadable"] = n
         return out
 
 
@@ -259,3 +304,56 @@ def vlm_serve_limits(cfg) -> dict:
         if v > 0:
             out[arg] = v
     return out
+
+
+#: At most this many individual markers are located in a document-level report.
+#: A page of pure noise could otherwise return thousands, and the count already
+#: tells the caller how bad it is.
+UNREADABLE_MAX_SPOTS = 100
+
+#: Characters of surrounding text kept with each located marker, each side.
+UNREADABLE_CONTEXT = 24
+
+
+def unreadable_report(pages: "List[OcrPage]") -> dict:
+    """Count and locate every character the engine could not identify.
+
+    Returns ``{}`` when the document is clean, so the key is simply absent from a
+    response that has nothing to report.
+
+    Each located marker carries the confidence of the line it sits in, which is
+    the finest granularity available: surya reports one mean token probability per
+    block (``PolygonBox.confidence``), not per character, so a per-character score
+    would have to be invented. ``conf`` is therefore the line's, and is the signal
+    worth acting on — a low-confidence line with an unreadable glyph is a line to
+    re-read at a higher DPI, while a high-confidence one is more likely a glyph
+    that genuinely is not in the model's repertoire.
+    """
+    total = 0
+    per_page = []
+    spots = []
+    for page in pages or []:
+        n_page = count_unreadable(getattr(page, "text", ""))
+        total += n_page
+        if n_page:
+            per_page.append({"page": page.index, "count": n_page})
+        for li, line in enumerate(getattr(page, "lines", []) or []):
+            text = getattr(line, "text", "") or ""
+            for off in unreadable_offsets(text):
+                if len(spots) >= UNREADABLE_MAX_SPOTS:
+                    break
+                lo = max(0, off - UNREADABLE_CONTEXT)
+                spots.append({
+                    "page": page.index,
+                    "line": li,
+                    "offset": off,
+                    "conf": round(float(getattr(line, "conf", 0.0) or 0.0), 4),
+                    "bbox": getattr(line, "bbox", None),
+                    "context": text[lo:off + UNREADABLE_CONTEXT + 1],
+                })
+    if not total:
+        return {}
+    report = {"count": total, "pages": per_page, "spots": spots}
+    if total > len(spots):
+        report["truncated"] = total - len(spots)
+    return report

codai/ocr/manager.py

@@ -32,7 +32,8 @@ import threading
 import time
 from typing import Dict, List, Optional
 
-from codai.ocr.base import OcrEngine, OcrPage, OcrError, load_pages
+from codai.ocr.base import (OcrEngine, OcrPage, OcrError, load_pages,
+                            unreadable_report)
 
 
 # Engine registry: name → factory(cfg) → OcrEngine. Imports are deferred inside each
@@ -512,6 +513,29 @@ class OcrManager:
             )
         return name
 
+    def engine_dpi(self, requested: Optional[str]) -> int:
+        """PDF rasterisation DPI for the engine that will serve this request.
+
+        The DPI has to be chosen BEFORE rasterising, and the engine decides it:
+        surya-2 accepts a far larger image than the shared default was feeding it
+        (see ``OcrConfig.surya_dpi``), while paddle and docTR have their own optima
+        and are left alone. An explicit per-request ``dpi`` wins over both.
+
+        Resolving the engine here also means an unknown or disabled engine is
+        refused before a large PDF is rasterised rather than after it. If the engine
+        later fails over to another one (``_with_fallback``), the pages keep the DPI
+        chosen for the engine that was asked for — re-rasterising mid-request would
+        cost more than the slightly-off resolution does.
+        """
+        cfg = self._require_cfg()
+        try:
+            name = self.resolve_engine(requested)
+        except OcrError:
+            # Let the real failure come from the engine path, with its own message.
+            return int(cfg.dpi)
+        per_engine = int(getattr(cfg, f"{name}_dpi", 0) or 0)
+        return per_engine if per_engine > 0 else int(cfg.dpi)
+
     async def _get_pool(self, name: str) -> _Pool:
         async with self._lock:
             pool = self._pools.get(name)
@@ -593,7 +617,7 @@ class OcrManager:
         configured text model; ``schema`` overrides the extraction schema for this call.
         """
         cfg = self._require_cfg()
-        use_dpi = int(dpi) if dpi else int(cfg.dpi)
+        use_dpi = int(dpi) if dpi else self.engine_dpi(engine)
         images = load_pages(data, filename=filename, content_type=content_type, dpi=use_dpi)
         # Share the GPU thermal governor with every other workload: block here until
         # temps are safe (wait_until_safe blocks, so run it off the event loop).
@@ -608,7 +632,7 @@ class OcrManager:
             from codai.ocr.extract import extract_fields
             structured_out = await extract_fields(full_text, cfg, schema=schema)
 
-        return {
+        out = {
             "engine": name,
             "num_pages": len(pages),
             "text": full_text,
@@ -617,6 +641,10 @@ class OcrManager:
             "signatures": signatures,
             "structured": structured_out,
         }
+        report = unreadable_report(pages)
+        if report:
+            out["unreadable"] = report
+        return out
 
     async def _detect_all(self, images, pages, detect: Optional[str]):
         """Run stamp/signature detection across pages; returns (stamps, signatures)

docs/ocr.md

@@ -247,7 +247,7 @@ back.
 ## Enable & configure
 
 Settings → **OCR** card, or `config.json` `"ocr"`. Key fields: `enabled`,
-`default_engine`, `dpi`, `max_concurrency`, `lang`, `build_retry_cooldown_s`,
+`default_engine`, `dpi`, `surya_dpi`, `max_concurrency`, `lang`, `build_retry_cooldown_s`,
 `vlm_gpu_memory_utilization`, `vlm_max_num_batched_tokens`, `vlm_max_num_seqs`,
 `vlm_max_model_len`, `fallback_engines`, `evict_drain_timeout_s`;
 per-engine `*_enabled` /
@@ -321,6 +321,68 @@ orchestrator: surya attaches straight to the endpoint the pool hands it. A pod r
 only the code baked into its image, so an install whose pods predate this needs its
 capability image rebuilt before the numbers appear.
 
+### Unreadable glyphs, and why Italian accents were the first to go
+
+Accented letters came back from surya as `U+FFFD` — `più` as `pi?`, `facoltà` as
+`facolt?` — nine of them in one page of raw JSON, as real bytes `EF BF BD`. That
+looks exactly like a decoding bug, and it is not one.
+
+`U+FFFD` is **a real entry in surya-ocr-2's vocabulary**: id 65421 of 65425, the
+only id that decodes to it, with `<unk>` a separate token at 65424. The tokenizer
+is `WordLevel` and character-level for Latin text, so `à` is a *single* token
+(2160) and an accent cannot be split across two — no partial-UTF-8 artefact is
+possible, and per-token decoding is byte-identical to decoding the whole string.
+Surya reads its text only from `message.content`, never from the per-token
+logprobs, and coderai's fan-out gateway forwards the upstream body as raw bytes.
+
+So the model **predicted** "there is a character here I cannot identify", and
+every hop delivered that faithfully. Two things follow.
+
+**It is a resolution problem.** Surya caps its own input at 3072×2048 =
+6,291,456 pixels in `scale_to_fit`, which is called with **defaults only** — no
+surya setting raises it, `IMAGE_DPI_HIGHRES` included. Measured, for A4:
+
+| `dpi` | rasterised | Mpixels | model sees | Mpixels |
+|------:|-----------:|--------:|-----------:|--------:|
+| 200 | 1654×2338 | 3.87 | 1652×2352 | 3.89 |
+| 250 | 2067×2922 | 6.04 | 2072×2912 | 6.03 |
+| 255 | 2108×2980 | 6.28 | 2100×2968 | 6.23 |
+| 300 | 2481×3507 | 8.70 | 2100×2968 | 6.23 |
+| 400 | 3308×4676 | 15.47 | 2100×2968 | 6.23 |
+
+The shared `ocr.dpi = 200` was therefore handing the model 62% of the pixels it
+accepts, and a diacritic sits above the x-height at ~3 pixels tall. `ocr.surya_dpi`
+(default **250**) is surya's own rasterisation DPI for this reason; paddle and
+docTR have their own optima and keep `ocr.dpi`. Above ~255 nothing more reaches
+the model, though supersampling down from 300–400 can still render glyphs more
+cleanly, at the cost of rasterisation time and memory.
+
+`dpi` only affects **PDF** input. An uploaded image is used at the resolution it
+arrives, so a low-resolution scan cannot be rescued by this setting.
+
+**It must not be silent.** `pi?` indexes, searches and reads wrong for ever, and
+for judicial documents a known unknown beats a quiet corruption. Every result now
+counts its unreadable glyphs, and the key is **omitted entirely when there are
+none**, so a clean response is byte-for-byte what it always was:
+
+```json
+{ "text": "...", "pages": [ { "index": 3, "unreadable": 2,
+    "lines": [ { "text": "la pi\ufffd alta", "conf": 0.41, "unreadable": 1 } ] } ],
+  "unreadable": {
+    "count": 2,
+    "pages": [ { "page": 3, "count": 2 } ],
+    "spots": [ { "page": 3, "line": 0, "offset": 5, "conf": 0.41,
+                 "bbox": [...], "context": "la pi\ufffd alta facolt" } ] } }
+```
+
+The `conf` on a spot is the **line's** — the finest granularity that exists, since
+surya reports one mean token probability per block, not per character. It is also
+the useful signal: a low-confidence line with an unreadable glyph is a line to
+re-read at a higher DPI, while a high-confidence one is more likely a glyph that is
+genuinely not in the model's repertoire. `spots` is capped (100) with a
+`truncated` count, and the text itself is never rewritten — stripping the marker
+would destroy the only record of where the loss is.
+
 ### Scaling in `model` mode
 
 An OCR pipeline drives one server address itself — that is what keeps the page

tests/test_ocr_unreadable_and_dpi.py

+"""Accented letters came back from surya as U+FFFD, and it looked like an
+encoding bug. It was not one.
+
+U+FFFD is a real entry in surya-ocr-2's vocabulary: id 65421 of 65425, the only
+id that decodes to it, with ``<unk>`` a separate token at 65424. The tokenizer is
+``WordLevel`` and character-level for Latin text, so ``a`` with an accent is ONE
+token and an accent cannot be split across two -- no partial-UTF-8 artefact is
+possible, and per-token decoding is byte-identical to decoding the whole string.
+Surya reads text only from ``message.content``, and coderai's gateway forwards
+the upstream body as raw bytes. So the model PREDICTED "I cannot read this
+glyph", and the pipeline delivered that faithfully.
+
+Two things follow, and these are the tests for both:
+
+1. It is a resolution problem. ``scale_to_fit`` caps surya's input at
+   3072x2048 = 6.29 Mpixels and is called with DEFAULTS ONLY, so no surya
+   setting raises it. An A4 page reaches that at ~255 DPI, so the shared
+   ``ocr.dpi = 200`` was handing the model 3.89 of 6.29 Mpixels -- 62% -- and a
+   diacritic is ~3 pixels tall at 200 DPI.
+2. It must stop being silent. ``piu`` with the accent eaten indexes and searches
+   wrong for ever; for judicial documents a known unknown beats a quiet
+   corruption.
+"""
+
+import pathlib
+import sys
+
+ROOT = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from codai.ocr.base import (UNREADABLE, OcrLine, OcrPage, count_unreadable,   # noqa: E402
+                            unreadable_offsets, unreadable_report)
+from codai.ocr import manager as om                                           # noqa: E402
+
+
+# --------------------------------------------------------------- the marker #
+def test_the_marker_is_the_replacement_character():
+    assert UNREADABLE == "�"
+    assert UNREADABLE.encode("utf-8") == b"\xef\xbf\xbd"
+
+
+def test_counting_and_locating():
+    text = "pi� facolt�"
+    assert count_unreadable(text) == 2
+    assert unreadable_offsets(text) == [2, 10]
+
+
+def test_clean_text_is_clean():
+    assert count_unreadable("perché così più") == 0
+    assert unreadable_offsets("perché") == []
+
+
+def test_empty_and_none_are_safe():
+    assert count_unreadable("") == 0
+    assert count_unreadable(None) == 0
+    assert unreadable_offsets(None) == []
+
+
+# ------------------------------------------------ reporting, without noise #
+def test_a_clean_line_is_byte_for_byte_what_it_always_was():
+    """The key is omitted when there is nothing to say, so every existing
+    consumer of a clean result sees exactly the response it saw before."""
+    d = OcrLine(text="perché", bbox=[0, 0, 10, 10], conf=0.9).to_dict()
+    assert d == {"text": "perché", "bbox": [0, 0, 10, 10], "conf": 0.9}
+    assert "unreadable" not in d
+
+
+def test_a_dirty_line_says_so():
+    d = OcrLine(text="pi�", bbox=[0, 0, 1, 1], conf=0.4).to_dict()
+    assert d["unreadable"] == 1
+
+
+def test_a_clean_page_does_not_grow_a_key():
+    d = OcrPage(index=0, text="tutto bene").to_dict()
+    assert "unreadable" not in d
+
+
+def test_a_dirty_page_says_so():
+    d = OcrPage(index=0, text="pi� e facolt�").to_dict()
+    assert d["unreadable"] == 2
+
+
+def test_a_clean_document_reports_nothing_at_all():
+    assert unreadable_report([OcrPage(index=0, text="va bene")]) == {}
+    assert unreadable_report([]) == {}
+    assert unreadable_report(None) == {}
+
+
+def test_the_document_report_counts_locates_and_scores():
+    page = OcrPage(
+        index=3,
+        text="la pi� alta facolt�",
+        lines=[OcrLine(text="la pi� alta", bbox=[1, 2, 3, 4], conf=0.41),
+               OcrLine(text="facolt�", bbox=[5, 6, 7, 8], conf=0.88)],
+    )
+    r = unreadable_report([page])
+    assert r["count"] == 2
+    assert r["pages"] == [{"page": 3, "count": 2}]
+    assert [s["line"] for s in r["spots"]] == [0, 1]
+    # The confidence carried is the LINE's -- the finest surya reports.
+    assert r["spots"][0]["conf"] == 0.41
+    assert r["spots"][1]["conf"] == 0.88
+    assert r["spots"][0]["bbox"] == [1, 2, 3, 4]
+    # Context makes a report readable without fetching the page back.
+    assert "pi�" in r["spots"][0]["context"]
+
+
+def test_the_report_counts_pages_separately():
+    r = unreadable_report([OcrPage(index=0, text="pi�"),
+                           OcrPage(index=1, text="ok"),
+                           OcrPage(index=2, text="a�b�")])
+    assert r["count"] == 3
+    assert r["pages"] == [{"page": 0, "count": 1}, {"page": 2, "count": 2}]
+
+
+def test_a_page_of_pure_noise_cannot_return_a_megabyte():
+    """The count already says how bad it is; the located spots are capped."""
+    from codai.ocr.base import UNREADABLE_MAX_SPOTS
+    line = OcrLine(text=UNREADABLE * 500, bbox=[0, 0, 1, 1], conf=0.1)
+    page = OcrPage(index=0, text=UNREADABLE * 500, lines=[line])
+    r = unreadable_report([page])
+    assert r["count"] == 500
+    assert len(r["spots"]) == UNREADABLE_MAX_SPOTS
+    assert r["truncated"] == 500 - UNREADABLE_MAX_SPOTS
+
+
+def test_the_text_itself_is_left_alone():
+    """Rewriting or stripping the marker would destroy the one thing that says
+    WHERE the loss is. The report is additive."""
+    page = OcrPage(index=0, text="pi�")
+    assert page.to_dict()["text"] == "pi�"
+
+
+# --------------------------------------------------------- per-engine DPI #
+class _Cfg:
+    def __init__(self, **kw):
+        self.dpi = 200
+        self.surya_dpi = 250
+        self.default_engine = "surya"
+        for k, v in kw.items():
+            setattr(self, k, v)
+
+
+def _mgr(cfg, resolved="surya"):
+    m = om.OcrManager.__new__(om.OcrManager)
+    m._require_cfg = lambda: cfg
+    m.resolve_engine = lambda requested: resolved
+    return m
+
+
+def test_surya_gets_its_own_dpi():
+    assert _mgr(_Cfg()).engine_dpi("surya") == 250
+
+
+def test_paddle_is_left_on_the_shared_default():
+    """paddle and docTR have their own optima and are not surya's business."""
+    assert _mgr(_Cfg(), resolved="paddle").engine_dpi("paddle") == 200
+
+
+def test_zero_means_use_the_shared_default():
+    assert _mgr(_Cfg(surya_dpi=0)).engine_dpi("surya") == 200
+
+
+def test_a_missing_per_engine_field_is_not_an_error():
+    cfg = _Cfg()
+    del cfg.surya_dpi
+    assert _mgr(cfg).engine_dpi("surya") == 200
+
+
+def test_an_unresolvable_engine_falls_back_rather_than_raising_here():
+    """The engine path owns that error and words it properly; rasterising must
+    not be where an unknown engine surfaces."""
+    from codai.ocr.base import OcrError
+    m = om.OcrManager.__new__(om.OcrManager)
+    cfg = _Cfg()
+    m._require_cfg = lambda: cfg
+
+    def _boom(requested):
+        raise OcrError("unknown OCR engine 'nope'", status=400)
+    m.resolve_engine = _boom
+    assert m.engine_dpi("nope") == 200
+
+
+def test_an_explicit_request_dpi_still_wins():
+    import inspect
+    src = inspect.getsource(om.OcrManager.ocr_document)
+    assert "int(dpi) if dpi else self.engine_dpi(engine)" in src
+
+
+def test_the_default_saturates_an_a4_page_without_overshooting_much():
+    """The ceiling is surya's, hard-coded and called with defaults:
+    3072x2048 = 6291456 pixels. This is the arithmetic that chose 250, kept here
+    so a later edit to surya_dpi has to face it.
+    """
+    cap = 3072 * 2048
+    a4_w_in, a4_h_in = 8.27, 11.69
+    px = lambda dpi: int(a4_w_in * dpi) * int(a4_h_in * dpi)   # noqa: E731
+    assert px(200) / cap < 0.63            # the old default wasted 37%
+    assert 0.9 < px(250) / cap <= 1.0      # the new one saturates it
+    assert px(300) > cap                   # beyond here surya downscales again