a coarser scan is not a saving, it is a different page (v0.4.12)

**The accent regression, reported from production hours after 0.4.7 shipped.**
Italian accents stopped surviving surya again. The cause was this afternoon's
scan passthrough, not the old detokenisation fault: its guard used an ABSOLUTE
150-DPI floor instead of the DPI the page would have been rendered at. surya's
own default is 250 (`surya_dpi`), and Aruba leaves it unset, so every scanned
page holding a 150-200 DPI JPEG went to the model at native size -- 25-40 % less
linear resolution than the render it replaced -- and surya-ocr-2 is measured to
read badly below 250.

The reasoning error is the part worth keeping: passing a scan through was
always meant to be "the same pixels, without the work". Fewer pixels is not a
saving. The floor is now the TARGET DPI, and `page_image_min_dpi` (default 0)
can only RAISE it. Verified against a 250-DPI engine: a 150 or 200 DPI scan is
rendered up as before; 250 and 300 DPI scans still pass through, which is where
the shortcut was always legitimate.

It also honours the REQUEST. `ocr_document` resolves
`use_dpi = int(dpi) if dpi else engine_dpi(engine)` and hands that one value to
the pod forward, the local render and (on the pod) this guard -- so "is this
scan detailed enough" is asked about what the caller asked for, never about a
default they overrode. Pinned, because a future change reading the engine
default would silently hand a 200-DPI scan to a request that wanted 300.

**And the rasterisation hop now encodes, instead of shipping raw pixels.**
Measured on A4 at 250 DPI (6.04 Mpix, surya's own default):

    raw RGB over the pipe        18.1 MB   + 109.7 ms for the parent to encode
                                           it for the engine anyway
    PNG level 1 in the worker     0.72 MB  104.7 ms, in the WORKER
    the parent's lazy open                   0.12 ms

The encode is not added, it MOVES: off the single parent process onto the
raster pool, which is parallel. The pipe carries 25x less, the second encode
disappears, and the parent holds 0.72 MB per page in flight instead of 18.1 MB
-- at 48 pages that is 35 MB instead of 870 MB, which is the quantity this
whole subsystem keeps getting into trouble over. PNG, not JPEG, unless the
operator asked: a page about to be read must not pass through a lossy codec by
default. Raw RGB stays the fallback, so the protocol still reads on an older
parent.

Seven existing tests had encoded the wrong floor and were corrected; one of
them hardcoded pixel counts that assumed A4 while the fixture is 609 pt, so it
measured 244 DPI while claiming 250.

11 new tests. 3368 passed, 13 pre-existing failures (broker, studio, manual
client) untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.11"
+__version__ = "0.4.12"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/templates/settings.html

@@ -1183,13 +1183,14 @@
         <span class="form-hint">Only used when the format above is JPEG.</span>
       </div>
       <div class="form-group" style="flex:1;min-width:150px">
-        <label class="form-label">Pass a scan through above (DPI)</label>
-        <input type="number" id="s-ocr-page-image-min-dpi" class="form-input" min="0" placeholder="150">
-        <span class="form-hint">A scan coarser than this is rendered at the configured DPI
-        instead, so the model still gets the pixel density the rest of the pipeline was
-        tuned for. A scan is also rendered when it is more than
-        <span id="s-ocr-page-image-factor-label">3</span>&times; the target, where
-        rendering down is cheaper than shipping it whole.</span>
+        <label class="form-label">Extra floor for passing a scan through (DPI)</label>
+        <input type="number" id="s-ocr-page-image-min-dpi" class="form-input" min="0" placeholder="0">
+        <span class="form-hint"><b>0 = the engine&rsquo;s own DPI</b>, which is the only
+        setting under which a passed-through scan cannot change what the model sees: a
+        coarser scan is rendered up instead, exactly as before. Raise this only to demand
+        <i>more</i> than the target. A scan is also rendered when it exceeds
+        <span id="s-ocr-page-image-factor-label">3</span>&times; the target, where rendering
+        down is cheaper than shipping it whole.</span>
       </div>
       <div class="form-group" style="flex:1;min-width:130px">
         <label class="form-label">&hellip;and below this &times; DPI</label>
@@ -2326,7 +2327,7 @@ async function loadSettings(){
     document.getElementById('s-ocr-pod-http-async').value = (ocr.pod_http_async ?? true) ? '1' : '0';
     document.getElementById('s-ocr-page-image-format').value = ocr.page_image_format ?? 'auto';
     document.getElementById('s-ocr-page-image-quality').value = ocr.page_image_jpeg_quality ?? 95;
-    document.getElementById('s-ocr-page-image-min-dpi').value = ocr.page_image_min_dpi ?? 150;
+    document.getElementById('s-ocr-page-image-min-dpi').value = ocr.page_image_min_dpi ?? 0;
     document.getElementById('s-ocr-page-image-max-factor').value = ocr.page_image_max_dpi_factor ?? 3;
     document.getElementById('s-ocr-page-image-factor-label').textContent =
       String(ocr.page_image_max_dpi_factor ?? 3);

codai/config.py

@@ -1099,14 +1099,18 @@ class OcrConfig:
     pod_http_async: bool = True
     page_image_format: str = "auto"
     page_image_jpeg_quality: int = 95
-    # The bounds within which a scan may be passed through. Below the floor the
-    # embedded image is genuinely coarser than the configured DPI and rendering
-    # it up at least gives the model the pixel density the rest of the pipeline
-    # was tuned for; far above the target, rendering DOWN is cheaper than
-    # shipping the whole thing for the engine to scale. Expressed in real DPI
-    # rather than as a ratio because "is 150 DPI enough to read" is a question
-    # about documents, which an operator can answer.
-    page_image_min_dpi: float = 150.0
+    # An ADDITIONAL floor on passing a scan through, in real DPI. 0 means the
+    # floor is the engine's own target DPI, which is the only setting under
+    # which a passed-through scan cannot change what the model sees.
+    #
+    # It shipped as an absolute 150 and that was wrong: a 150-DPI scan then
+    # reached a 250-DPI engine at native size -- 40 % less linear resolution
+    # than the render it replaced -- and surya-ocr-2 lost Italian accents on
+    # those pages. Reported from production the same afternoon. The point of
+    # the passthrough was "the same pixels without the work"; fewer pixels is
+    # not a saving, it is a different page. Raise this only to demand MORE than
+    # the target.
+    page_image_min_dpi: float = 0.0
     page_image_max_dpi_factor: float = 3.0
     # How many pages of one document are rasterised at a time. The document is read
     # in windows, so residency is O(window) instead of O(document): a 400-page

codai/ocr/workers/page_image.py

@@ -144,7 +144,7 @@ def data_uri(mime: str, raw: bytes) -> str:
 _OBJ_IMAGE = 3
 
 
-def extract_embedded(page, dpi: int, min_dpi: float = 150.0,
+def extract_embedded(page, dpi: int, min_dpi: float = 0.0,
                      max_dpi_factor: float = 3.0):
     """``(mime, raw, (w, h))`` if this page IS one embedded JPEG, else ``None``.
 
@@ -169,11 +169,20 @@ def extract_embedded(page, dpi: int, min_dpi: float = 150.0,
       error.
     * **It covers the page.** 98 % in both axes, anchored at the origin, or the
       model is being shown a crop.
-    * **Between ``min_dpi`` and ``max_dpi_factor`` x the target.** Below the
-      floor the scan is genuinely coarser than the configured DPI and rendering
-      it up at least hands the model the pixel density the rest of the pipeline
-      was tuned for; far above it, rendering DOWN to the target is cheaper than
-      shipping the whole thing and letting the engine scale it.
+    * **At least the TARGET DPI, and at most ``max_dpi_factor`` x it.** This
+      guard was wrong when it shipped and the error is worth recording: it used
+      an absolute floor (150 DPI) instead of the DPI the page would have been
+      rendered at. A 150-DPI scan then went to a 250-DPI engine at its native
+      size -- a 40 % linear resolution drop in what the model sees -- and
+      surya-ocr-2 lost Italian accents on exactly those pages, which is a
+      regression reported from production within hours of the change.
+
+      The justification for passing a scan through was never "fewer pixels", it
+      was "the same pixels, without the work". A coarser image is not a saving,
+      it is a different page. So the floor is the target itself, and
+      ``min_dpi`` only RAISES it; above ``max_dpi_factor`` x the target,
+      rendering down is cheaper than shipping the whole thing for the engine to
+      scale.
     """
     try:
         if int(page.get_rotation() or 0) % 360 != 0:
@@ -200,11 +209,18 @@ def extract_embedded(page, dpi: int, min_dpi: float = 150.0,
             return None
         if abs(left) / pw > 0.02 or abs(bottom) / ph > 0.02:
             return None
-        # 2 % of slack on the floor, because pdfium derives the effective DPI
-        # from the image's own bounding box rather than the page: a plain
-        # 150-DPI scan of an A4 page reports 150.0 across and 149.957 down, and
-        # an exact floor would reject it over four hundredths of a dot.
-        lo = float(min_dpi) * 0.98
+        # The floor is what we WOULD have rendered; min_dpi only raises it.
+        # 0 (the default) therefore means "as detailed as the target", which is
+        # the only setting under which passing a scan through cannot change what
+        # the model sees.
+        floor = float(dpi)
+        if min_dpi and float(min_dpi) > floor:
+            floor = float(min_dpi)
+        # 2 % of slack, because pdfium derives the effective DPI from the
+        # image's own bounding box rather than the page: a plain 150-DPI scan of
+        # an A4 page reports 150.0 across and 149.957 down, and an exact floor
+        # would reject it over four hundredths of a dot.
+        lo = floor * 0.98
         hi = float(dpi) * float(max_dpi_factor)
         for eff in (float(md.horizontal_dpi), float(md.vertical_dpi)):
             if not (lo <= eff <= hi):
@@ -233,7 +249,7 @@ class Policy:
     __slots__ = ("fmt", "jpeg_quality", "min_dpi", "max_dpi_factor")
 
     def __init__(self, fmt: str = "auto", jpeg_quality: int = 95,
-                 min_dpi: float = 150.0, max_dpi_factor: float = 3.0):
+                 min_dpi: float = 0.0, max_dpi_factor: float = 3.0):
         fmt = (fmt or "auto").strip().lower()
         self.fmt = fmt if fmt in FORMATS else "auto"
         try:
@@ -243,7 +259,7 @@ class Policy:
         try:
             self.min_dpi = max(0.0, float(min_dpi))
         except Exception:
-            self.min_dpi = 150.0
+            self.min_dpi = 0.0
         try:
             self.max_dpi_factor = max(1.0, float(max_dpi_factor))
         except Exception:
@@ -262,7 +278,7 @@ class Policy:
         g = lambda k, d: getattr(cfg, k, d) if cfg is not None else d  # noqa: E731
         return cls(g("page_image_format", "auto"),
                    g("page_image_jpeg_quality", 95),
-                   g("page_image_min_dpi", 150.0),
+                   g("page_image_min_dpi", 0.0),
                    g("page_image_max_dpi_factor", 3.0))
 
     def as_dict(self) -> dict:
@@ -274,7 +290,7 @@ class Policy:
     def from_dict(cls, d):
         d = d or {}
         return cls(d.get("fmt", "auto"), d.get("jpeg_quality", 95),
-                   d.get("min_dpi", 150.0), d.get("max_dpi_factor", 3.0))
+                   d.get("min_dpi", 0.0), d.get("max_dpi_factor", 3.0))
 
     def encode(self, img):
         """``(mime, bytes)`` under this policy."""

codai/ocr/workers/raster_worker.py

@@ -37,11 +37,28 @@ guards live in page_image.extract_embedded; `open` carries the policy.
     -> {"cmd": "shutdown"}
     <- {"ok": true}                         then exit
 
-Raw RGB rather than PNG for a RENDERED page: the parent rebuilds it with
-Image.frombytes, and a page about to be read by an OCR engine must not pass
-through a lossy codec, while PNG would cost more CPU than the pipe it saves. A
-page that was never rendered is a different matter -- its bytes are already a
-JPEG, and re-rendering them to ship them raw would be work in both directions.
+A rendered page is encoded HERE, losslessly, rather than shipped as raw RGB.
+Measured on an A4 page at 250 DPI (6.04 Mpix), which is surya's own default:
+
+    raw RGB over the pipe            18.1 MB   and the parent then spent
+                                               109.7 ms encoding it anyway
+    PNG level 1 encoded here          0.72 MB  104.7 ms, in THIS process
+    the parent's lazy Image.open                 0.12 ms
+
+So the encode does not get added, it MOVES: off the single parent process and
+onto this pool, which is parallel, while the pipe carries 25x less and the
+parent holds 0.72 MB per page in flight instead of 18.1 MB. That last number is
+the one that matters -- a GPU-less orchestrator rasterising at 48 pages in
+flight is the difference between 35 MB and 870 MB of page bitmaps.
+
+The parent opens it lazily and passes the bytes onward, so the page is decoded
+only if something actually wants pixels (ink measurement, a detector) and the
+engine hop reuses these exact bytes instead of encoding a second time.
+
+PNG, not JPEG, unless the operator asked for JPEG: a page about to be read by
+an OCR engine must not pass through a lossy codec by default. Raw RGB remains
+the fallback when no policy was sent, which keeps the protocol readable by an
+older parent.
 
 Errors answer {"ok": false, "error": str, "status": int} and leave the process
 usable -- one undecodable document must not cost the pool a worker.
@@ -187,9 +204,22 @@ def main() -> int:
                         continue
                     bitmap = page.render(scale=doc.scale)
                     img = bitmap.to_pil().convert("RGB")
-                    raw = img.tobytes()
-                    metas.append({"w": img.width, "h": img.height,
-                                  "nbytes": len(raw)})
+                    # Encode here, under the policy, so the parent reuses these
+                    # bytes rather than encoding its own copy of the same page.
+                    enc = None
+                    if doc.policy is not None:
+                        try:
+                            enc = doc.policy.encode(img)
+                        except Exception:
+                            enc = None          # a render is worth more than a codec
+                    if enc is not None:
+                        mime, raw = enc
+                        metas.append({"w": img.width, "h": img.height,
+                                      "nbytes": len(raw), "mime": mime})
+                    else:
+                        raw = img.tobytes()
+                        metas.append({"w": img.width, "h": img.height,
+                                      "nbytes": len(raw)})
                     blobs.append(raw)
                     # Drop each page before rendering the next: this process is
                     # as bound by residency as its parent.