-
Stefy Lanza (nextime / spora ) authored
Measured on A4 at 200 DPI (3.4-3.9 Mpix), best of three: pdfium page.render() 21-46 ms PNG encode, PIL default (level 6) 86-353 ms 1846 KB PNG encode, level 1 (lossless) 123 ms 2117 KB JPEG q95 8.4 ms 596 KB the page's own embedded JPEG 0.1 ms 147 KB So most of what is called rasterisation is not rasterisation, and on the surya path it was paid TWICE per page: once in subprocess_engine to hand the page to the isolated-venv worker, once inside surya's own client to build the data URI (surya/inference/backends/openai_client.py:32, with "image/png" hard-coded into the URI it builds). Against roughly 5 s on the GPU per page -- 12 pages/min/pod at Running: 0-3 reqs, measured on two live pods -- that is CPU the pods need in order to hold 48 pages in flight. Two ways out, both shipped, both default: * A page that already IS an image is sent as the PDF holds it. A scan is usually one full-page DCTDecode object, and extract_embedded() takes it in 0.1 ms with no render, no encode and no resample. Every guard in there is a way those bytes would have been a WRONG page rather than a slower one: one image object and nothing else (a stamp or a text layer must not be dropped), DCTDecode alone (Flate is raw pixels, CCITT and JBIG2 are bitonal, JPX is JPEG 2000), grey or RGB only (a CMYK JPEG out of a PDF decodes inverted), /Rotate 0 and an axis-aligned positive matrix (otherwise the bytes are a sideways page, which reads as gibberish rather than as an error), 98 % coverage anchored at the origin, and an effective DPI within [min_dpi, max_dpi_factor x target]. * When a page must be encoded, it is encoded once, at zlib level 1. Lossless, 2.6x faster, and the bytes are carried forward so the request reuses them. The carrier is an attribute on the PIL image, recorded with the size it was true for. Any PIL transform returns a NEW object and so drops it, which is the design: a missed passthrough costs milliseconds, a wrong one shows the model a plausible page at the wrong resolution. chandra and olmOCR therefore scale BEFORE converting, since convert() was quietly invalidating every page. End to end, one page from PDF bytes to data URI: a page that must be rendered 610 ms -> 234 ms (jpeg: 68 ms) a scanned page 610 ms -> 6.9 ms, 1895 KB -> 147 KB New: ocr.page_image_format (auto | png | jpeg), page_image_jpeg_quality, page_image_min_dpi, page_image_max_dpi_factor, with the admin API and the settings form. `png` always re-encodes, which is what a controlled comparison needs; `jpeg` is lossy and stays opt-in, because whether it costs accuracy is a question about a corpus and not about a codec. The rasteriser worker loads page_image by path: it may not import codai.ocr (377 MB via litellm), and the guards must be the SAME ones the parent applies or a pool outage would change what the model is shown. ocr_worker.py may not import even that -- it runs in the engine's own venv -- so its copy of the carrier convention is tested against page_image's to keep the two from drifting. 42 new tests; 3306 passed with the 13 pre-existing failures (broker, studio, manual client) untouched. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
5d148965