-
Stefy Lanza (nextime / spora ) authored
New engine `olmocr` — allenai/olmOCR-2 (Apache-2.0), a document VLM for the pages the detector-based engines read badly: old scans, maths, multi-column, dense tables. Equations as LaTeX, tables as HTML, reading order sorted; no bounding boxes, so paddle/surya stay the choice when something has to point at the page. It is served three ways (`ocr.olmocr_serve`): "model" (default) routes through coderai's OWN model manager, so a GGUF+mmproj or HF entry in models.json gets VRAM accounting, eviction and the thermal governor — and it works without CUDA; "vllm" gives it its own vLLM instance; "server" attaches to an OpenAI endpoint already running it. olmOCR's trained no-anchoring-v4 prompt is sent verbatim, the page is rendered at 1288 px, the YAML front matter lands in the new `OcrPage.meta` (omitted for engines that report none), and a page the model calls rotated is turned and asked once more. Four fixes the production Surya failures exposed — 169 consecutive failed boots between 08:00 and 09:35 today, every one against the same 11.34 of 23.56 GiB free against a 14.13 GiB demand, each costing the caller a 35 s hang: - The pool evicted for an engine AFTER loading it. For an engine that boots a server claiming a fixed share of the card, the load is what fails. Engines now declare `prelaunch_vram_gb()` and the pool frees that much first. - That reservation is the WHOLE POOL's, asked for once. Per instance it would have asked the manager to free 24 x 8.4 GB on this install's 24-instance surya pool, and an engine that reserved up front was then evicted for a second time, after its server had already taken the memory. - A failed build was retried on every single request. `ocr.build_retry_cooldown_s` (60 s) now fails fast with the reason the boot gave. - `ocr.vlm_gpu_memory_utilization` (default 0.35): the OCR VLM's own share of the card instead of the one `vllm.gpu_memory_utilization` asks for on behalf of an LLM. 0.35 of 24 GB fits surya-2 / olmOCR-2 and leaves the resident engines alone; 0.6 was more than the card had left. Per-instance footprints corrected to match: a surya worker in a served mode speaks only HTTP (0.05 GB, not 1 GB x 24), and an olmocr instance is one in-flight page, holding nothing of its own. Eviction in the other direction is unchanged and tested: the model manager's external releaser tears down the pools AND stops both VLM engines' vLLM services, and the pools rebuild on the next request. 18 new tests (no model, no GPU, no network). Docs: docs/ocr.md. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B7EnDW6RSd2nhEPZMzMkqi
75200c0b
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| static | ||
| templates | ||
| __init__.py | ||
| auth.py | ||
| download_worker.py | ||
| routes.py |