• Stefy Lanza (nextime / spora )'s avatar
    ocr: olmOCR-2 as an alternative engine, and stop the Surya-2 crash loop (v0.2.24) · 75200c0b
    Stefy Lanza (nextime / spora ) authored
    New engine `olmocr` — allenai/olmOCR-2 (Apache-2.0), a document VLM for the pages
    the detector-based engines read badly: old scans, maths, multi-column, dense
    tables. Equations as LaTeX, tables as HTML, reading order sorted; no bounding
    boxes, so paddle/surya stay the choice when something has to point at the page.
    
    It is served three ways (`ocr.olmocr_serve`): "model" (default) routes through
    coderai's OWN model manager, so a GGUF+mmproj or HF entry in models.json gets VRAM
    accounting, eviction and the thermal governor — and it works without CUDA; "vllm"
    gives it its own vLLM instance; "server" attaches to an OpenAI endpoint already
    running it. olmOCR's trained no-anchoring-v4 prompt is sent verbatim, the page is
    rendered at 1288 px, the YAML front matter lands in the new `OcrPage.meta` (omitted
    for engines that report none), and a page the model calls rotated is turned and
    asked once more.
    
    Four fixes the production Surya failures exposed — 169 consecutive failed boots
    between 08:00 and 09:35 today, every one against the same 11.34 of 23.56 GiB free
    against a 14.13 GiB demand, each costing the caller a 35 s hang:
    
    - The pool evicted for an engine AFTER loading it. For an engine that boots a
      server claiming a fixed share of the card, the load is what fails. Engines now
      declare `prelaunch_vram_gb()` and the pool frees that much first.
    - That reservation is the WHOLE POOL's, asked for once. Per instance it would have
      asked the manager to free 24 x 8.4 GB on this install's 24-instance surya pool,
      and an engine that reserved up front was then evicted for a second time, after
      its server had already taken the memory.
    - A failed build was retried on every single request. `ocr.build_retry_cooldown_s`
      (60 s) now fails fast with the reason the boot gave.
    - `ocr.vlm_gpu_memory_utilization` (default 0.35): the OCR VLM's own share of the
      card instead of the one `vllm.gpu_memory_utilization` asks for on behalf of an
      LLM. 0.35 of 24 GB fits surya-2 / olmOCR-2 and leaves the resident engines alone;
      0.6 was more than the card had left.
    
    Per-instance footprints corrected to match: a surya worker in a served mode speaks
    only HTTP (0.05 GB, not 1 GB x 24), and an olmocr instance is one in-flight page,
    holding nothing of its own. Eviction in the other direction is unchanged and
    tested: the model manager's external releaser tears down the pools AND stops both
    VLM engines' vLLM services, and the pools rebuild on the next request.
    
    18 new tests (no model, no GPU, no network). Docs: docs/ocr.md.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01B7EnDW6RSd2nhEPZMzMkqi
    75200c0b
routes.py 238 KB