• Stefy Lanza (nextime / spora )'s avatar
    runpod: let a pod serve the VLM OCR engines it was configured for (v0.2.29) · 8ae02a35
    Stefy Lanza (nextime / spora ) authored
    A pod asked for surya or olmOCR was downgraded to docTR UNCONDITIONALLY. The reasoning
    was sound — both are VLMs needing a model server, locally they ride coderai's own vLLM
    engine, and a pod image will not get one (that venv alone is ~10.5 GB against a ~12 GB
    ceiling, and it would be a second torch beside the one docTR and Surya already need) —
    but the conclusion was too broad. In their SERVED modes these engines only speak HTTP,
    and PIL and requests are already in the image. So a pod can run them whenever the
    operator has pointed them at a reachable endpoint, including at a vLLM pod, which is the
    composition to prefer over a fat image.
    
    - _ocr_external_server(engine) decides it: olmocr_serve="server" + olmocr_server_url, or
      surya_serve="llamacpp" + surya_server_url. The local-only modes ("vllm", "model") still
      downgrade, and the log now names the setting to fix instead of only the symptom.
    - The mode, URL and model ids are seeded to the pod. config.py gained the env overrides
      for surya_serve/surya_server_url and fallback_engines; only olmOCR's existed.
    - fallback_engines is pinned to the engines the image actually carries: "auto" would hand
      a failed page to an engine with no server on that pod, which the fallback added in
      0.2.28 made newly reachable.
    - Engine selection is alias-aware (`alias or served or path`, the convention used
      throughout the manager). An entry aliased "olmocr" over an HF repo path was read as an
      unknown engine and silently downgraded. Aliases are how one model carries several
      configs and how several models share an endpoint, so this path has to honour them.
    - The licence decision now survives a downgrade: `wanted` was reassigned to "doctr"
      before the `wanted == "surya"` check, making that branch's own documented intent
      ("asking for surya by name IS the decision") unreachable.
    
    olmOCR needs no pip dependency anywhere — it imports only PIL and requests — which is
    why nothing flagged its absence from the profiles; ocr.txt now records that and why a
    pod has no vLLM.
    
    19 new tests; 278 pass across the OCR, front-gate, vLLM-share, cluster, remote-execution,
    placement and routing suites.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    8ae02a35
__init__.py 4.21 KB