• Stefy Lanza (nextime / spora )'s avatar
    ocr: the pod image chooses where the pipeline runs (v0.3.70) · 5813524c
    Stefy Lanza (nextime / spora ) authored
    An OCR VLM needs a model server AND the surya pipeline that drives it. Which
    machine runs the second one is now chosen by the pod IMAGE, and by nothing else:
    
      coderai-vllm      the pod serves the model; the ORCHESTRATOR rasterises every
                        page and sends a ~1.06 MB base64 PNG per A4 page at 250 DPI
                        -- and again for every resumption round. Unchanged.
      coderai-vllm-ocr  the pod carries surya's client venv and pypdfium2, so it is
                        sent a ONE-PAGE PDF (~474 KB, 1.7 ms to split here against
                        ~105 ms to rasterise) and reads it on the card with its own
                        vLLM over loopback.
    
    A separate `pod_ocr_pipeline` flag was built first and removed. Two switches
    that can disagree is the invisible mismatch that cost three rounds of blaming a
    pod image for orchestrator bugs; choosing the image IS choosing the mode, it
    already lives in models.json and on the model page, and an image that cannot run
    a pipeline can no longer be asked to.
    
    PAGE-LEVEL FAN-OUT IS UNCHANGED. One page is one request in both modes --
    `placement.choose` picks an upstream per page -- so every pod in the pool shares
    the work and none owns a document. Sending a whole PDF to one pod would turn an
    8-way parallel read into a serial one, which is why the split is per page.
    
    THE TUNING TRAVELS, not just the wiring. surya_dpi, the page budget, the
    continuation count, the repeat-detector thresholds, langs, the page window, the
    resident cap. Every one was set in response to a measured failure, and a pod
    reading pages at surya's own defaults would be indistinguishable from the
    "pre-fix pod image" that was wrongly blamed three times. A test cross-checks
    every CODERAI_OCR_* the plan sends against what config.py actually reads: a
    variable sent and never read is a setting that silently does not apply, which is
    the exact shape of two bugs found earlier today.
    
    FALLBACK, and what it refuses to hide:
      * no pod rented -> the WHOLE document is read locally, not half each way (a
        document read by two pipelines is the hardest kind of result to explain);
      * a pod answers 500 -> raised, not quietly re-read here; a broken fleet must
        stay visible;
      * the image lacks the pipeline -> the orchestrator does what it always did.
    
    The image, built and verified (14306 MB, /healthz OK):
      venvs/surya  1.6 GB  torch 2.14.1+cpu cuda=None + ultralytics 8.4.177
      venvs/vllm    11 GB
      main venv:   pypdfium2, and no torch at all (light core intact)
    A CUDA torch in the surya venv would add ~4.2 GB AND put surya's small local
    line detector on the card serving the model; the venv check asserts
    torch.version.cuda is None so a resolve cannot quietly undo it.
    
    35 tests. Two flaws of my own that they caught: four passed alone and failed in
    the suite because they patched sys.modules while `from codai.models import
    placement` resolves the name as an attribute of the already-imported package --
    now patched on the real module, verified in both orders; and _plan_env used a
    hand-rolled stub that tested the stub, now built from a real
    RunpodModelConfig. Suite at the 25 pre-existing failures.
    
    NOT YET EXERCISED AGAINST A REAL POD: the orchestrator path has only run against
    fakes, and the image is local -- not pushed or signed.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    5813524c
Name
Last commit
Last update
..
schemas_builtin Loading commit data...
workers Loading commit data...
__init__.py Loading commit data...
base.py Loading commit data...
budget.py Loading commit data...
chandra.py Loading commit data...
detect.py Loading commit data...
doctr.py Loading commit data...
extract.py Loading commit data...
gateway.py Loading commit data...
manager.py Loading commit data...
olmocr.py Loading commit data...
paddle.py Loading commit data...
rasterize.py Loading commit data...
resume.py Loading commit data...
schemas.py Loading commit data...
served_model.py Loading commit data...
subprocess_engine.py Loading commit data...
surya.py Loading commit data...
venv_build.py Loading commit data...