• Stefy Lanza (nextime / spora )'s avatar
    ocr: the pod image chooses where the pipeline runs (v0.3.70) · 5813524c
    Stefy Lanza (nextime / spora ) authored
    An OCR VLM needs a model server AND the surya pipeline that drives it. Which
    machine runs the second one is now chosen by the pod IMAGE, and by nothing else:
    
      coderai-vllm      the pod serves the model; the ORCHESTRATOR rasterises every
                        page and sends a ~1.06 MB base64 PNG per A4 page at 250 DPI
                        -- and again for every resumption round. Unchanged.
      coderai-vllm-ocr  the pod carries surya's client venv and pypdfium2, so it is
                        sent a ONE-PAGE PDF (~474 KB, 1.7 ms to split here against
                        ~105 ms to rasterise) and reads it on the card with its own
                        vLLM over loopback.
    
    A separate `pod_ocr_pipeline` flag was built first and removed. Two switches
    that can disagree is the invisible mismatch that cost three rounds of blaming a
    pod image for orchestrator bugs; choosing the image IS choosing the mode, it
    already lives in models.json and on the model page, and an image that cannot run
    a pipeline can no longer be asked to.
    
    PAGE-LEVEL FAN-OUT IS UNCHANGED. One page is one request in both modes --
    `placement.choose` picks an upstream per page -- so every pod in the pool shares
    the work and none owns a document. Sending a whole PDF to one pod would turn an
    8-way parallel read into a serial one, which is why the split is per page.
    
    THE TUNING TRAVELS, not just the wiring. surya_dpi, the page budget, the
    continuation count, the repeat-detector thresholds, langs, the page window, the
    resident cap. Every one was set in response to a measured failure, and a pod
    reading pages at surya's own defaults would be indistinguishable from the
    "pre-fix pod image" that was wrongly blamed three times. A test cross-checks
    every CODERAI_OCR_* the plan sends against what config.py actually reads: a
    variable sent and never read is a setting that silently does not apply, which is
    the exact shape of two bugs found earlier today.
    
    FALLBACK, and what it refuses to hide:
      * no pod rented -> the WHOLE document is read locally, not half each way (a
        document read by two pipelines is the hardest kind of result to explain);
      * a pod answers 500 -> raised, not quietly re-read here; a broken fleet must
        stay visible;
      * the image lacks the pipeline -> the orchestrator does what it always did.
    
    The image, built and verified (14306 MB, /healthz OK):
      venvs/surya  1.6 GB  torch 2.14.1+cpu cuda=None + ultralytics 8.4.177
      venvs/vllm    11 GB
      main venv:   pypdfium2, and no torch at all (light core intact)
    A CUDA torch in the surya venv would add ~4.2 GB AND put surya's small local
    line detector on the card serving the model; the venv check asserts
    torch.version.cuda is None so a resolve cannot quietly undo it.
    
    35 tests. Two flaws of my own that they caught: four passed alone and failed in
    the suite because they patched sys.modules while `from codai.models import
    placement` resolves the name as an attribute of the already-imported package --
    now patched on the real module, verified in both orders; and _plan_env used a
    hand-rolled stub that tested the stub, now built from a real
    RunpodModelConfig. Suite at the 25 pre-existing failures.
    
    NOT YET EXERCISED AGAINST A REAL POD: the orchestrator path has only run against
    fakes, and the image is local -- not pushed or signed.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    5813524c
base.py 31.8 KB