• Stefy Lanza (nextime / spora )'s avatar
    ocr: the pod image chooses where the pipeline runs (v0.3.70) · 5813524c
    Stefy Lanza (nextime / spora ) authored
    An OCR VLM needs a model server AND the surya pipeline that drives it. Which
    machine runs the second one is now chosen by the pod IMAGE, and by nothing else:
    
      coderai-vllm      the pod serves the model; the ORCHESTRATOR rasterises every
                        page and sends a ~1.06 MB base64 PNG per A4 page at 250 DPI
                        -- and again for every resumption round. Unchanged.
      coderai-vllm-ocr  the pod carries surya's client venv and pypdfium2, so it is
                        sent a ONE-PAGE PDF (~474 KB, 1.7 ms to split here against
                        ~105 ms to rasterise) and reads it on the card with its own
                        vLLM over loopback.
    
    A separate `pod_ocr_pipeline` flag was built first and removed. Two switches
    that can disagree is the invisible mismatch that cost three rounds of blaming a
    pod image for orchestrator bugs; choosing the image IS choosing the mode, it
    already lives in models.json and on the model page, and an image that cannot run
    a pipeline can no longer be asked to.
    
    PAGE-LEVEL FAN-OUT IS UNCHANGED. One page is one request in both modes --
    `placement.choose` picks an upstream per page -- so every pod in the pool shares
    the work and none owns a document. Sending a whole PDF to one pod would turn an
    8-way parallel read into a serial one, which is why the split is per page.
    
    THE TUNING TRAVELS, not just the wiring. surya_dpi, the page budget, the
    continuation count, the repeat-detector thresholds, langs, the page window, the
    resident cap. Every one was set in response to a measured failure, and a pod
    reading pages at surya's own defaults would be indistinguishable from the
    "pre-fix pod image" that was wrongly blamed three times. A test cross-checks
    every CODERAI_OCR_* the plan sends against what config.py actually reads: a
    variable sent and never read is a setting that silently does not apply, which is
    the exact shape of two bugs found earlier today.
    
    FALLBACK, and what it refuses to hide:
      * no pod rented -> the WHOLE document is read locally, not half each way (a
        document read by two pipelines is the hardest kind of result to explain);
      * a pod answers 500 -> raised, not quietly re-read here; a broken fleet must
        stay visible;
      * the image lacks the pipeline -> the orchestrator does what it always did.
    
    The image, built and verified (14306 MB, /healthz OK):
      venvs/surya  1.6 GB  torch 2.14.1+cpu cuda=None + ultralytics 8.4.177
      venvs/vllm    11 GB
      main venv:   pypdfium2, and no torch at all (light core intact)
    A CUDA torch in the surya venv would add ~4.2 GB AND put surya's small local
    line detector on the card serving the model; the venv check asserts
    torch.version.cuda is None so a resolve cannot quietly undo it.
    
    35 tests. Two flaws of my own that they caught: four passed alone and failed in
    the suite because they patched sys.modules while `from codai.models import
    placement` resolves the name as an attribute of the already-imported package --
    now patched on the real module, verified in both orders; and _plan_env used a
    hand-rolled stub that tested the stub, now built from a real
    RunpodModelConfig. Suite at the 25 pre-existing failures.
    
    NOT YET EXERCISED AGAINST A REAL POD: the orchestrator path has only run against
    fakes, and the image is local -- not pushed or signed.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    5813524c
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...