a dead socket is not a pod refusing the pipeline (v0.4.6)
A pooled keep-alive connection that a pod closes while idle fails on its next use: pod OCR at https://.../v1/ocr is unreachable: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) _post_page_to_pod turned every transport failure into OcrError(503), 503 is in _POD_CANNOT_RUN_PIPELINE, so one dead TCP connection was read as "these pods cannot run the OCR pipeline" and set the whole pool aside for five minutes. It happened twelve times in four hours on a production run: the GPU-less orchestrator rasterised every page of every document for those windows and climbed to 40 GB at ~0.8 GB/min, while sixteen rented A40s sat idle and willing. An ANSWER may condemn an image; a socket may not. So: * PodUnreachable, a distinct error for "nothing came back", which never sets the pool aside. A pod that ANSWERS 404/501/503 still does. * One page is retried on a fresh connection (nothing was sent, so there is no side effect) and then on another pod — but a pod is dropped from the pool only after it fails to answer /healthz. Writing a pod off because an idle connection was reused has cost seven healthy cards before. * A timeout is NOT retried: the pod may be ninety seconds into a 130-second read, and sending the page again doubles the work on that card. * An active cooldown is no longer extended. It used to be, silently: every further refusal pushed the expiry out while only the first one printed, so a five-minute window could last twenty with nothing in the log. Two more ways to end up local, both of which were SILENT: * The gate compared the request's engine against the three VLM engine names. /v1/ocr has always also accepted the MODEL a caller knows in place of the engine driving it — resolve_engine maps the Surya-2 checkpoint to 'surya' and the catalogue advertises those ids as aliases — so a caller naming the model got no pod-side pipeline and no message. The switch has to be read the way a READ reads it, for the same reason it has to be read the way a LAUNCH reads it. * Serve mode not 'model', an empty model id, an engine with no pod pipeline, a standing refusal: each returned "" or None without a word. The log of the run that prompted this holds 8,278 successful pod reads and nothing at all explaining the thousands of documents in between. Every one of them now says which way the decision went, rate limited rather than said once, because a cooldown is minutes and a run is hours. And, since reading a page here means rasterising here, ocr.pod_pipeline: prefer (default) wait up to ocr.pod_pipeline_wait_s for a pod, then read it here — a pool with nothing free is usually one still booting. A timed-out wait is not paid again for five minutes, so the ceiling cannot become a per-page tax. auto read it here at once (0.4.1-0.4.5 behaviour). require never rasterise on this box: 503, saying why. Tests: 32 new (tests/test_ocr_pod_pipeline_policy.py) covering the alias, all four silent routes, the three policies and the transport/answer line. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
Showing
This diff is collapsed.
This diff is collapsed.
Please
register
or
sign in
to comment