A batch of documents gave one flawless 19-second burst — 22 documents, correct
page counts, clean accents — and then no completions at all for nine minutes,
while the rented A40 sat at 2% KV-cache usage with `Waiting: 0`. The GPU was
never the bottleneck. Two local mistakes compounded:

`surya_instances: 64` reads like GPU concurrency and is not: surya, paddle and
docTR spawn ONE OS PROCESS per instance, in their own venv. 64 of them meant 37
GB of RSS, swap exhausted and load 25 on a 16-core host that also runs some
thirty other containers. Nine hours after the batch ended, 66 of those workers
were still resident with 19 seconds of CPU time each.

Then it fed itself. The workers' /health probe (5s, against coderai's OWN
gateway) began timing out under the load the workers were creating, and a
timed-out probe counted as "the endpoint is gone" — so each worker was killed
and respawned, which added load, which timed out more probes.

  - the process-per-instance engines are capped at one worker per core and
    never more than half of free RAM, with a log line naming the figure and the
    reason. CODERAI_OCR_INSTANCE_CEILING overrides it for a measured host. The
    in-process engines (chandra, olmOCR) are untouched: an instance there is a
    concurrency slot, not a process;
  - a health probe now means "gone" only on a definite 404 "unknown OCR engine
    lease", which is the case re-resolving was written for: a front that
    restarted. A timeout means busy and the worker stays up, because respawning
    cannot fix a front that is down — the replacement asks the same front for a
    lease. A 503 means the pool has nothing rented this second, which the lease
    does not care about.

GPU-side concurrency remains runpod.scale_up_inflight_per_pod and max_pods; the
local worker pool is a hard ceiling on how many pods surya can keep fed, which
is the standing argument for running it in-process.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW