A 5,000-document run leased 48 warm pods and then read 278 pages ON THE
ORCHESTRATOR -- a box with no GPU -- while every one of those cards sat
idle. Each page logged "the pod-side pipeline could not run (can't start
new thread)". Zero documents were read on the pods.

The container's task budget was 2048 (podman's --pids-limit default) and
the pod pool had sized itself 2048 from ocr.max_concurrency: 4096. It
reached 1992 threads; the next one could not exist. Nothing had looked for
a limit, so nothing reported one.

Three separable mistakes, all three fixed:

  * `thread_pool.task_budget()` reads what the process is actually allowed
    -- cgroup v2 pids.max walked up to the root (a limit on any ancestor
    binds as hard as one on the leaf), minus pids.current because the other
    processes in the cgroup already spend it, and RLIMIT_NPROC -- and
    `pool_sizes()` fits inside it, keeping 256 tasks in reserve so a
    rasteriser worker, an isolated-venv engine or a `podman exec` to look at
    the box can still fork. The general pool is protected ahead of the pod
    pool: an install that cannot answer /healthz cannot be diagnosed. On the
    numbers above it now sizes 256 + 1506 instead of asking for 2304.

  * A lease took a POD-POOL thread per in-flight page, so the page width an
    install wants decided how many thread stacks it needs. A lease is a
    bounded wait (POD_LEASE_ATTEMPT_S), so a separate 256-thread lease pool
    serves thousands of pages by cycling. It also stops a cold start filling
    every thread with leases waiting for pods, leaving none to carry the
    pages those pods were for.

  * "Cannot create a thread" was treated as "the pods cannot run the
    pipeline", and the fallback for that is the local pipeline -- which needs
    MORE of this box than the path that just failed for want of it. Local
    exhaustion now answers 503 naming both ways out (the task budget, or
    ocr.max_concurrency) and leaves the pages for the cards.

AND THE TEARDOWN, from the same run: the client's own release sweep answered
500 five times in three seconds, because terminating a pod that was already
gone raised. By the time a fleet teardown reaches its last pod some are
always gone -- reaped for idle, shed by a scale-down, or released by a retry
of the same call -- so a teardown that was working looked like one that had
failed. Already gone is the goal, not a failure: it answers 200 with
`already_gone: true`, while a real failure still raises so a pod left
billing is never reported as stopped.

Also: a 400 on an upload path now logs why. A malformed or truncated
multipart body is rejected before any handler runs, so 19 of 137 batches
failed in that run with no record anywhere but the access line -- not the
filename, not the size, not even whether it was the body or a field.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM