The admin RunPod page showed:

    error: Unexpected token '<', "<!doctype "... is not valid JSON

That '<' is nginx's HTML error page. The page was not failing to find data; its
fetch never got an answer, the proxy gave up, and JSON.parse was handed HTML.
Measured from inside the host, mid-run, with 64 pages in flight:

    /healthz            200 in 0.002s     no thread needed
    /v1/models          200 in 0.006s     no thread needed
    /metrics            200 in 0.18s      no thread needed
    /v1/runpod/status   no answer in 30s  needs a thread

`asyncio.to_thread` hands work to the loop's default executor, and a loop that
was never told otherwise builds one with min(32, cpu_count + 4) — 20 threads on
a 16-core host. Nothing in this codebase ever called set_default_executor. That
number is for incidental background work measured in milliseconds; what we put
on it is `pool.acquire()` waiting out a pod boot (MINUTES) and a per-page OCR
forward (30-300s each), `ocr.max_concurrency` of them at once. The log says it
plainly: "engine 'surya': 8 instance(s) ready, 8 lane(s) each — 64 pages at
once". 64 blocking calls, 20 threads. Everything else waits behind a page.

Nothing was broken. The server could not get a thread to answer on.

codai/api/thread_pool.py now sizes two pools from the concurrency the install
is CONFIGURED for rather than from its CPU count — these threads wait on other
machines, so what matters is how many remote calls may be outstanding, not how
many cores can run Python:

  * the DEFAULT executor, floor 32, for everything to_thread is used for today;
  * a separate POD executor, floor 64, for the calls that block for minutes, so
    that however many pages are in flight a control-plane request still finds a
    free thread.

Both entry points install it (the front blocks on pool.acquire for whole pod
boots, so it needs this at least as much as the engine). The pod pool copies
the context the way to_thread does — the per-page options and the token trace
are ContextVars, and run_in_executor does not copy by itself; a page losing its
budget because it changed pools would be a nasty way to learn that.

/v1/runpod/spend also took its snapshot ON the loop: pods_status() takes the
pool locks and the ledger reads a file, so a lock held by a provisioning thread
stalled every request rather than one. It runs in a thread now.

A thread waiting on a socket costs a stack and nothing else. The asymmetry is
the point: too small costs availability, too large costs memory we have.

Also fixes a test of my own from an hour ago that asserted "tomorrow 02:00" is
within 24 hours and went red at 00:22. The test was wrong about the clock, not
the code about the booking.

15 tests, including the live failure reproduced: fill the pod pool and check
the control plane still answers. Suite: 3166 passed, the same 13 known-stale
failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM