the pod pool was a silent ceiling on fleet size (v0.4.32)

A page dispatched to a pod holds a pod-pool thread for its whole life, so the
pool's width decides how many pods an orchestrator can actually feed:

    pods_feedable = pod_pool_width / max_inflight_per_pod

With _MAX_POD at 2048 and a 64-page-per-pod ceiling, that was 32 pods —
whatever max_pods said. An operator who set max_pods=128 got 96 pods' worth of
nothing, and no line anywhere explained it: not ocr.max_concurrency (4096 here,
so it looked generous), not max_pods, not the cgroup (pids.max 16384 with 79 in
use on the box this was measured on). A silent fleet ceiling is a worse failure
than a slow one, because every symptom points at the pods.

Raised to 8192, which feeds 128 pods at 64 pages each. Past that an async
client really is the right answer rather than more thread stacks.

Nothing that protects the box changes. `pool_sizes()` still clamps against
`task_budget()` — the cgroup's pids.max minus what is running minus headroom —
still keeps _TASK_HEADROOM in reserve so a busy install can still be diagnosed,
and still gives the pod pool away before the general pool, because an install
that cannot answer /healthz cannot be fixed. Tests pin all three.

One existing test asserted `pod == 2048` for a configured width of 2048. That
was only true because _MAX_POD clamped there; the property it meant to protect
is that the configured width reaches both pools, which is `want * 2`. Restated
that way, so the next raise does not break it again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.31"
+__version__ = "0.4.32"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/thread_pool.py

@@ -63,11 +63,28 @@ _MIN_POD = 64
 #: Ceilings. A blocked thread is cheap but not free, and a runaway config
 #: should not be able to ask for ten thousand stacks.
 _MAX_DEFAULT = 256
-#: Raised from 512 once the sizing above started working: a run that wants
-#: 100 pages/s at ~20 s per page needs ~2000 forwards outstanding, and every
-#: one of them is a thread blocked on a socket. Beyond this an async client is
-#: the right answer, not more stacks.
-_MAX_POD = 2048
+#: Raised from 512, then from 2048, as the sizing below proved able to defend
+#: itself: a run that wants 100 pages/s at ~20 s per page needs ~2000 forwards
+#: outstanding, and every one of them is a thread blocked on a socket.
+#:
+#: 2048 was a silent FLEET ceiling, which is a worse failure than a slow one. A
+#: page on the pod path holds a pod-pool thread for its whole life, so this
+#: constant -- not ocr.max_concurrency, not max_pods, not the cgroup -- decided
+#: how many pods an orchestrator could feed: 2048 threads / 64 pages per pod =
+#: 32 pods, whatever max_pods said. An operator raising max_pods to 128 got 96
+#: pods' worth of nothing, and no line anywhere said why.
+#:
+#: Nothing here removes a real limit. `pool_sizes()` still clamps against
+#: `task_budget()` -- the cgroup's pids.max minus what is already running minus
+#: headroom -- and still gives the pod pool away first so the general pool can
+#: keep answering /healthz. This only stops a CONSTANT from being the binding
+#: limit on a box that has the tasks to spare: measured here, pids.max 16384
+#: with 79 in use.
+#:
+#: Past this an async client really is the right answer rather than more
+#: stacks; 8192 covers 128 pods at 64 pages each, which is a fleet large enough
+#: that the next bottleneck is somewhere else entirely.
+_MAX_POD = 8192
 
 #: Width of the LEASE pool. Deliberately small and NOT derived from
 #: `max_concurrency`: a lease is a bounded wait (`POD_LEASE_ATTEMPT_S`) that

tests/test_ocr_pod_pool_width.py

@@ -73,10 +73,19 @@ def test_an_unreadable_config_is_reported_not_disguised(monkeypatch):
 
 
 def test_the_configured_width_reaches_both_pools(holder):
+    """The configured page concurrency must size BOTH pools.
+
+    The pod pool gets two threads per in-flight page -- room for every page to
+    be waiting on a pod and still have one for the forward -- so the assertion
+    is `want * 2`, not a literal. It used to read `pod == 2048` for want=2048,
+    which was only true because _MAX_POD happened to clamp there; raising that
+    ceiling for a 128-pod fleet broke the literal while the property it was
+    protecting held.
+    """
     holder(2048)
     general, pod = thread_pool.pool_sizes()
     assert general == thread_pool._MAX_DEFAULT      # 256
-    assert pod == 2048
+    assert pod == min(thread_pool._MAX_POD, 2048 * 2)
     assert pod > thread_pool._MIN_POD * 10
 
 

tests/test_pod_pool_fleet_ceiling.py

+"""The pod thread pool must not be a silent ceiling on fleet size.
+
+A page dispatched to a pod holds a pod-pool thread for its whole life, so the
+pool's width -- not ocr.max_concurrency, not max_pods, not the cgroup -- decides
+how many pods an orchestrator can actually feed:
+
+    pods_feedable = pod_pool_width / max_inflight_per_pod
+
+With the old _MAX_POD of 2048 and a 64-page-per-pod ceiling that was 32 pods,
+whatever max_pods said. An operator who set max_pods=128 got 96 pods' worth of
+nothing and no line anywhere explained it. That is a worse failure than a slow
+one, because it looks like the pods are the problem.
+
+What must NOT change: the pool is still bounded by the task budget, and the pod
+pool still gives way first so the general pool can keep answering /healthz. A
+ceiling that protects the box is right; a constant that caps the fleet on a box
+with 16,000 spare tasks is not.
+"""
+import pytest
+
+import codai.api.thread_pool as tp
+
+
+PER_POD = 64        # max_inflight_per_pod in the measured configuration
+
+
+def test_the_ceiling_can_feed_a_128_pod_fleet():
+    assert tp._MAX_POD >= 128 * PER_POD, (
+        "a %d-wide pod pool feeds only %d pods at %d pages each"
+        % (tp._MAX_POD, tp._MAX_POD // PER_POD, PER_POD))
+
+
+def test_the_old_ceiling_would_not_have():
+    """Guards the test: 2048 really was a 32-pod limit."""
+    assert 2048 // PER_POD == 32
+
+
+def test_pod_pool_tracks_configured_concurrency(monkeypatch):
+    """Two threads per in-flight page, so a page can wait on a pod and still
+    have one for the forward."""
+    monkeypatch.setattr(tp, "_configured_concurrency", lambda: 4096)
+    monkeypatch.setattr(tp, "task_budget", lambda: 0)      # unbounded
+    _general, pod = tp.pool_sizes()
+    assert pod == min(tp._MAX_POD, 8192)
+
+
+@pytest.mark.parametrize("want,expect_pods", [(1024, 32), (2048, 64), (4096, 128)])
+def test_concurrency_translates_to_fleet_size(monkeypatch, want, expect_pods):
+    monkeypatch.setattr(tp, "_configured_concurrency", lambda: want)
+    monkeypatch.setattr(tp, "task_budget", lambda: 0)
+    _general, pod = tp.pool_sizes()
+    assert pod // PER_POD >= expect_pods
+
+
+# --- the protections that must survive the raise ---
+
+def test_the_task_budget_still_binds(monkeypatch):
+    """A small cgroup must still win over the raised constant."""
+    monkeypatch.setattr(tp, "_configured_concurrency", lambda: 4096)
+    monkeypatch.setattr(tp, "task_budget", lambda: 1024)
+    general, pod = tp.pool_sizes()
+    assert general + pod <= 1024 - tp._TASK_HEADROOM + tp._MIN_POD, \
+        "pools must fit the budget (floors aside)"
+    assert pod < tp._MAX_POD, "the constant must not override a tight budget"
+
+
+def test_the_pod_pool_gives_way_first(monkeypatch):
+    """The general pool answers /healthz and the admin pages; an install that
+    cannot be looked at cannot be fixed."""
+    monkeypatch.setattr(tp, "_configured_concurrency", lambda: 4096)
+    monkeypatch.setattr(tp, "task_budget", lambda: 2048)
+    general, _pod = tp.pool_sizes()
+    assert general >= tp._MIN_DEFAULT
+
+
+def test_both_floors_survive_a_tiny_budget(monkeypatch):
+    monkeypatch.setattr(tp, "_configured_concurrency", lambda: 4096)
+    monkeypatch.setattr(tp, "task_budget", lambda: 100)
+    general, pod = tp.pool_sizes()
+    assert (general, pod) == (tp._MIN_DEFAULT, tp._MIN_POD)
+
+
+def test_headroom_is_kept(monkeypatch):
+    """A pool that spends the last task takes diagnosis away with it."""
+    monkeypatch.setattr(tp, "_configured_concurrency", lambda: 100000)
+    monkeypatch.setattr(tp, "task_budget", lambda: 4096)
+    general, pod = tp.pool_sizes()
+    assert general + pod <= 4096 - tp._TASK_HEADROOM