every rented pod was capped at 32 pages by its thread pool (v0.4.43)

A pod has no config.json -- its settings arrive as env vars -- and the
thread pools are built before the config manager has read them. So on
every pod `_configured_concurrency()` returned 0 and the default pool
took its floor of 32 workers. Every page on a pod runs in that pool.

The pod said so in its own log, every boot:

    [threads] default pool 32 workers, pod pool 64 (FLOORS -- the config
              was not readable from here yet ...)
    [vllm]   Maximum concurrency for 32,768 tokens per request: 72.20x
    [ocr]    engine 'surya': 8 instance(s) ready, 64 lane(s) each

Measured 2026-10-19: 64 pages dispatched per pod, `Running: 32` on every
card, GPU KV cache at 4-10%. Raising the orchestrator's dispatch width
could never help; the cap was inside the pod.

The sizing now falls back to CODERAI_OCR_MAX_CONCURRENCY, the env var
the orchestrator already sends every pod, so a pod sizes its pools the
way an install with a config file always did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.42"
+__version__ = "0.4.43"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/thread_pool.py

@@ -168,6 +168,18 @@ def _configured_concurrency() -> int:
                 want = max(want, int(getattr(ocr, "max_concurrency", 0) or 0))
         except Exception:
             continue
+    if want <= 0:
+        # A RENTED POD HAS NO CONFIG FILE. Its settings arrive as environment
+        # variables, and the pools are built before the config manager has
+        # read them -- so on every pod this returned 0 and the default pool
+        # took its floor of 32. Every page on a pod runs in that pool, which
+        # capped each pod at 32 pages at once while surya was ready for 512,
+        # vLLM reported room for 72x, and GPU KV cache sat at 4-10%. Measured
+        # 2026-10-19: 64 pages dispatched per pod, `Running: 32` on every card.
+        try:
+            want = int(os.environ.get("CODERAI_OCR_MAX_CONCURRENCY", "") or 0)
+        except ValueError:
+            want = 0
     return want
 
 

tests/test_thread_pool_sized_from_env.py

+"""A rented pod sizes its thread pools from its ENVIRONMENT.
+
+A pod has no config.json; its settings arrive as env vars, and the thread
+pools are built before the config manager has read them. So on every pod
+`_configured_concurrency()` returned 0 and the default pool took its floor of
+32 -- and every page on a pod runs in that pool. Measured 2026-10-19: the
+orchestrator dispatched 64 pages per pod, every card reported `Running: 32`,
+GPU KV cache sat at 4-10%, surya was ready for 512 and vLLM for 72x.
+"""
+import pytest
+
+from codai.api import thread_pool as tp
+
+
+def test_the_env_sizes_the_pools_when_no_config_is_readable(monkeypatch):
+    monkeypatch.setenv("CODERAI_OCR_MAX_CONCURRENCY", "4096")
+    monkeypatch.setattr(tp, "task_budget", lambda: 0)
+    assert tp._configured_concurrency() >= 4096
+    general, pod = tp.pool_sizes()
+    assert general > 32, "the floor is what capped every pod at 32 pages"
+
+
+def test_no_env_and_no_config_still_takes_the_floor(monkeypatch):
+    monkeypatch.delenv("CODERAI_OCR_MAX_CONCURRENCY", raising=False)
+    import importlib
+    for mod in ("codai.admin.routes", "codai.config"):
+        try:
+            m = importlib.import_module(mod)
+            monkeypatch.setattr(m, "config_manager", None, raising=False)
+        except Exception:
+            pass
+    assert tp._configured_concurrency() == 0
+    assert tp.pool_sizes()[0] == tp._MIN_DEFAULT
+
+
+def test_a_malformed_env_does_not_break_sizing(monkeypatch):
+    monkeypatch.setenv("CODERAI_OCR_MAX_CONCURRENCY", "lots")
+    import importlib
+    for mod in ("codai.admin.routes", "codai.config"):
+        try:
+            m = importlib.import_module(mod)
+            monkeypatch.setattr(m, "config_manager", None, raising=False)
+        except Exception:
+            pass
+    assert tp._configured_concurrency() == 0