-
Stefy Lanza (nextime / spora ) authored
A page holds its thread for the whole read, a minute or more. _Pool.recognize used asyncio.to_thread, which puts that wait on the default executor: capped at 256 and shared with model lookups and the admin pages. On a rented A40 the engine had 512 lanes and vLLM sat at 9% KV cache, so the thread pool, not the GPU, set the pod's ceiling. Past 256 the model lookup each page's chat request needs (off the loop since v0.4.52) would also have queued behind the page waits. Page reads now use the pod pool. It is sized from ocr.max_concurrency (up to 16384, within the task budget), and on a pod nothing else uses it. Co-Authored-By:
Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
80cb8374