Read from the pod's own drained log, ten minutes of batch OCR on 2026-10-13:

  Running: 15-19 reqs, Waiting: 0 reqs, GPU KV cache usage: 5-7%
  Avg generation throughput: ~1,900-2,100 tokens/s, every minute
  524 x "POST /v1/chat/completions" 200 OK, no aborts
  2,644 generated tokens PER REQUEST (second pod: 4,010)
  MM cache hit rate: 25-27%

Three things in those numbers, and only the third is a defect.

The GPU was never the bottleneck: 5-7% of the KV cache and `Waiting: 0` for
ten solid minutes. So a card to itself cannot be the fix, and the co-loaded
pyannote cost nothing measurable — the limit was never VRAM or batch slots.

Oversubscription converts throughput into latency, not into stalling. 64
pages into max_pods x scale_up_inflight_per_pod = 32 slots queues correctly,
but every page's latency rises by the queue, and a client counting finished
DOCUMENTS sees nothing until a document's LAST page lands: spread 64 pages
over sixteen documents and all sixteen advance while none completes. The
pool now says so at build time instead of leaving it to be inferred.

The defect: 2,644 generated tokens for a page whose text is ~400. surya
re-runs a page whose generation repeated itself (`_should_retry`) up to three
more times, each at a rising temperature and each able to reach its token
cap — four full generations to arrive at the same unreadable answer — and its
repeat detector cannot tell "the model broke" from "this page really does
repeat", which a tax table does. The 25-27% multimodal cache hit rate is the
direct evidence: a quarter of all requests re-sent an image the pod had
already read. None of it was visible: a page generated four times over and a
page that was merely slow look identical from outside.

`max_retries` is unreachable from surya's settings — the vLLM backend calls
chat_completions_batch without it, so the default 3 was the only value there
has ever been. So the decision is hooked, beside the token-trace hook already
in that module, and `ocr.surya_max_retries` is a CEILING (asking for 10 where
surya allows 3 gets 3): 1 by default, 0 to disable, -1 to hand it back.

Every retry is logged, and a page that needed more than one call carries
`model_attempts` in its result — always, not only under surya_token_trace. A
page read first time stays byte-identical to what it always returned. The
counter is per THREAD, like the trace, or one lane would be blamed for
another lane's retries.

Also pins SURYA_INFERENCE_PARALLEL=1 in the served modes: a lane is our unit
of concurrency, and left unset surya sizes its own thread pool from a
built-in GPU table and fans out inside the worker, so instances x lanes would
stop being the real number of pages in flight.

19 tests in tests/test_surya_retry_budget.py; settings carry the new knob.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW