document the client pod ceiling and the cost figures to watch

The throughput guide told a client to lease pods and said nothing about
bounding what that lease may grow to, because there was no way to. Adds
the staging recipe (one call per stage, no restart), what the three
$/hr figures now mean and which one to alarm on, and that a `degraded`
status document is stale rather than an idle fleet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

docs/ocr-client-throughput-guide.md

+# How to call coderai's OCR for maximum throughput
+
+Measured 2026-10-18 on rented A40-class pods, through `POST /v1/ocr/batch`.
+Every number came from the real HTTP path.
+
+**Supersedes the earlier note.** Two things in it were wrong: there is no
+request-count ceiling on the OCR path, and the pages-per-request escalation it
+implied is unnecessary. Corrections are marked below.
+
+---
+
+## 1. The request — one page per call
+
+```
+POST /v1/ocr/batch
+Authorization: Bearer <token>
+Content-Type: multipart/form-data
+
+files              ONE file per call
+engine             surya
+detect             off
+dpi                250
+max_tokens         4096                  <- was 12288
+max_continuations  1                     <- was 3
+```
+
+One file per call, always. There is no reason to pack:
+
+* a batch returns only when its slowest document returns, so one degenerate
+  page holds the rest hostage;
+* a failure anywhere costs the whole batch;
+* it saves no memory — bytes in flight are `pages × bytes per page` either way.
+
+`max_tokens` and `max_continuations` are per-request fields. Changing them needs
+no deploy on either side, and they are worth +33% throughput and −25% cost.
+
+## 2. Concurrency — one rule
+
+```
+in_flight = 64 × pods_ready        (capped at 8192)
+```
+
+Read `pods_ready` from `GET /v1/runpod/status` → `summary.pods_ready` and
+recompute as the fleet grows and shrinks. One request carries one page, so the
+request count *is* the page count.
+
+**CORRECTION.** The earlier note said the server admits 256 concurrently and
+queues 2048, so above ~2300 outstanding requests you would be refused. That is
+false for OCR: `server.max_parallel_requests` and `server.queue_max_size` gate
+`/v1/chat/completions` only. On the OCR path every request you open is accepted
+immediately. The limits that do exist are pages, not requests:
+
+| limit | value | what it is |
+|---|---|---|
+| `max_inflight_per_pod × pods` | 64 × pods | pages a pod will take at once |
+| `ocr.max_concurrency` | 8192 | pages admitted into the OCR manager |
+| pod thread pool | 8192 | one thread per page in flight to a pod |
+
+All three line up at **8192 pages = 128 pods**, which is the configured
+`max_pods`. Asking for more than 8192 does not fail fast — the surplus waits for
+a pod lease and eventually times out, which is worse than being refused. Stay at
+or under `64 × pods_ready`.
+
+## 3. Memory — yours to bound, not ours
+
+The upload is read into memory when the request arrives, **before** any page
+limit applies. So the bytes held are set by what you send, not by a server
+setting:
+
+```
+8192 single-page requests × ~175 KB        ≈ 1.4 GB
+  + the per-page PDF split from each       ≈ 2.8 GB total      fine
+same 8192 slots at 4 files per call        ≈ 11 GB             avoid
+```
+
+Another reason for one file per call.
+
+## 4. Expected rates
+
+Per pod, single-page documents, A40-class card at $0.59/hr:
+
+| setting | pages/s | pages/$ | $/page |
+|---|---|---|---|
+| `max_tokens` 12288, `max_continuations` 3 | 0.413 | 2,523 | $0.000396 |
+| **`max_tokens` 4096, `max_continuations` 1** | **0.549** | **3,349** | **$0.000298** |
+
+Multiply by `pods_ready`. At the 128-pod ceiling that is ~70 pages/s, ~253,000
+pages/hour.
+
+Page latency is **not** uniform: p50 44 s, p95 169 s, worst ~450 s. Set the
+client timeout to at least **600 s per request** and do not treat a slow page as
+a failure.
+
+## 5. Which engine
+
+| engine | pages/s | $/page | success | loops | generations/page |
+|---|---|---|---|---|---|
+| `surya` | 0.549 | $0.000298 | 98% | 4% | 1.73 |
+| `chandra` | 0.251 | $0.000651 | **100%** | **0%** | **1.02** |
+
+`surya` for bulk. Its failures are mechanically identifiable, so the cheapest
+route to high quality is surya first, chandra only where needed:
+
+```
+re-read with chandra when:
+    text shorter than 50 characters                 (empty)
+    OR share of distinct 20-char windows < 0.5      (degenerate repetition)
+```
+
+~6% of pages qualify, so combined cost is ~$0.000319/page — near chandra's
+quality at near surya's price. No human triage: both tests are mechanical.
+
+## 6. Reading the response
+
+* **A 200 is not necessarily a good page.** A looping or truncated page answers
+  200. Check `num_pages`, text length, and the repetition test above.
+* **`fallback_engines` is off**, deliberately: a degraded page is never silently
+  re-read by another engine. You decide what to re-read.
+* **Expect ~2% of pages to error** (502 from a pod, seen at that rate). Retry
+  them — they are not deterministic failures.
+* **~10% of this archive is unreadable by anyone**: 487 of 4,961 PDFs have no
+  `%PDF` header or no `%%EOF`. Screen locally before sending; each one
+  otherwise costs a request and returns an error.
+
+## 7. Before a large run
+
+1. Lease the fleet you want:
+
+   ```json
+   POST /v1/runpod/scale
+   {"model": "datalab-to/surya-ocr-2", "pods": 8, "max_pods": 8}
+   ```
+
+   **Send both.** `pods` is a FLOOR — "hold at least this many ready". It says
+   nothing about how large the fleet may become, and the autoscaler will keep
+   growing toward the configured ceiling for as long as pages keep arriving.
+   `max_pods` is the ceiling for the life of your lease, and it is the field
+   that bounds what a stage costs. It may only ever narrow the operator's
+   configured ceiling; a larger number is clamped and reported in `notes`.
+
+   A cold pod needs **~12 minutes** to pull its image and get weights
+   resident, so the first pages of an unscaled run are slow.
+
+2. Poll `GET /v1/runpod/status` until `summary.pods_ready` matches. The
+   per-model rows under `schedules` report `effective_max_pods` (the ceiling
+   actually in force), `client_max_pods` (your lease's own) and
+   `client_lease_expires_in_s`.
+3. Then feed, holding `in_flight = 64 × pods_ready`, recomputed as that number
+   changes.
+
+### Staging a campaign
+
+Each stage is one call, no restart in between:
+
+```json
+{"pods":   8, "max_pods":   8}     ← stage 1
+{"pods":  32, "max_pods":  32}     ← stage 2
+{"pods":  64, "max_pods":  64}     ← stage 3
+{"pods": 128, "max_pods": 128}     ← the rest
+{"pods": 0, "teardown": true}      ← done; give the fleet back now
+```
+
+`{"pods": 0, "max_pods": N}` is a cap with no warm floor — useful for a long
+tail where you want the ceiling but do not need pods held ready. `{"pods": 0}`
+alone releases the whole lease. The lease expires by itself after 30 minutes,
+so a client that dies does not leave pods billing; **re-send the current stage
+every few minutes** for a run longer than that.
+
+### Watch the right cost number
+
+`live_hourly_usd` from `GET /v1/runpod/status` counts **every pod that bills**,
+including the ones still booting — RunPod charges from creation, and a pod is
+not healthy for its first ~12 minutes. Alongside it:
+
+* `live_hourly_serving_usd` — the part that is actually reading pages
+* `live_hourly_booting_usd` — the part that is paid boot producing nothing
+
+A high booting share means the ramp is too steep: you are paying for cards that
+cannot help yet. Settle a stage before asking for the next one. (Before
+v0.4.34 `live_hourly_usd` counted only healthy pods, so a booting fleet
+reported as free — it read $0.00 while 13 pods billed $7.67/hr.)
+
+If a `degraded` key appears in the status document, one of its three sources
+did not answer in time and that part of the answer is stale — it is not a
+report of an idle fleet.
+
+Pods are not kept warm between campaigns (`min_pods: 0`, `keep_warm: false`), so
+an idle fleet costs nothing and every run starts cold by design.

docs/ocr-throughput-measurements-2026-10-18.md

+# OCR throughput and cost on rented GPUs — measurements, 2026-10-18
+
+All figures measured through the **full orchestrator path** (`POST /v1/ocr/batch`
+on coderai), never against a pod directly: a number taken by bypassing the
+orchestrator describes a path no client will use.
+
+Corpus: 500 single-page PDFs (`~/antani2`), median 89 KB, p95 589 KB, all
+structurally valid. 100 of them per rung, same 100 every time.
+
+---
+
+## 1. How throughput is computed, and why the obvious ways are wrong
+
+Three numbers can be derived from one rung and only one of them extrapolates:
+
+| method | 100-page rung | what it measures |
+|---|---|---|
+| completions ÷ elapsed, at 80/100 | 0.526 pages/s | the **fastest** 80 pages — completion-order bias |
+| completions ÷ total elapsed | 0.085 pages/s | includes a **draining tail** with the card idle |
+| **concurrency ÷ mean latency** | **0.413 pages/s** | steady state with the queue full |
+
+The third is the right one for a long campaign: a million-page queue never
+drains, so the tail idle never happens, and no page is privileged by finishing
+first. Every `pages/s` below is `slots ÷ mean page latency`.
+
+---
+
+## 2. Page budget — surya on one A40 ($0.59/hr)
+
+Concurrency 32, `dpi=250`, `detect=off`.
+
+| `max_tokens` / `max_continuations` | pages/s | pages/$ | $/page | 996,760 pages | chars med | mean | p95 | worst page |
+|---|---|---|---|---|---|---|---|---|
+| 12288 / 3 *(sent today)* | 0.413 | 2,523 | $0.000396 | **$395** | 1,952 | 77.4 s | 133 s | 1,137 s |
+| **4096 / 1** ← best | **0.549** | **3,349** | **$0.000298** | **$298** | 1,970 | 58.3 s | 169 s | 450 s |
+| 2048 / 1 | 0.500 | 3,052 | $0.000327 | $327 | 1,970 | 64.0 s | 281 s | 598 s |
+
+**+33% throughput, −25% cost, identical text.** Quality is unchanged across all
+three: `uniq20` median 1.00, zero U+FFFD, 97–98 of 100 pages ok.
+
+### Why there is an optimum rather than "lower is better"
+
+2048 is worse than 4096. Below ~4096 tokens a legitimately long page no longer
+fits in one generation and needs a continuation, and prefix caching is **off**
+for this model (`enable_prefix_caching=False` in the pod's vLLM config), so each
+continuation re-pays the full image prefill. It shows up precisely where
+expected: p95 latency rises 169 s → 281 s while the median page is unchanged.
+
+### Where the budget actually goes
+
+The budget never binds a normal page — the median page is 1,952 characters,
+roughly 600 tokens. It bounds the damage a **degenerate** one can do:
+
+```
+slowest 5% of pages consumed 43% of card time at cap 12288, 27% at cap 4096
+
+consiglio_stato-sen-2013-201300001_p0003   12288: 1137s  38,014 chars  uniq20 0.040
+                                            4096:  450s  17,586 chars  uniq20 0.071
+dgn_cc_0151460094_2996246_p0045            12288: 1078s  25,282 chars  uniq20 0.045
+                                            4096:  380s  11,714 chars  uniq20 0.048
+```
+
+`uniq20` is the share of distinct 20-character windows; 0.04 means 96% of the
+output is repetition. At `12288 × (1+3)` one looping page may emit ~49,000
+tokens. These pages still loop — the engine's repetition problem is untouched —
+but each costs ~2.5× less and writes ~20,000 fewer characters of garbage.
+
+**Recommended client setting: `max_tokens=4096`, `max_continuations=1`.** Both
+are per-request fields: no deploy, no restart.
+
+---
+
+## 3. Engine comparison — surya vs chandra, same card, same budget, same pages
+
+| engine | card | pages/s | pages/$ | $/page | chars med | notes |
+|---|---|---|---|---|---|---|
+| surya | A40 | 0.549 | 3,349 | $0.000298 | 1,970 | cap 4096 / cont 1 |
+| chandra | A40 | *pending* | | | | cap 4096 / cont 1, 128 instances |
+
+Chandra could not be measured at all until the pod image was rebuilt — see §5.
+
+---
+
+## 4. Quality, surya at cap 12288 (96 pages inspected)
+
+```
+chars    min 0   p25 1,213   median 1,952   p75 2,614   max 12,748
+uniq20   median 1.000   p25 0.990          (low = repeating)
+U+FFFD   0 on every page
+looping (uniq20 < 0.5)   1 of 96
+empty   (<50 chars)      7 of 96
+```
+
+**Zero replacement characters** across every page — the accent corruption this
+corpus was suspected of is absent, with `surya_repair_accents` on.
+
+Of the 7 empty pages, 4 have near-blank sources (1.7–12.7 KB) and **3 do not**
+(9.6 KB, 9.6 KB, 59.9 KB) — ~3% of real pages returning nothing, which is now a
+larger quality question than looping and is not yet explained.
+
+Corpus hygiene, separately: **487 of 4,961 PDFs in the multi-page archive
+(~10%) are structurally broken** — no `%PDF` header or no `%%EOF`. They fail on
+arrival and cost nothing, but they will not be read either.
+
+---
+
+## 5. Defects found and fixed while measuring
+
+Every one of these had to be fixed before any number could be trusted; each was
+only visible once the previous one was gone.
+
+| ver | defect | evidence |
+|---|---|---|
+| 0.4.24 | `/admin/api/system/reload` built a bare `Request({})` and raised on every call — it had never reloaded anything | `KeyError 'type'` |
+| 0.4.25 | admission gate defaulted to **2 in flight / 6 queued**; a pod was handed 64 through env while the front in front of it admitted 2 | pod at `Running: 1, Waiting: 0`; 219 pages in 1210 s, same as reading them one at a time |
+| 0.4.26 | the 0.4.25 diagnostic compared *requests* against *pages* and fired against the value coderai itself had just derived | warned "only 256 request(s)" about its own choice |
+| 0.4.27 | `max_pods` counted only **healthy** pods, so an adopted pod that was briefly unhealthy left room under a cap of 1 | two A40s on a `max_pods=1` pool, $1.18/hr |
+| 0.4.28 | `pod.inflight` was a ratchet — raised in `_pick`, lowered only in `release`, no TTL — so one lost page took a pod out of service permanently | pool said 64/64 in flight; pod said `Running: 31` at 10.6% KV |
+| 0.4.29 | two pod-registration paths with no identity check between them | precautionary; see commit for what was and was not concluded |
+| 0.4.30 | corrected an unsupported claim in 0.4.29 about what caused the double billing | — |
+| 0.4.31 | `CODERAI_OCR_CHANDRA_INSTANCES` / `OLMOCR_INSTANCES` were **sent to every pod and read by nobody**; a pod has no config file, so those engines silently ran one instance | `[ocr] engine 'chandra': 1 instance(s) ready` → after the fix, `128 instance(s) ready`, `Running: 1` → `12–13` |
+
+Pod image `ghcr.io/nextime/coderai-vllm-ocr:0.4.31` rebuilt, scanned, pushed and
+signed — `sha256:9f96db69…`, cosign `{"git":"9632e66","version":"0.4.31"}`.
+
+### Settings that matter, and were wrong by default
+
+```
+server.max_parallel_requests   2  →  sized to the pool
+server.queue_max_size          6  →  sized with it
+```
+
+The gate must be derived from pool capacity, not fixed:
+
+```
+healthy_pods × max_inflight_per_pod ÷ pages_per_document
+```
+
+The same install needs **8** for ~9-page documents and **64** for single-page
+ones. 0.4.25 derives it from `ocr.max_concurrency`, which does not know how many
+pods exist; deriving from live pool capacity is still open.
+
+---
+
+## 6. Extrapolation to 996,760 pages
+
+At the best measured setting (surya, A40, cap 4096 / cont 1):
+
+```
+pages/s per pod   0.549          →  1,976 pages/hour
+$/page            $0.000298
+GPU cost          ~$298
+pod-hours         504
+```
+
+Cost is **independent of fleet size** — pages per dollar is fixed, so more pods
+buy time, not money. Only boot overhead scales (~12 min per pod before it serves
+a page, ≈ $0.12):
+
+| fleet | wall clock | boot overhead |
+|---|---|---|
+| 1 pod | 504 h (21 days) | $0.12 |
+| 16 | 32 h | $1.9 |
+| 32 | 16 h | $3.8 |
+| 64 | 7.9 h | $7.6 |
+| 128 | **3.9 h** | $15.1 |
+
+**Not included:** the ~10% structurally broken PDFs; the ~3% of real pages
+returning empty; and the fact that these are single-page documents — a denser
+page costs more tokens, so treat $298 as the floor for this page profile.
+
+---
+
+## 7. Open
+
+- **A100-SXM4-80GB sweep for surya** (concurrency 32/64/96 at cap 4096/cont 1).
+  An A100 SXM is ~3.4× an A40's price, so it must beat 3.4× on pages/s to win on
+  pages/$. The A40 sat pinned at `Running: 32` with KV at only 12%, so its
+  ceiling may not be VRAM at all — in which case the extra 32 GB buys nothing.
+- **chandra** at the same budget and card (run in progress).
+- **~3% of real pages return empty** — unexplained.
+- **2 pages per 100 answered 502** from the pod in two consecutive runs.
+- **Derive the admission gate from live pool capacity** rather than from
+  `ocr.max_concurrency`.
+- **Early loop detection.** A page is only judged degenerate *after* it has run
+  the budget to its cap. `uniq20` collapses to ~0.04 on those pages, so the
+  signal is cheap and available mid-stream; aborting there would cut the worst
+  case far below what any budget can.
+- `enable_prefix_caching=False` on the pod's vLLM makes every continuation
+  re-pay the image prefill. Worth checking whether this model supports it.