ocr: resumption must respect the context window, and never extend a loop (v0.3.50)

Two defects that a throughput collapse on a live install made visible. With
surya_max_tokens raised to 12288, 262 resumption segments spent 1.6M generated
tokens in 80 minutes, 122 of them running the full cap dry, while 253 of those
262 ended in "came back repeating itself".

Resumption re-sends the text so far as INPUT, so the model's context window --
not the continuation budget -- is the real ceiling. The budget did not know
that: at a 12288-token cap, the third segment of a page asked for 12288 more on
top of ~28,500 already in a 32,768-token window and was rejected outright. The
page was then reported as one that could not be resumed AT ALL, discarding text
already paid for.

  - surya_context_tokens, or 0 to read the server's own max_model_len from
    /v1/models. Configuration wins: an operator may serve a shorter window than
    the checkpoint advertises.
  - Each segment asks for the smaller of the cap and the room left, measured
    from what the server said the prompt cost (usage.prompt_tokens), not
    guessed; the guess covers the first resumption only.
  - Out of room stops, KEEPS what it has, and says so -- naming the fix that
    helps, which is a SMALLER max_tokens, not a bigger budget.
  - A refusal for length is not retried: the fallback resumption adds a user
    turn, so it is longer than the request just refused.
  - Not knowing the window changes nothing and guesses nothing.

The second defect: a reply that trips surya's repeat detector hit the cap
because it is LOOPING, not because the page is long. Resuming it can only
extend the loop, and surya re-runs the whole page anyway -- so one page on that
path cost four full generations. It is no longer resumed, and it is not marked
truncated either: that is surya's retry to make, and "truncated" would send the
caller looking for a bigger budget it cannot use.

A fixed token cap is also an accidental repetition circuit-breaker, which is
why raising it slowed the LOOPING pages by the ratio of the caps rather than
every page. Documented in docs/ocr.md with the window arithmetic (2048 -> ~13
segments, 12288 -> 2).

Tests: 8 new (window clamp, stop-one-segment-early, length refusal not
retried, unknown window, configuration over server, a cap below the 256-token
floor is never mistaken for a full window, a looping reply is not resumed, a
genuinely long page still is). 62 pass in the surya/blank-page suites; the
floor-versus-cap bug was caught by the existing 44.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.3.49"
+__version__ = "0.3.50"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -4142,6 +4142,7 @@ def build_settings_dict(c, gpu_cards):
             "surya_max_retries": getattr(c.ocr, "surya_max_retries", 1),
             "surya_max_tokens": getattr(c.ocr, "surya_max_tokens", 0),
             "surya_max_continuations": getattr(c.ocr, "surya_max_continuations", 2),
+            "surya_context_tokens": getattr(c.ocr, "surya_context_tokens", 0),
             "surya_langs": c.ocr.surya_langs,
             "surya_token_trace": c.ocr.surya_token_trace,
             "surya_repair_accents": c.ocr.surya_repair_accents,
@@ -4971,6 +4972,10 @@ async def api_save_settings(request: Request, username: str = Depends(require_ad
             o.surya_max_continuations = _i(
                 "surya_max_continuations",
                 getattr(o, "surya_max_continuations", 2), 0)
+        if "surya_context_tokens" in d:
+            o.surya_context_tokens = _i(
+                "surya_context_tokens",
+                getattr(o, "surya_context_tokens", 0), 0)
         if "surya_max_tokens" in d:
             o.surya_max_tokens = _i("surya_max_tokens",
                                     getattr(o, "surya_max_tokens", 0), 0)

codai/admin/templates/settings.html

@@ -1208,6 +1208,7 @@
       <div class="form-group" style="flex:1;min-width:130px"><label class="form-label" title="How many times Surya may RE-run a page whose generation repeated itself. Surya's own number is 3 and each retry is another FULL generation at a higher temperature: a batch measured on 2026-10-13 spent 2,644 generated tokens per page on text worth ~400, with a quarter of all requests re-sending an image the pod had already read. 1 keeps the rescue a retry does buy and drops the long tail; 0 disables it; -1 means whatever Surya decides. A retried page is reported as model_attempts in the result.">Retries / page</label><input type="number" id="s-ocr-surya-max-retries" class="form-input" min="-1" placeholder="1"></div>
       <div class="form-group" style="flex:1;min-width:140px"><label class="form-label" title="Cap on the tokens Surya may generate for ONE page. 0 uses Surya's own 2048, which its vLLM backend never overrides. Surya-ocr-2 emits block markup (a category and a bbox per block) BEFORE the text, so a reply stopped at the cap is incomplete markup and parses to a few blocks or to none — the page then arrives as a short fragment or as no text at all, which looks exactly like a page the model could not read. A dense page can legitimately need several thousand tokens. model_attempts.truncated in a page's result tells you when this is what went wrong.">Max tokens / page</label><input type="number" id="s-ocr-surya-max-tokens" class="form-input" min="0" placeholder="0 = 2048"></div>
       <div class="form-group" style="flex:1;min-width:140px"><label class="form-label" title="How many times a page whose reply hit the token cap may be RESUMED from where it stopped. Raising Max tokens only moves the cliff — a dense enough page overflows any fixed number, and incomplete block markup parses to a fragment or to nothing. Resumption sends the text so far back as a prefilled assistant turn so the model carries on the exact string. 2 by default, because returning a page known to be incomplete is a correctness failure, not a speed trade-off; 0 restores the old behaviour. A page still unfinished afterwards is marked model_attempts.complete = false so a caller can refuse it.">Resume cut-off pages</label><input type="number" id="s-ocr-surya-max-continuations" class="form-input" min="0" placeholder="2"></div>
+      <div class="form-group" style="flex:1;min-width:150px"><label class="form-label" title="The serving model's context window, in tokens. 0 asks the server for its own max_model_len (vLLM reports it on /v1/models) — leave it at 0 unless the server will not say, or is served with a shorter --max-model-len than the checkpoint advertises. This is the ceiling resumption really runs into: every resumption re-sends the text so far as INPUT, so once the window is nearly full, asking for another whole Max tokens is not a bigger segment, it is an HTTP 400 — and a page that was nearly whole gets reported as a page that could not be resumed at all. Knowing the number lets such a page stop one segment early, keep what it already has and say so. It also means a SMALLER Max tokens fits more text in the same window, because the last segment wastes less of it.">Context window</label><input type="number" id="s-ocr-surya-context-tokens" class="form-input" min="0" placeholder="0 = ask the server"></div>
       <label style="display:flex;align-items:center;gap:.4rem;cursor:pointer" title="Local mode only. On, Surya runs on this machine's GPU or refuses to load — it will not quietly fall back to the CPU, where a page takes minutes. Off, the CPU is used deliberately."><input type="checkbox" id="s-ocr-surya-use-gpu" checked><span>Require GPU (Local)</span></label>
       <div class="form-group" style="flex:1;min-width:120px"><label class="form-label">Langs</label><input type="text" id="s-ocr-surya-langs" class="form-input" placeholder="it"></div>
       <label style="display:flex;align-items:center;gap:.4rem;cursor:pointer" title="Return the model's own per-token bytes wherever a page came back unreadable. A diagnostic — leave off unless you are chasing lost characters."><input type="checkbox" id="s-ocr-surya-token-trace"><span>Trace unreadable tokens</span></label>
@@ -2210,6 +2211,7 @@ async function loadSettings(){
     document.getElementById('s-ocr-surya-max-retries').value = ocr.surya_max_retries ?? 1;
     document.getElementById('s-ocr-surya-max-tokens').value = ocr.surya_max_tokens ?? 0;
     document.getElementById('s-ocr-surya-max-continuations').value = ocr.surya_max_continuations ?? 2;
+    document.getElementById('s-ocr-surya-context-tokens').value = ocr.surya_context_tokens ?? 0;
     document.getElementById('s-ocr-surya-use-gpu').checked = ocr.surya_use_gpu !== false;
     document.getElementById('s-ocr-surya-langs').value = ocr.surya_langs ?? 'it';
     document.getElementById('s-ocr-surya-token-trace').checked = !!ocr.surya_token_trace;
@@ -2503,6 +2505,7 @@ async function saveSettings(silent){
       // 0 is meaningful (resumption off), so an explicit NaN check.
       surya_max_continuations: (function(v){ v = parseInt(v); return isNaN(v) ? 2 : v; })(
         document.getElementById('s-ocr-surya-max-continuations').value),
+      surya_context_tokens: parseInt(document.getElementById('s-ocr-surya-context-tokens').value) || 0,
       surya_langs: document.getElementById('s-ocr-surya-langs').value.trim() || 'it',
       surya_token_trace: document.getElementById('s-ocr-surya-token-trace').checked,
       surya_repair_accents: document.getElementById('s-ocr-surya-repair-accents').checked,

codai/config.py

@@ -1090,6 +1090,19 @@ class OcrConfig:
     #: stops making progress stops early regardless, so a generous budget costs
     #: nothing on a page that cannot use it. 0 disables resumption.
     surya_max_continuations: int = 8
+    #: The serving model's context window, or 0 to ask the server for its own
+    #: `max_model_len` (vLLM reports it on `/v1/models`).
+    #:
+    #: This is the ceiling resumption actually runs into, and the one it used to
+    #: hit the hard way: each round re-sends the text so far as INPUT, so asking
+    #: for another full `surya_max_tokens` once the window is nearly full is not
+    #: a bigger segment, it is an HTTP 400 -- and a page that was nearly whole
+    #: gets reported as a page that could not be resumed at all. Knowing the
+    #: number lets a page stop one segment early, keep what it has, and say so.
+    #:
+    #: Set it when the server does not report a window, or when it is served
+    #: with a shorter `--max-model-len` than the checkpoint advertises.
+    surya_context_tokens: int = 0
     #: A page whose text is this short gets its ink measured, so a short result
     #: can be explained rather than guessed at. 0 disables the measurement.
     blank_page_max_chars: int = 200

codai/ocr/surya.py

@@ -80,7 +80,10 @@ class SuryaEngine(SubprocessOcrEngine):
                 "max_tokens": max(0, int(getattr(self.cfg, "surya_max_tokens", 0) or 0)),
                 # Resumptions allowed for a page whose reply hit the cap.
                 "max_continuations": max(0, int(getattr(
-                    self.cfg, "surya_max_continuations", 2) or 0))}
+                    self.cfg, "surya_max_continuations", 2) or 0)),
+                # 0 = ask the server for its own max_model_len.
+                "context_tokens": max(0, int(getattr(
+                    self.cfg, "surya_context_tokens", 0) or 0))}
 
     #: Modes where the recognition VLM lives in a model server we only talk to.
     SERVED_MODES = ("model", "vllm", "llamacpp", "server")

docs/ocr.md

@@ -1112,6 +1112,69 @@ drops unknown form fields **silently**. Before these parameters existed, a
 caller could send `max_tokens=8000` on every request for a week and never be
 told it did nothing.
 
+### The context window is the ceiling, not the budget
+
+Resumption re-sends the text so far as **input**. The continuation budget is
+therefore not what limits how long a page may be — the model's context window
+is, and the budget only decides how much of that window gets spent before
+coderai gives up.
+
+That arithmetic is the whole reason `surya_max_tokens` should be *modest* rather
+than as large as it will go. The two settings multiply into the same total
+capacity, but a large cap spends the window in coarse lumps:
+
+| `surya_max_tokens` | segments that fit a 32,768-token window (image ≈ 4,000) |
+|---|---|
+| 2,048 | ~13 |
+| 4,096 | ~7 |
+| 12,288 | **2** |
+
+A 12,288-token cap buys no more text than a 2,048-token cap — it buys the same
+text in two steps instead of thirteen, and it makes every wasted step six times
+more expensive. Measured on one install on 2026-10-13, with the cap at 12,288:
+262 resumption segments spent 1.6 million generated tokens, 122 of them running
+the full cap dry, and the third segment of any page was **rejected outright** —
+12,288 asked on top of ~28,500 already in the window.
+
+So coderai now establishes the window and respects it:
+
+- `surya_context_tokens`, or `0` to read the server's own `max_model_len` from
+  `/v1/models` (vLLM reports it). Configuration wins, because an operator may
+  serve with a shorter `--max-model-len` than the checkpoint advertises.
+- Each segment asks for **the smaller of the cap and the room left**, and the
+  room left is measured from what the server said the prompt actually cost
+  (`usage.prompt_tokens`), not guessed — the guess is used for the first
+  resumption only.
+- A page with no usable room left stops, **keeps what it has**, and is marked
+  incomplete. Previously it was refused by the server and reported as a page
+  that could not be resumed at all, discarding text already paid for.
+- A refusal *for length* is not retried. The fallback resumption adds a user
+  turn, so it is longer than the request just refused; retrying it only spends
+  another round trip to be told the same thing.
+- Not knowing the window is survivable, and nothing is guessed: with no number,
+  resumption behaves exactly as it did before and the budget rules.
+
+### A reply that repeats itself did not run out of room
+
+A generation that trips surya's repeat detector hit the cap because it is
+**looping**, not because the page is long. Resuming it can only extend the loop
+— and surya is about to re-run the whole page anyway, which is what its repeat
+detector does, so every token spent resuming is spent on text that is then
+thrown away.
+
+This was the single largest waste the cap-and-resumption combination could
+produce. On the same install, **253 of 262** resumptions ended in "came back
+repeating itself", and 187 of those pages were then re-run in full: one page on
+the bad path cost four full generations at 12,288 tokens each. Such a reply is
+now not resumed at all, and it is **not** marked truncated either — it is a
+degenerate reply, which is surya's retry to make. Calling it truncated would
+send the caller looking for a bigger budget it cannot use.
+
+Worth keeping in mind when reading a throughput regression: a fixed token cap is
+also an accidental repetition circuit-breaker. Raising it does not slow a page
+down by the ratio of the caps; it slows down the *looping* pages by that ratio,
+and they are the ones that were already costing the most.
+
 ---
 
 ## An empty page and a 71-character page are the same bug