an operator may decline a self-harming page budget (v0.4.22)

A per-request `max_tokens` wins over the engine's own default and over the
operator's configured budget. That is right -- only the caller knows whether it
is sending a four-page contract or a fifty-page judgment -- and it stays the
default.

But it cost a run, and the operator had no say. A client sent
`max_tokens: 3072` on every page of a corpus whose pages need just over that.
Each page therefore generated to the cap, stopped mid-text, and paid a
continuation that RE-SENT the first 3072 tokens to retrieve the last few
hundred; by the second cut-off the model was often repeating, so the page was
refused a resume and re-run whole at a higher temperature.

Measured across 594 continuations on the live fleet: 532 pages reached segment
2, 44 segment 3, 11 segment 4, 7 segment 5. That distribution is the signature
of a budget MARGINALLY too small, which is the most expensive kind -- nearly
every page pays double, rather than a few pages paying a lot. The cards sat at
`GPU KV cache usage: 1.4%` with `Running: 0-4 reqs` while the pool grew to 65
pods at $47/hr, because the time was going on sequential round trips and not on
inference. Adding pods could not help and did not.

So `<engine>_max_tokens_floor` (surya, chandra, olmocr): an operator may state
a minimum. Off by default -- an install that silently overrode a caller would
be lying about its API -- never LOWERS a budget, because a caller asking for
more knows something the floor does not, and never invents one where the caller
asked for nothing (omitted means "use the configured value", which may well be
higher) or asked for 0 (which means unlimited, so raising it would cap it).

Applied once at the top of `ocr_document`, before either pipeline sees the
opts, so a local read and a pod forward cannot disagree about the budget. Said
once per engine per five minutes with both numbers and the reason, because a
5,000-document run would otherwise write the line 30,000 times, and because an
operator should not have to find this commit to learn why a small budget is
expensive. Seeded to pods as `CODERAI_OCR_<ENGINE>_MAX_TOKENS_FLOOR`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.21"
+__version__ = "0.4.22"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/api/runpod_worker.py

@@ -1504,6 +1504,8 @@ def _plan(engine, image, args, mcfg, api_key, entry, served,
                         f"{_eng}_max_continuations"),
                        (f"CODERAI_OCR_{_E}_CONTEXT_TOKENS",
                         f"{_eng}_context_tokens"),
+                       (f"CODERAI_OCR_{_E}_MAX_TOKENS_FLOOR",
+                        f"{_eng}_max_tokens_floor"),
                        (f"CODERAI_OCR_{_E}_MAX_RETRIES", f"{_eng}_max_retries"),
                        (f"CODERAI_OCR_{_E}_INSTANCES", f"{_eng}_instances"),
                        (f"CODERAI_OCR_{_E}_LANES", f"{_eng}_lanes"),

codai/config.py

@@ -1306,6 +1306,26 @@ class OcrConfig:
     #: stops making progress stops early regardless, so a generous budget costs
     #: nothing on a page that cannot use it. 0 disables resumption.
     surya_max_continuations: int = 8
+    #: FLOOR on a caller-supplied page budget. 0 = off, and off is the default:
+    #: a per-request `max_tokens` exists so the caller can choose, and an
+    #: install that silently overrode it would be lying about an API.
+    #:
+    #: Why it exists at all. A client sent `max_tokens: 3072` on every page of a
+    #: corpus whose pages need just over that. Each page therefore generated to
+    #: the cap, stopped mid-text, and paid a continuation that RE-SENT the first
+    #: 3072 tokens to retrieve the last few hundred -- and by the second cut-off
+    #: the model was often repeating, so the page was refused a resume and
+    #: re-run whole at a higher temperature. Measured across 594 continuations:
+    #: 532 pages reached segment 2, which is the signature of a budget that is
+    #: marginally too small rather than far too small -- the most expensive
+    #: case, because nearly every page pays double. The cards sat at 1.4% KV
+    #: cache while the fleet grew to 65 pods, and the operator paying for them
+    #: had no way to decline.
+    #:
+    #: So: an operator may state a minimum, and a request below it is raised to
+    #: it and SAID ONCE per engine. Never lowered -- a caller asking for more
+    #: than the floor knows something this setting does not.
+    surya_max_tokens_floor: int = 0
     #: surya's repetition detector, retuned. 0 on all three leaves its own
     #: defaults (4 repeats, a 500-character window, scaling 3.0).
     #:
@@ -1437,6 +1457,8 @@ class OcrConfig:
     #: so a generation cut off before the body closes parses to metadata and no
     #: text at all.
     olmocr_max_continuations: int = 8
+    #: See `surya_max_tokens_floor`. 0 = off.
+    olmocr_max_tokens_floor: int = 0
     #: The serving model's context window, or 0 to leave resumption uncapped by
     #: one. Resumption re-sends everything so far as INPUT, so the window — not
     #: the budget — is what a long page runs out of.
@@ -1489,6 +1511,8 @@ class OcrConfig:
     #: and reads in the finished text exactly like a page the model could not
     #: transcribe. 0 disables it; a request may override it per document.
     chandra_max_continuations: int = 8
+    #: See `surya_max_tokens_floor`. 0 = off.
+    chandra_max_tokens_floor: int = 0
     #: The serving model's context window, or 0 for no ceiling of our own.
     chandra_context_tokens: int = 0
     chandra_temperature: float = 0.0     # chandra's own default: greedy
@@ -2362,6 +2386,12 @@ class ConfigManager:
                      "surya_max_continuations", int),
                     ("CODERAI_OCR_SURYA_CONTEXT_TOKENS",
                      "surya_context_tokens", int),
+                    ("CODERAI_OCR_SURYA_MAX_TOKENS_FLOOR",
+                     "surya_max_tokens_floor", int),
+                    ("CODERAI_OCR_CHANDRA_MAX_TOKENS_FLOOR",
+                     "chandra_max_tokens_floor", int),
+                    ("CODERAI_OCR_OLMOCR_MAX_TOKENS_FLOOR",
+                     "olmocr_max_tokens_floor", int),
                     ("CODERAI_OCR_SURYA_MAX_RETRIES", "surya_max_retries", int),
                     ("CODERAI_OCR_SURYA_REPEAT_MAX_REPEATS",
                      "surya_repeat_max_repeats", int),

codai/ocr/manager.py

@@ -761,6 +761,56 @@ def _pod_pipeline_policy(cfg) -> tuple:
 POD_LEASE_ATTEMPT_S = 2.0
 
 
+def floor_page_opts(cfg, engine: str, opts) -> dict:
+    """Raise a caller's page budget to the operator's minimum, if there is one.
+
+    A per-request `max_tokens` exists so the caller can choose, and it WINS --
+    over the engine's own default and over the operator's configured budget.
+    That is the right default and it stays the default: this does nothing until
+    `<engine>_max_tokens_floor` is set, and it never lowers a budget, because a
+    caller asking for more than the floor knows something the floor does not.
+
+    What it is for. A client sent `max_tokens: 3072` on every page of a corpus
+    whose pages need just over that, so each page generated to the cap, stopped
+    mid-text, and paid a continuation that RE-SENT the first 3072 tokens to
+    fetch the last few hundred -- and by the second cut-off the model was often
+    repeating, so the page was refused a resume and re-run whole at a higher
+    temperature. 532 of 594 observed continuations were segment 2: the
+    signature of a budget marginally too small, which is the most expensive
+    kind because nearly every page pays double. The cards idled at 1.4% KV
+    cache, the fleet grew to 65 pods, and the operator paying for them had no
+    way to decline.
+
+    Returns a NEW dict; the caller's own is never mutated.
+    """
+    out = dict(opts or {})
+    name = str(engine or "").strip().lower() or "surya"
+    try:
+        floor = int(getattr(cfg, f"{name}_max_tokens_floor", 0) or 0)
+    except (TypeError, ValueError):
+        floor = 0
+    if floor <= 0:
+        return out
+    try:
+        asked = int(out.get("max_tokens") or 0)
+    except (TypeError, ValueError):
+        asked = 0
+    if asked <= 0 or asked >= floor:
+        return out
+    out["max_tokens"] = floor
+    # Once per engine, not per page: a 5,000-document run would otherwise write
+    # this line 30,000 times and bury it.
+    _say_rate_limited(
+        ("tokens-floor", name),
+        f"[ocr] a request asked for max_tokens={asked} on {name}, below this "
+        f"install's {name}_max_tokens_floor={floor} — raised to the floor. A "
+        f"budget just under what a page needs is the expensive case: the page "
+        f"generates to the cap, stops mid-text, and the continuation re-sends "
+        f"everything already produced to fetch the rest.",
+        every=300.0)
+    return out
+
+
 def _is_local_exhaustion(exc: BaseException) -> bool:
     """Did this fail because THIS BOX has no resources left?
 
@@ -1599,6 +1649,15 @@ class OcrManager:
             )
         return name
 
+    def resolve_engine_quietly(self, requested) -> str:
+        """`resolve_engine` without the refusal, for callers that only need a
+        name to look a setting up by. The real resolution still happens on the
+        read path and still reports."""
+        try:
+            return self.resolve_engine(requested)
+        except Exception:
+            return str(requested or "").strip().lower() or "surya"
+
     def engine_dpi(self, requested: Optional[str]) -> int:
         """PDF rasterisation DPI for the engine that will serve this request.
 
@@ -1758,6 +1817,10 @@ class OcrManager:
         """
         cfg = self._require_cfg()
         use_dpi = int(dpi) if dpi else self.engine_dpi(engine)
+        # Before either pipeline sees them, so the local read and the pod
+        # forward cannot disagree about the budget.
+        page_opts = floor_page_opts(cfg, self.resolve_engine_quietly(engine),
+                                    page_opts)
 
         # Price the document BEFORE rendering it: once the pages exist the memory is
         # already spent, so admission control has to work from the page tree.

tests/test_ocr_max_tokens_floor.py

+"""An operator may decline a self-harming page budget.
+
+A per-request `max_tokens` wins over the engine's own default and over the
+operator's configured budget. That is correct -- only the caller knows whether
+it is sending a four-page contract or a fifty-page judgment -- and it stays the
+default here.
+
+But it cost a real run. A client sent `max_tokens: 3072` on every page of a
+corpus whose pages need just over that, so each page generated to the cap,
+stopped mid-text, and paid a continuation that RE-SENT the first 3072 tokens to
+fetch the last few hundred. By the second cut-off the model was often repeating
+itself, so the page was refused a resume and re-run whole at a higher
+temperature. Measured across 594 continuations: **532 pages reached segment 2**,
+44 segment 3, 11 segment 4, 7 segment 5 -- the signature of a budget
+*marginally* too small, which is the most expensive kind, because nearly every
+page pays double rather than a few pages paying a lot.
+
+Meanwhile the cards sat at `GPU KV cache usage: 1.4%`, the fleet grew to 65
+pods, and the operator paying for them had no way to decline.
+
+So `<engine>_max_tokens_floor` lets them state a minimum. Off by default,
+never lowers a budget, and says so once per engine rather than once per page.
+"""
+
+import types
+
+import pytest
+
+from codai.config import OcrConfig
+from codai.ocr import manager as om
+
+
+@pytest.fixture(autouse=True)
+def _quiet():
+    """The rate limiter is module state and outlives a test."""
+    om._SAID_AT.clear()
+    yield
+    om._SAID_AT.clear()
+
+
+def _cfg(**kw):
+    c = OcrConfig()
+    for k, v in kw.items():
+        setattr(c, k, v)
+    return c
+
+
+# ------------------------------------------------------------ off by default #
+
+def test_the_floor_is_off_by_default():
+    """An install that silently overrode a caller would be lying about an API."""
+    c = OcrConfig()
+    assert c.surya_max_tokens_floor == 0
+    assert c.chandra_max_tokens_floor == 0
+
+
+def test_with_no_floor_the_caller_is_untouched():
+    out = om.floor_page_opts(_cfg(), "surya", {"max_tokens": 3072})
+    assert out == {"max_tokens": 3072}
+
+
+# ---------------------------------------------------------------- the floor #
+
+def test_a_budget_below_the_floor_is_raised():
+    """THE CASE: 3072 against pages that need just over it."""
+    out = om.floor_page_opts(_cfg(surya_max_tokens_floor=12288),
+                             "surya", {"max_tokens": 3072})
+    assert out["max_tokens"] == 12288
+
+
+def test_a_budget_above_the_floor_is_left_alone():
+    """A caller asking for more knows something the floor does not."""
+    out = om.floor_page_opts(_cfg(surya_max_tokens_floor=8192),
+                             "surya", {"max_tokens": 20000})
+    assert out["max_tokens"] == 20000
+
+
+def test_a_budget_exactly_at_the_floor_is_untouched():
+    out = om.floor_page_opts(_cfg(surya_max_tokens_floor=8192),
+                             "surya", {"max_tokens": 8192})
+    assert out["max_tokens"] == 8192
+
+
+def test_a_caller_that_said_nothing_is_not_given_a_budget():
+    """Omitted means 'use the configured value', which is not the floor's job
+    to override -- the configured value may well be higher."""
+    out = om.floor_page_opts(_cfg(surya_max_tokens_floor=8192), "surya", {})
+    assert "max_tokens" not in out
+
+
+def test_an_explicit_zero_is_not_raised():
+    """0 is a meaningful setting: unlimited. Raising it would cap it."""
+    out = om.floor_page_opts(_cfg(surya_max_tokens_floor=8192),
+                             "surya", {"max_tokens": 0})
+    assert out.get("max_tokens") == 0
+
+
+def test_other_options_survive():
+    out = om.floor_page_opts(_cfg(surya_max_tokens_floor=8192), "surya",
+                             {"max_tokens": 100, "max_continuations": 2})
+    assert out["max_continuations"] == 2
+
+
+def test_the_callers_dict_is_not_mutated():
+    asked = {"max_tokens": 3072}
+    om.floor_page_opts(_cfg(surya_max_tokens_floor=8192), "surya", asked)
+    assert asked == {"max_tokens": 3072}
+
+
+def test_none_opts_are_accepted():
+    assert om.floor_page_opts(_cfg(surya_max_tokens_floor=8192),
+                              "surya", None) == {}
+
+
+# ------------------------------------------------------------- per engine #
+
+def test_the_floor_is_per_engine():
+    """chandra's budgets are not surya's; one floor must not set the other's."""
+    cfg = _cfg(surya_max_tokens_floor=12288, chandra_max_tokens_floor=0)
+    assert om.floor_page_opts(cfg, "chandra",
+                              {"max_tokens": 3072})["max_tokens"] == 3072
+    assert om.floor_page_opts(cfg, "surya",
+                              {"max_tokens": 3072})["max_tokens"] == 12288
+
+
+def test_an_unknown_engine_falls_back_without_raising():
+    out = om.floor_page_opts(_cfg(surya_max_tokens_floor=8192),
+                             "something-else", {"max_tokens": 10})
+    assert out["max_tokens"] == 10
+
+
+def test_a_config_without_the_setting_is_fine():
+    """An install whose config predates this."""
+    out = om.floor_page_opts(types.SimpleNamespace(), "surya",
+                             {"max_tokens": 10})
+    assert out["max_tokens"] == 10
+
+
+def test_a_nonsense_floor_is_ignored():
+    out = om.floor_page_opts(_cfg(surya_max_tokens_floor="not a number"),
+                             "surya", {"max_tokens": 10})
+    assert out["max_tokens"] == 10
+
+
+# ------------------------------------------------------------- it says so #
+
+def test_it_says_what_it_did_and_why(capsys):
+    om.floor_page_opts(_cfg(surya_max_tokens_floor=12288),
+                       "surya", {"max_tokens": 3072})
+    out = capsys.readouterr().out
+    assert "3072" in out and "12288" in out
+    assert "surya_max_tokens_floor" in out
+    # The reason, not just the fact: an operator should not have to find the
+    # commit to learn why a small budget is expensive.
+    assert "re-sends" in out
+
+
+def test_it_is_said_once_per_engine_not_once_per_page(capsys):
+    """A 5,000-document run must not write this 30,000 times."""
+    cfg = _cfg(surya_max_tokens_floor=12288)
+    for _ in range(500):
+        om.floor_page_opts(cfg, "surya", {"max_tokens": 3072})
+    assert capsys.readouterr().out.count("max_tokens_floor") == 1