-
Stefy Lanza (nextime / spora ) authored
A per-request `max_tokens` wins over the engine's own default and over the operator's configured budget. That is right -- only the caller knows whether it is sending a four-page contract or a fifty-page judgment -- and it stays the default. But it cost a run, and the operator had no say. A client sent `max_tokens: 3072` on every page of a corpus whose pages need just over that. Each page therefore generated to the cap, stopped mid-text, and paid a continuation that RE-SENT the first 3072 tokens to retrieve the last few hundred; by the second cut-off the model was often repeating, so the page was refused a resume and re-run whole at a higher temperature. Measured across 594 continuations on the live fleet: 532 pages reached segment 2, 44 segment 3, 11 segment 4, 7 segment 5. That distribution is the signature of a budget MARGINALLY too small, which is the most expensive kind -- nearly every page pays double, rather than a few pages paying a lot. The cards sat at `GPU KV cache usage: 1.4%` with `Running: 0-4 reqs` while the pool grew to 65 pods at $47/hr, because the time was going on sequential round trips and not on inference. Adding pods could not help and did not. So `<engine>_max_tokens_floor` (surya, chandra, olmocr): an operator may state a minimum. Off by default -- an install that silently overrode a caller would be lying about its API -- never LOWERS a budget, because a caller asking for more knows something the floor does not, and never invents one where the caller asked for nothing (omitted means "use the configured value", which may well be higher) or asked for 0 (which means unlimited, so raising it would cap it). Applied once at the top of `ocr_document`, before either pipeline sees the opts, so a local read and a pod forward cannot disagree about the budget. Said once per engine per five minutes with both numbers and the reason, because a 5,000-document run would otherwise write the line 30,000 times, and because an operator should not have to find this commit to learn why a small budget is expensive. Seeded to pods as `CODERAI_OCR_<ENGINE>_MAX_TOKENS_FLOOR`. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
0ed00f5b