-
Stefy Lanza (nextime / spora ) authored
Resumption was surya's alone. chandra and olmOCR stopped at their configured cap and returned whatever they had — and what they had is the worse half of the problem: chandra emits HTML, so a cut-off reply is unclosed markup that parses to a fragment of the page or to nothing, and olmOCR's reply opens with YAML front matter, so one cut off before the body closes parses to metadata and NO TEXT AT ALL. In the finished page both are indistinguishable from a model that could not read it, which is the confusion this whole area exists to remove. The rules now live once, in codai/ocr/resume.py, and they are the ones surya learned the hard way rather than a fresh guess: * both vLLM flags or neither — `continue_final_message` AND `add_generation_prompt: false`. With the second left at its default the chat template appends a fresh assistant turn and the model answers again about the same image, which reads exactly like a model repeating itself and did for a whole day; * never resume a reply that is already looping: it did not run out of room, it ran out of page, and resuming can only extend the loop; * the context window is the ceiling, not the budget. Resumption re-sends everything so far as INPUT, so the room shrinks by what was just produced, and a segment that no longer fits means the cap is too BIG, not too small; * progress has to be real: a segment identical to the previous one, or one that comes back looping, ends the loop; * `finish_reason` is authoritative both ways — "stop" with a full budget is a complete page and must not be resumed, which a token count alone cannot tell. Both engines' ask paths now return a `resume.Reply` carrying the finish reason and the usage counts (they were discarding both), take a `prefix` that goes back as the prefilled assistant turn, and report `model_attempts` in the same shape surya produces — including the budget in force, so a caller can tell a request that landed from one that was ignored. Being in-process, they have no thread boundary to cross: `page_options()` reaches them directly. New settings, in config.json, the admin API and the settings page: chandra_max_continuations / chandra_context_tokens and olmocr_max_continuations / olmocr_context_tokens. The UI parses them as numbers rather than falling back on `||`, because 0 is a real setting here (resumption off) and not "use the default". chandra's `detect_repeat_token` moves to being shared rather than copied: it is a port of chandra's upstream util and answers "has this reply fallen into a repetition loop", which is not chandra-specific. 42 tests: tests/test_ocr_resume_shared.py (28, the rules) and tests/test_ocr_resume_engines.py (14, each engine's own ask path). The existing chandra/olmocr/confidence/pod-TLS harnesses were updated to the new ask contract — it changed deliberately, and the stubs say why. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
639a48f8