-
Stefy Lanza (nextime / spora ) authored
Three defects, found from a live batch that reported ~98 pages of about 71 characters, 100 with NO text at all, and 13 reading perfectly. 1. `model_attempts` could not reach anyone. `_page_from_dict` copied the worker's diagnostics onto page.meta BY NAME and knew only `unreadable_trace` and `accents_repaired` — so the field added in v0.3.45 was dropped at the subprocess pipe, and was being hunted for in a live batch that could not have contained it. The same reassembly had already dropped `accents_repaired` once, reporting null for a repair that worked. It now forwards by exclusion: every key that is not an OcrPage field becomes meta. A key-by-key copy means each new diagnostic is born broken and nobody finds out until they need it. 2. An empty page and a 71-character page are the same bug. surya-ocr-2 emits block markup — a category and a bbox per block, BEFORE the text — so a reply stopped at its token cap is incomplete markup, and incomplete markup parses to a few blocks or to none. The cap is surya's `max_tokens_default=2048`, which its vLLM backend never overrides, and the same batch averaged 2,644 generated tokens per request: above it. `ocr.surya_max_tokens` raises it (0 = surya's own), opt-in because a higher cap costs tokens on every page. What is not optional is telling the cases apart, so a page that hit the cap now carries tokens, cap and `truncated: true` with an explanation that it is a cut-off reply and NOT a page the model could not read. Counts are per THREAD, like the trace, so one lane never reports another lane's verdict. 3. The fleet could grow by at most one pod per BOOT. `_should_grow` opened on a boolean held for the whole provision, and a provision is ~12 minutes — so a burst needing four pods took forty minutes to be served by four, whatever max_pods said (measured: 64 pages at 16 per pod got two pods, ready ten minutes apart; reported as "no real scale-out"). The grower also could not see the SIZE of the demand: `min(inflight) >= threshold` answers "is the fleet full?", so the only request it could formulate was "one more", and the callers blocked in acquire() — the actual measure of unmet demand — were invisible. Both are counts now: `_booting` (booting pods ARE capacity, so one demand is never provisioned for twice) and `_waiting`. Growth runs while pods + booting < demand, bounded by max_pods and by the new `scale_up_max_parallel_boots` (default 4) — a real limit, because the pods the demand justifies are the GPUs that start billing together. The original single-flight anticipatory trigger is KEPT beside it: max_inflight_per_pod defaults to unlimited, so requests are often never blocked and a merely saturated fleet would otherwise never grow at all. A prediction gets one pod; measured demand may get several. Found while testing (3): acquire() ignored its own timeout when a sibling process was booting a pod for the same pool — it waited a flat 15s and looped back into the same decision without re-checking the deadline, for as long as the sibling's registry entry survived (up to 15 minutes). A test hung on it. 42 tests across tests/test_runpod_scale_out.py (new) and tests/test_surya_retry_budget.py. Two of my own test harnesses were wrong and are fixed, not worked around: a sys.modules ledger stub that `from codai.api import runpod_ledger` ignores once anything has imported the real module, and a static count of increments against decrements, which cannot express "one increment, four mutually exclusive exits" — replaced by runtime balance tests. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
97b9cdb5