• Stefy Lanza (nextime / spora )'s avatar
    ocr+runpod: a cut-off reply is not an unreadable page; a fleet is not one pod per boot (v0.3.47) · 97b9cdb5
    Stefy Lanza (nextime / spora ) authored
    Three defects, found from a live batch that reported ~98 pages of about 71
    characters, 100 with NO text at all, and 13 reading perfectly.
    
    1. `model_attempts` could not reach anyone. `_page_from_dict` copied the
       worker's diagnostics onto page.meta BY NAME and knew only
       `unreadable_trace` and `accents_repaired` — so the field added in v0.3.45
       was dropped at the subprocess pipe, and was being hunted for in a live
       batch that could not have contained it. The same reassembly had already
       dropped `accents_repaired` once, reporting null for a repair that worked.
       It now forwards by exclusion: every key that is not an OcrPage field
       becomes meta. A key-by-key copy means each new diagnostic is born broken
       and nobody finds out until they need it.
    
    2. An empty page and a 71-character page are the same bug. surya-ocr-2 emits
       block markup — a category and a bbox per block, BEFORE the text — so a
       reply stopped at its token cap is incomplete markup, and incomplete markup
       parses to a few blocks or to none. The cap is surya's
       `max_tokens_default=2048`, which its vLLM backend never overrides, and the
       same batch averaged 2,644 generated tokens per request: above it.
       `ocr.surya_max_tokens` raises it (0 = surya's own), opt-in because a higher
       cap costs tokens on every page. What is not optional is telling the cases
       apart, so a page that hit the cap now carries tokens, cap and
       `truncated: true` with an explanation that it is a cut-off reply and NOT a
       page the model could not read. Counts are per THREAD, like the trace, so
       one lane never reports another lane's verdict.
    
    3. The fleet could grow by at most one pod per BOOT. `_should_grow` opened on
       a boolean held for the whole provision, and a provision is ~12 minutes — so
       a burst needing four pods took forty minutes to be served by four, whatever
       max_pods said (measured: 64 pages at 16 per pod got two pods, ready ten
       minutes apart; reported as "no real scale-out"). The grower also could not
       see the SIZE of the demand: `min(inflight) >= threshold` answers "is the
       fleet full?", so the only request it could formulate was "one more", and
       the callers blocked in acquire() — the actual measure of unmet demand —
       were invisible. Both are counts now: `_booting` (booting pods ARE capacity,
       so one demand is never provisioned for twice) and `_waiting`. Growth runs
       while pods + booting < demand, bounded by max_pods and by the new
       `scale_up_max_parallel_boots` (default 4) — a real limit, because the pods
       the demand justifies are the GPUs that start billing together. The original
       single-flight anticipatory trigger is KEPT beside it: max_inflight_per_pod
       defaults to unlimited, so requests are often never blocked and a merely
       saturated fleet would otherwise never grow at all. A prediction gets one
       pod; measured demand may get several.
    
    Found while testing (3): acquire() ignored its own timeout when a sibling
    process was booting a pod for the same pool — it waited a flat 15s and looped
    back into the same decision without re-checking the deadline, for as long as
    the sibling's registry entry survived (up to 15 minutes). A test hung on it.
    
    42 tests across tests/test_runpod_scale_out.py (new) and
    tests/test_surya_retry_budget.py. Two of my own test harnesses were wrong and
    are fixed, not worked around: a sys.modules ledger stub that `from codai.api
    import runpod_ledger` ignores once anything has imported the real module, and
    a static count of increments against decrements, which cannot express "one
    increment, four mutually exclusive exits" — replaced by runtime balance tests.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    97b9cdb5
ocr_worker.py 48.3 KB