• Stefy Lanza (nextime / spora )'s avatar
    a control plane that rented pods to deliver a teardown (v0.3.87) · 7d0f24aa
    Stefy Lanza (nextime / spora ) authored
    Seven items from a production field report. Three of them were one cause.
    
    /v1/runpod is NOT in the remote gateway's local list. It carries a `model`
    field like an inference request, so the gateway read it as one, resolved the
    model to its pod, and FORWARDED the control-plane call there. Asking for six
    pods back rented a seventh to ask on; the pod has no runpod configuration of
    its own, so it answered
    
        404 {"detail": "'datalab-to/surya-ocr-2' has no runpod block"}
    
    which reads exactly like the orchestrator refusing a name it should know. The
    same call blocked for minutes while a pod booted to receive it. An operator
    could not turn off $3.54/hr from the API. Reproduced here while testing the
    fix: two teardown requests rented two A40s, which is the clearest possible
    statement of the bug.
    
    surya answered 200 with empty pages for a whole document. Three failures in a
    row, each of which had to be fixed:
    
      * its client catches the HTTP error itself and returns a generation with no
        tokens, so the lease check in recognize_image — which only ran in `except`
        — never ran, and the pod address was never re-resolved;
      * the repetition detector was handed that empty string, found no progress in
        it and re-ran the page at a rising temperature, four full paid generations
        per page, `tokens: [0,0,0,0]`;
      * and the page came back as read. We had already measured the ink and said
        there IS something on it; the judgement never reached the response, so a
        client treating an empty read as an unreadable scan consumed roughly a
        thousand documents from its queue and wrote them off.
    
    A page with ink and no text is now `unread`: it goes to the fallback engine
    like any other degraded page, it is listed in `unread_pages`, and a document
    whose every inked page came back empty is no longer a 200 (`ocr.unread_status`,
    502 by default; a partially read document stays 200 with the list).
    
    `bge-m3` and `BAAI/bge-m3` each rented their own A40 — the pool key was
    whichever name the request carried. Keyed on the model ENTRY now, so every name
    for one set of weights lands on one pool.
    
    A client read timeout cannot bound an OCR call: it times each READ, so a
    response that dribbles never trips it and one stayed open nineteen minutes
    against timeout=420. `ocr.request_timeout_s` is the server-side stopwatch.
    
    Chunking: `it-legal` accepts a decorated heading (`**DISPOSITIVO**`) where the
    implementation it ports does not, which splits one more section and leaves a
    short trailing chunk. That was true and undocumented — the table printed the
    strict pattern. Documented now, with `it-legal-strict` named as the
    byte-identical one for re-chunking an existing index. `min_chunk_chars` folds a
    runt chunk into its predecessor, off by default because merging moves a
    boundary, and never silently: the absorbed section label is reported, since
    `conclusioni` is what a party asked for and `dispositivo` is what the court
    decided.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    7d0f24aa
Name
Last commit
Last update
..
schemas_builtin Loading commit data...
workers Loading commit data...
__init__.py Loading commit data...
base.py Loading commit data...
budget.py Loading commit data...
chandra.py Loading commit data...
detect.py Loading commit data...
doctr.py Loading commit data...
extract.py Loading commit data...
gateway.py Loading commit data...
manager.py Loading commit data...
olmocr.py Loading commit data...
paddle.py Loading commit data...
rasterize.py Loading commit data...
resume.py Loading commit data...
schemas.py Loading commit data...
served_model.py Loading commit data...
subprocess_engine.py Loading commit data...
surya.py Loading commit data...
venv_build.py Loading commit data...