• Stefy Lanza (nextime / spora )'s avatar
    ocr: hand the card over at a page boundary, so OCR takes turns like any model · 4be32ede
    Stefy Lanza (nextime / spora ) authored
    Eviction now does the same clean swap the model manager does for a busy model
    (release_idle_vram waits for a request boundary rather than evicting mid-request).
    OcrManager._release_vram tore pools down immediately, which lost whatever page was
    being recognised — and with it the caller's document. Taking turns on one card means
    the handover waits for the handover point.
    
    - _Pool tracks in-flight pages (threading-guarded: incremented on the event loop, read
      from the manager's eviction thread) and drain_and_release() waits for them to finish
      before tearing down, bounded by ocr.evict_drain_timeout_s (default 60 s) so a wedged
      engine cannot block a model load forever.
    - A rebuild waits out an in-progress drain, instead of re-evicting the model that just
      took its turn and loading into VRAM the drain is about to free.
    - Evicted stays temporary: the next OCR request rebuilds and asks for room again, so
      OCR and models can alternate as many times as the traffic asks for.
    
    4 new tests, including the two that matter: an in-flight page survives an eviction that
    arrives mid-recognition, and three consecutive OCR/model turns each rebuild and
    re-reserve.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    4be32ede
routes.py 238 KB