• Stefy Lanza (nextime / spora )'s avatar
    ocr: hand the card over at a page boundary, so OCR takes turns like any model · 4be32ede
    Stefy Lanza (nextime / spora ) authored
    Eviction now does the same clean swap the model manager does for a busy model
    (release_idle_vram waits for a request boundary rather than evicting mid-request).
    OcrManager._release_vram tore pools down immediately, which lost whatever page was
    being recognised — and with it the caller's document. Taking turns on one card means
    the handover waits for the handover point.
    
    - _Pool tracks in-flight pages (threading-guarded: incremented on the event loop, read
      from the manager's eviction thread) and drain_and_release() waits for them to finish
      before tearing down, bounded by ocr.evict_drain_timeout_s (default 60 s) so a wedged
      engine cannot block a model load forever.
    - A rebuild waits out an in-progress drain, instead of re-evicting the model that just
      took its turn and loading into VRAM the drain is about to free.
    - Evicted stays temporary: the next OCR request rebuilds and asks for room again, so
      OCR and models can alternate as many times as the traffic asks for.
    
    4 new tests, including the two that matter: an in-flight page survives an eviction that
    arrives mid-recognition, and three consecutive OCR/model turns each rebuild and
    re-reserve.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    4be32ede
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...