• Stefy Lanza (nextime / spora )'s avatar
    ocr: participate in thermal throttling + VRAM eviction · 3a01becb
    Stefy Lanza (nextime / spora ) authored
    OCR ran outside the shared GPU governors: /v1/ocr(/batch) and run_ocr_step
    call OcrManager.ocr_document directly, bypassing request_model, so OCR
    neither waited on thermal cooldown nor evicted/released VRAM against
    LLM/diffusion models.
    
    - thermal: ocr_document now awaits thermal.wait_until_safe (via to_thread)
      before OCR-ing pages, so OCR honours the same cooldown/front-pause gate as
      every other GPU workload.
    - VRAM (both directions):
      * evict-before-build: when a pool loads its instances, ask the model
        manager to free size*per-instance-VRAM first (_evict_models_for_vram),
        so OCR contends on equal footing instead of OOMing.
      * releaser: OcrManager registers _release_vram(needed_gb) via
        register_external_vram_releaser, tearing down built pools so loading an
        LLM/diffusion model can reclaim OCR VRAM. Pools rebuild lazily next request.
    - engine cleanup() (SYNC, eviction-thread safe) + vram_gb() footprint per
      engine: docTR ~0.7 GB (frees model + empty_cache), subprocess ~2.5 GB
      (kills worker). _Pool tracks all instances and release_sync() tears them down.
    - configure() now tears down stale pools on reconfigure (prompt VRAM free
      instead of GC-deferred).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01Mw2KQiswmD69T45fTfjKwW
    3a01becb
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...