• Stefy Lanza (nextime / spora )'s avatar
    ocr: participate in thermal throttling + VRAM eviction · 3a01becb
    Stefy Lanza (nextime / spora ) authored
    OCR ran outside the shared GPU governors: /v1/ocr(/batch) and run_ocr_step
    call OcrManager.ocr_document directly, bypassing request_model, so OCR
    neither waited on thermal cooldown nor evicted/released VRAM against
    LLM/diffusion models.
    
    - thermal: ocr_document now awaits thermal.wait_until_safe (via to_thread)
      before OCR-ing pages, so OCR honours the same cooldown/front-pause gate as
      every other GPU workload.
    - VRAM (both directions):
      * evict-before-build: when a pool loads its instances, ask the model
        manager to free size*per-instance-VRAM first (_evict_models_for_vram),
        so OCR contends on equal footing instead of OOMing.
      * releaser: OcrManager registers _release_vram(needed_gb) via
        register_external_vram_releaser, tearing down built pools so loading an
        LLM/diffusion model can reclaim OCR VRAM. Pools rebuild lazily next request.
    - engine cleanup() (SYNC, eviction-thread safe) + vram_gb() footprint per
      engine: docTR ~0.7 GB (frees model + empty_cache), subprocess ~2.5 GB
      (kills worker). _Pool tracks all instances and release_sync() tears them down.
    - configure() now tears down stale pools on reconfigure (prompt VRAM free
      instead of GC-deferred).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01Mw2KQiswmD69T45fTfjKwW
    3a01becb
subprocess_engine.py 7.97 KB