• Stefy Lanza (nextime / spora )'s avatar
    ocr: make managed Surya-2 vLLM reclaimable by VRAM eviction · 2c419247
    Stefy Lanza (nextime / spora ) authored
    The Surya-2 engine serves its VLM via a managed vLLM subprocess
    (vllm_worker.ensure_service). That subprocess is not a manager-tracked
    model, and the OCR VRAM releaser only tore down the lightweight worker
    pools (killing the HTTP workers) — it never stopped the vLLM holding the
    bulk VRAM (gpu_memory_utilization × card). So on-request eviction saw the
    VRAM as used, found nothing it could free, and (with no CPU fallback) a
    docTR/Paddle load on the same card could OOM.
    
    - vllm_worker: add stop_service_for(cfg, model_path, served_name) (stop ONE
      service, return estimated GB freed) + is_running() helper.
    - OcrManager._release_vram now also stops the managed Surya-2 vLLM
      (_stop_surya_vllm) when surya_serve == "vllm", so the registered external
      releaser actually reclaims its VRAM. Next OCR request re-boots it via
      ensure_service. Targeted by (model_path, served_name) so an LLM vLLM
      instance is left alone.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01Mw2KQiswmD69T45fTfjKwW
    2c419247
vllm_worker.py 11 KB