• Stefy Lanza (nextime / spora )'s avatar
    ocr: size a VLM engine's vLLM against free VRAM, and never answer 503 (v0.2.25) · b1004ebd
    Stefy Lanza (nextime / spora ) authored
    surya-on-vLLM was failing every ~20 minutes in production with
    "Available KV cache memory: -2.81 GiB", and the cause was the 0.35 share itself.
    vLLM's --gpu-memory-utilization is a fraction of the card's TOTAL memory and every
    other process' allocation counts against it, so with 3.6 GB of resident bge-m3 on a
    24 GB 3090 a 0.35 budget left surya about 5 GB — less than its weights (1.37) plus
    activation peak at max_model_len=18432 plus CUDA graphs (0.30). Each attempt burned
    ~3 minutes of boot and profiling, then died, and the next OCR request started over.
    
    - vlm_gpu_memory_utilization is now the engine's OWN share and is translated at launch
      against what else holds the card (effective_gmu): 0.35 with 3.6 GB resident becomes
      0.50, i.e. a 12 GB budget of which 8.4 GB is the engine's. Capped at 0.92 so the
      driver keeps its room.
    - The evict-first demand (prelaunch_free_gb) now covers the share plus that headroom,
      so a genuinely crowded card gets models evicted rather than the share clamped.
    - stop_service_for reports what the instance actually reserved, recorded at launch,
      instead of the caller's share — under-reporting sent the evictor hunting for VRAM
      that was already free.
    - Stopping the managed vLLM is no longer gated on the engine's current serve mode: a
      mode changed while a service was up leaked its share of the card until a restart.
    - ocr.fallback_engines (default "auto"): an engine that cannot serve hands the document
      to the other enabled engines instead of returning 503, fewest-ways-to-fail first. The
      response's "engine" field names whoever actually read the pages. This also covers a
      pool evicted mid-document — the failed pages are re-read rather than failing.
    
    21 new tests; the two older vLLM-worker tests now patch the package attribute as well
    as sys.modules, without which the fake worker silently did not apply.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    b1004ebd
manager.py 21.6 KB