• Stefy Lanza (nextime / spora )'s avatar
    ocr: size a VLM engine's vLLM against free VRAM, and never answer 503 (v0.2.25) · b1004ebd
    Stefy Lanza (nextime / spora ) authored
    surya-on-vLLM was failing every ~20 minutes in production with
    "Available KV cache memory: -2.81 GiB", and the cause was the 0.35 share itself.
    vLLM's --gpu-memory-utilization is a fraction of the card's TOTAL memory and every
    other process' allocation counts against it, so with 3.6 GB of resident bge-m3 on a
    24 GB 3090 a 0.35 budget left surya about 5 GB — less than its weights (1.37) plus
    activation peak at max_model_len=18432 plus CUDA graphs (0.30). Each attempt burned
    ~3 minutes of boot and profiling, then died, and the next OCR request started over.
    
    - vlm_gpu_memory_utilization is now the engine's OWN share and is translated at launch
      against what else holds the card (effective_gmu): 0.35 with 3.6 GB resident becomes
      0.50, i.e. a 12 GB budget of which 8.4 GB is the engine's. Capped at 0.92 so the
      driver keeps its room.
    - The evict-first demand (prelaunch_free_gb) now covers the share plus that headroom,
      so a genuinely crowded card gets models evicted rather than the share clamped.
    - stop_service_for reports what the instance actually reserved, recorded at launch,
      instead of the caller's share — under-reporting sent the evictor hunting for VRAM
      that was already free.
    - Stopping the managed vLLM is no longer gated on the engine's current serve mode: a
      mode changed while a service was up leaked its share of the card until a restart.
    - ocr.fallback_engines (default "auto"): an engine that cannot serve hands the document
      to the other enabled engines instead of returning 503, fewest-ways-to-fail first. The
      response's "engine" field names whoever actually read the pages. This also covers a
      pool evicted mid-document — the failed pages are re-read rather than failing.
    
    21 new tests; the two older vLLM-worker tests now patch the package attribute as well
    as sys.modules, without which the fake worker silently did not apply.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    b1004ebd
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...