• Stefy Lanza (nextime / spora )'s avatar
    ocr: size a VLM engine's vLLM against free VRAM, and never answer 503 (v0.2.25) · b1004ebd
    Stefy Lanza (nextime / spora ) authored
    surya-on-vLLM was failing every ~20 minutes in production with
    "Available KV cache memory: -2.81 GiB", and the cause was the 0.35 share itself.
    vLLM's --gpu-memory-utilization is a fraction of the card's TOTAL memory and every
    other process' allocation counts against it, so with 3.6 GB of resident bge-m3 on a
    24 GB 3090 a 0.35 budget left surya about 5 GB — less than its weights (1.37) plus
    activation peak at max_model_len=18432 plus CUDA graphs (0.30). Each attempt burned
    ~3 minutes of boot and profiling, then died, and the next OCR request started over.
    
    - vlm_gpu_memory_utilization is now the engine's OWN share and is translated at launch
      against what else holds the card (effective_gmu): 0.35 with 3.6 GB resident becomes
      0.50, i.e. a 12 GB budget of which 8.4 GB is the engine's. Capped at 0.92 so the
      driver keeps its room.
    - The evict-first demand (prelaunch_free_gb) now covers the share plus that headroom,
      so a genuinely crowded card gets models evicted rather than the share clamped.
    - stop_service_for reports what the instance actually reserved, recorded at launch,
      instead of the caller's share — under-reporting sent the evictor hunting for VRAM
      that was already free.
    - Stopping the managed vLLM is no longer gated on the engine's current serve mode: a
      mode changed while a service was up leaked its share of the card until a restart.
    - ocr.fallback_engines (default "auto"): an engine that cannot serve hands the document
      to the other enabled engines instead of returning 503, fewest-ways-to-fail first. The
      response's "engine" field names whoever actually read the pages. This also covers a
      pool evicted mid-document — the failed pages are re-read rather than failing.
    
    21 new tests; the two older vLLM-worker tests now patch the package attribute as well
    as sys.modules, without which the fake worker silently did not apply.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    b1004ebd
Name
Last commit
Last update
..
__init__.py Loading commit data...
_film_net.py Loading commit data...
_rife_ifnet.py Loading commit data...
app.py Loading commit data...
archive.py Loading commit data...
audio_backends.py Loading commit data...
audio_clean.py Loading commit data...
audio_gen.py Loading commit data...
audio_stems.py Loading commit data...
audiocraft_worker.py Loading commit data...
canary_worker.py Loading commit data...
characters.py Loading commit data...
colibri_worker.py Loading commit data...
compose.py Loading commit data...
crisperwhisper_worker.py Loading commit data...
custom_pipelines.py Loading commit data...
diarization.py Loading commit data...
ds4_kv_janitor.py Loading commit data...
ds4_worker.py Loading commit data...
embeddings.py Loading commit data...
environments.py Loading commit data...
faceswap.py Loading commit data...
fastlog.py Loading commit data...
h3_worker.py Loading commit data...
host_worker.py Loading commit data...
hub_compat.py Loading commit data...
images.py Loading commit data...
k3_worker.py Loading commit data...
kt_worker.py Loading commit data...
lipsync.py Loading commit data...
log.py Loading commit data...
lora_archs.py Loading commit data...
loras.py Loading commit data...
melotts_worker.py Loading commit data...
model_test.py Loading commit data...
models_transfer.py Loading commit data...
mux_remote.py Loading commit data...
ocr.py Loading commit data...
parler_worker.py Loading commit data...
pipelines.py Loading commit data...
pod_http.py Loading commit data...
pod_tls.py Loading commit data...
prompt_cache.py Loading commit data...
pyannote_worker.py Loading commit data...
ratelimit.py Loading commit data...
remote_gateway.py Loading commit data...
rerank.py Loading commit data...
runpod_client.py Loading commit data...
runpod_ledger.py Loading commit data...
runpod_worker.py Loading commit data...
spatial.py Loading commit data...
speaker_embeddings.py Loading commit data...
speaker_registry.py Loading commit data...
state.py Loading commit data...
stt_backends.py Loading commit data...
text.py Loading commit data...
transcriptions.py Loading commit data...
tts.py Loading commit data...
tts_backends.py Loading commit data...
uploads.py Loading commit data...
urlutils.py Loading commit data...
video.py Loading commit data...
vllm_worker.py Loading commit data...
voice_clone.py Loading commit data...
voice_convert.py Loading commit data...
vpr_salad_server.py Loading commit data...
xtts_worker.py Loading commit data...