• Stefy Lanza (nextime / spora )'s avatar
    one vLLM per card: wait for a starting engine, keep a busy one (v0.4.54) · ed4e2c2a
    Stefy Lanza (nextime / spora ) authored
    Measured on a rented A40 under 128 pages in flight. The engine stopped
    answering. _ensure_service_once treated any service that failed a 3 s health
    probe as gone: it dropped it from the table, without stopping the process,
    and launched another. A second caller during a start did the same, so two
    EngineCores came up one second apart. For twenty minutes vLLM was relaunched
    every few seconds and killed with -9 mid-start. This is also the earlier
    "10.22 of 47 GiB free" crash.
    
    - A service records "starting" until it is ready. Another caller waits for
      it (_StillStarting -> _wait_for_start) instead of launching a duplicate.
    - A live process that misses the probe gets a second 15 s probe. If that
      fails too it is BUSY and is handed out, not replaced.
    - A dead one is stopped properly before it is replaced.
    - The OCR gateway's /health no longer runs placement.any_candidate (which can
      start vLLM) on the event loop. Every surya engine probes it, and it froze a
      pod for up to 408 s. The no-blocking-calls guard now covers
      placement.any_candidate and placement.candidates, and fails on the old code.
    - RunPod 429s are honoured: retryAfter is in seconds (observed 11,871, about
      3.3 h), and no GraphQL call is made until it passes (capped at 6 h).
    Co-Authored-By: 's avatarClaude Opus 5.5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    ed4e2c2a
Name
Last commit
Last update
..
schemas_builtin Loading commit data...
workers Loading commit data...
__init__.py Loading commit data...
base.py Loading commit data...
budget.py Loading commit data...
chandra.py Loading commit data...
detect.py Loading commit data...
doctr.py Loading commit data...
extract.py Loading commit data...
gateway.py Loading commit data...
loopguard.py Loading commit data...
manager.py Loading commit data...
olmocr.py Loading commit data...
paddle.py Loading commit data...
rasterize.py Loading commit data...
resume.py Loading commit data...
schemas.py Loading commit data...
served_model.py Loading commit data...
subprocess_engine.py Loading commit data...
surya.py Loading commit data...
venv_build.py Loading commit data...