• Stefy Lanza (nextime / spora )'s avatar
    vllm: discover the share an engine needs instead of trusting a constant (v0.2.27) · aa085f85
    Stefy Lanza (nextime / spora ) authored
    The 0.60 shipped in 0.2.26 was measured from one model on one card, which is the same
    mistake as the 0.35 before it: a footprint is not knowable up front. It depends on the
    model, the card's size, the context and the multimodal limits vLLM profiles for — and
    the 0.2.26 boot proved the point, since capping max_num_batched_tokens moved the
    footprint only 11.13 -> 10.88 GiB (all of it CUDA graphs, 0.30 -> 0.05) while the
    ~9.4 GiB peak stayed put: vLLM sizes surya-ocr-2's encoder cache at 114,688 tokens and
    profiles it with one VIDEO item at maximum feature size.
    
    So the share is now an opening bid and the launcher measures the rest:
    
    - ensure_service escalates on the KV-exhaustion signature (_KV_EXHAUSTED: "No available
      memory for the cache blocks", a negative "Available KV cache memory", or vLLM's
      free-memory complaint), +0.12 per attempt to at most 4 attempts, capped at the 0.92
      ceiling, asking the evictor for the larger prelaunch figure before each retry.
    - An unrelated failure (missing model, broken venv) is NOT retried — escalating past it
      would burn minutes of boot per attempt for nothing.
    - What booted is persisted to ~/.coderai/vllm_gmu_learned.json keyed by
      model|card-total|ctx|max_num_batched_tokens, so later starts skip rediscovery, a
      different model/card/limit set discovers its own, and a configured share LOWER than
      what is known to work is ignored rather than reintroducing a dead budget.
    - A read-only config dir costs rediscovery, not a failure.
    
    The multimodal limits are deliberately NOT capped by default: doing so would stop that
    instance serving video at all, and the point is that any request type keeps working.
    docs/ocr.md records the measurement and how to opt in via vllm.extra_args.
    
    10 new tests covering escalation, the step, the ceiling, non-retryable failures,
    persistence, scoping and an unwritable store.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    aa085f85
vllm_worker.py 31.5 KB