• Stefy Lanza (nextime / spora )'s avatar
    vllm: discover the share an engine needs instead of trusting a constant (v0.2.27) · aa085f85
    Stefy Lanza (nextime / spora ) authored
    The 0.60 shipped in 0.2.26 was measured from one model on one card, which is the same
    mistake as the 0.35 before it: a footprint is not knowable up front. It depends on the
    model, the card's size, the context and the multimodal limits vLLM profiles for — and
    the 0.2.26 boot proved the point, since capping max_num_batched_tokens moved the
    footprint only 11.13 -> 10.88 GiB (all of it CUDA graphs, 0.30 -> 0.05) while the
    ~9.4 GiB peak stayed put: vLLM sizes surya-ocr-2's encoder cache at 114,688 tokens and
    profiles it with one VIDEO item at maximum feature size.
    
    So the share is now an opening bid and the launcher measures the rest:
    
    - ensure_service escalates on the KV-exhaustion signature (_KV_EXHAUSTED: "No available
      memory for the cache blocks", a negative "Available KV cache memory", or vLLM's
      free-memory complaint), +0.12 per attempt to at most 4 attempts, capped at the 0.92
      ceiling, asking the evictor for the larger prelaunch figure before each retry.
    - An unrelated failure (missing model, broken venv) is NOT retried — escalating past it
      would burn minutes of boot per attempt for nothing.
    - What booted is persisted to ~/.coderai/vllm_gmu_learned.json keyed by
      model|card-total|ctx|max_num_batched_tokens, so later starts skip rediscovery, a
      different model/card/limit set discovers its own, and a configured share LOWER than
      what is known to work is ignored rather than reintroducing a dead budget.
    - A read-only config dir costs rediscovery, not a failure.
    
    The multimodal limits are deliberately NOT capped by default: doing so would stop that
    instance serving video at all, and the point is that any request type keeps working.
    docs/ocr.md records the measurement and how to opt in via vllm.extra_args.
    
    10 new tests covering escalation, the step, the ceiling, non-retryable failures,
    persistence, scoping and an unwritable store.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    aa085f85
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...