• Stefy Lanza (nextime / spora )'s avatar
    vllm: size the share down too, from the cache a successful boot reports (v0.2.28) · bb2389ff
    Stefy Lanza (nextime / spora ) authored
    Escalation alone only ratchets upward, so an opening bid that is too generous was
    permanent: 0.60 of a 24 GB card for an engine that needs 0.25 is 8 GB no other model
    could have. A successful boot states how much of the budget was left for the cache,
    and `footprint = budget - cache` is the only direct measurement of an engine's real
    cost there is — so it is now read back and used.
    
    - _pump_logs captures "Available KV cache memory: N GiB" as it STREAMS. By the time the
      server answers its health check that line has scrolled out of the 80-line tail behind
      vLLM's route listing, so it cannot be recovered afterwards.
    - On a boot with clear slack (> 1 GiB beyond the ~2 GiB target), a smaller share is
      computed from the measured footprint and recorded for the next start.
    - The learned store is now a record per key, {"ok": booted, "failed": too small}, and
      the downward half may never propose a share at or below "failed" + a step. That floor
      is what stops it oscillating between a budget that works and one that does not.
    - Over-provisioning now costs one boot instead of the life of the process, and the flat
      float format the previous version wrote still loads.
    
    6 more tests (shrinking, the tight-fit no-op, the anti-oscillation floor, a missing
    cache figure, the old format, and the footprint round-trip): 16 in the file, 255 across
    the OCR/gate/cluster/remote-execution/placement files together.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    bb2389ff
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...