• Stefy Lanza (nextime / spora )'s avatar
    vllm: size the share down too, from the cache a successful boot reports (v0.2.28) · bb2389ff
    Stefy Lanza (nextime / spora ) authored
    Escalation alone only ratchets upward, so an opening bid that is too generous was
    permanent: 0.60 of a 24 GB card for an engine that needs 0.25 is 8 GB no other model
    could have. A successful boot states how much of the budget was left for the cache,
    and `footprint = budget - cache` is the only direct measurement of an engine's real
    cost there is — so it is now read back and used.
    
    - _pump_logs captures "Available KV cache memory: N GiB" as it STREAMS. By the time the
      server answers its health check that line has scrolled out of the 80-line tail behind
      vLLM's route listing, so it cannot be recovered afterwards.
    - On a boot with clear slack (> 1 GiB beyond the ~2 GiB target), a smaller share is
      computed from the measured footprint and recorded for the next start.
    - The learned store is now a record per key, {"ok": booted, "failed": too small}, and
      the downward half may never propose a share at or below "failed" + a step. That floor
      is what stops it oscillating between a budget that works and one that does not.
    - Over-provisioning now costs one boot instead of the life of the process, and the flat
      float format the previous version wrote still loads.
    
    6 more tests (shrinking, the tight-fit no-op, the anti-oscillation floor, a missing
    cache figure, the old format, and the footprint round-trip): 16 in the file, 255 across
    the OCR/gate/cluster/remote-execution/placement files together.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    bb2389ff
Name
Last commit
Last update
..
test_admin_auth.py Loading commit data...
test_admin_remote_ui.py Loading commit data...
test_audio_ml_backends.py Loading commit data...
test_audio_ml_endpoints.py Loading commit data...
test_broker_config.py Loading commit data...
test_broker_dispatch.py Loading commit data...
test_broker_protocol.py Loading commit data...
test_broker_streaming.py Loading commit data...
test_burst_placement.py Loading commit data...
test_compose.py Loading commit data...
test_front_ocr_swap_gate.py Loading commit data...
test_manual_multimodal_test_client.py Loading commit data...
test_ocr_olmocr.py Loading commit data...
test_ocr_vram_share_and_fallback.py Loading commit data...
test_remote_execution.py Loading commit data...
test_studio_composed_surfaces.py Loading commit data...
test_upgrade_launcher.py Loading commit data...
test_vllm_share_self_tuning.py Loading commit data...
test_whisper_server_local_models.py Loading commit data...