• Stefy Lanza (nextime / spora )'s avatar
    count what each pod has actually served (v0.4.71) · ef6657bd
    Stefy Lanza (nextime / spora ) authored
    The RunPod page showed only the instant: in-flight, KV occupancy, queue depth.
    Nothing said how much work a pod had done, so a healthy pod billed by the hour
    that had served nothing looked exactly like one carrying the fleet.
    
    Each pod now reports the requests it has finished and the prompt/generation
    tokens that went through it, read from the serving engine's own counters and
    shown as a "Served" column, with the live fleet's total beside the pod count.
    
    Two things had to be true for the number to be worth printing:
    
    * prometheus_client renames a Counter. vLLM DECLARES "vllm:num_preemptions"
      and serves "vllm:num_preemptions_total", alongside a "_created" sample whose
      value is a unix timestamp. Keying on the declared name alone silently dropped
      every counter -- so the preemption and prefix figures added in 0.4.69 never
      matched anything, and the page drew a dash that read as "not supported".
      Both spellings are accepted now, and "_created" is never read as a count.
    * vLLM's counters restart at zero with the engine process (model swap, crash,
      OOM kill), so a pod that had served four thousand requests would report
      twelve. Each drop is treated as a restart and the previous reading carried
      forward, which makes the column monotonic for the pod's lifetime.
    
    Also drops the prefix hit rate rather than fix its name: in vLLM 0.27.1 the
    prefix-cache hit counter is registered from the multimodal-cache counter, so a
    rate computed from it is not the prefix hit rate. A wrong number on an
    operator's page is worse than a dash.
    
    Verified against the shipped image's own prometheus_client, not a hand-written
    fixture: the exposition names and the "_created" timestamp in the tests are
    what ghcr.io/nextime/coderai-vllm:latest actually renders.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    ef6657bd
test_pod_gpu_telemetry.py 11.9 KB