-
Stefy Lanza (nextime / spora ) authored
The RunPod page showed only the instant: in-flight, KV occupancy, queue depth. Nothing said how much work a pod had done, so a healthy pod billed by the hour that had served nothing looked exactly like one carrying the fleet. Each pod now reports the requests it has finished and the prompt/generation tokens that went through it, read from the serving engine's own counters and shown as a "Served" column, with the live fleet's total beside the pod count. Two things had to be true for the number to be worth printing: * prometheus_client renames a Counter. vLLM DECLARES "vllm:num_preemptions" and serves "vllm:num_preemptions_total", alongside a "_created" sample whose value is a unix timestamp. Keying on the declared name alone silently dropped every counter -- so the preemption and prefix figures added in 0.4.69 never matched anything, and the page drew a dash that read as "not supported". Both spellings are accepted now, and "_created" is never read as a count. * vLLM's counters restart at zero with the engine process (model swap, crash, OOM kill), so a pod that had served four thousand requests would report twelve. Each drop is treated as a restart and the previous reading carried forward, which makes the column monotonic for the pod's lifetime. Also drops the prefix hit rate rather than fix its name: in vLLM 0.27.1 the prefix-cache hit counter is registered from the multimodal-cache counter, so a rate computed from it is not the prefix hit rate. A wrong number on an operator's page is worse than a dash. Verified against the shipped image's own prometheus_client, not a hand-written fixture: the exposition names and the "_created" timestamp in the tests are what ghcr.io/nextime/coderai-vllm:latest actually renders. Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
ef6657bd