• Stefy Lanza (nextime / spora )'s avatar
    count what each pod has actually served (v0.4.71) · ef6657bd
    Stefy Lanza (nextime / spora ) authored
    The RunPod page showed only the instant: in-flight, KV occupancy, queue depth.
    Nothing said how much work a pod had done, so a healthy pod billed by the hour
    that had served nothing looked exactly like one carrying the fleet.
    
    Each pod now reports the requests it has finished and the prompt/generation
    tokens that went through it, read from the serving engine's own counters and
    shown as a "Served" column, with the live fleet's total beside the pod count.
    
    Two things had to be true for the number to be worth printing:
    
    * prometheus_client renames a Counter. vLLM DECLARES "vllm:num_preemptions"
      and serves "vllm:num_preemptions_total", alongside a "_created" sample whose
      value is a unix timestamp. Keying on the declared name alone silently dropped
      every counter -- so the preemption and prefix figures added in 0.4.69 never
      matched anything, and the page drew a dash that read as "not supported".
      Both spellings are accepted now, and "_created" is never read as a count.
    * vLLM's counters restart at zero with the engine process (model swap, crash,
      OOM kill), so a pod that had served four thousand requests would report
      twelve. Each drop is treated as a restart and the previous reading carried
      forward, which makes the column monotonic for the pod's lifetime.
    
    Also drops the prefix hit rate rather than fix its name: in vLLM 0.27.1 the
    prefix-cache hit counter is registered from the multimodal-cache counter, so a
    rate computed from it is not the prefix hit rate. A wrong number on an
    operator's page is worse than a dash.
    
    Verified against the shipped image's own prometheus_client, not a hand-written
    fixture: the exposition names and the "_created" timestamp in the tests are
    what ghcr.io/nextime/coderai-vllm:latest actually renders.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    ef6657bd
Name
Last commit
Last update
..
__init__.py Loading commit data...
_film_net.py Loading commit data...
_rife_ifnet.py Loading commit data...
app.py Loading commit data...
archive.py Loading commit data...
audio_backends.py Loading commit data...
audio_clean.py Loading commit data...
audio_gen.py Loading commit data...
audio_stems.py Loading commit data...
audiocraft_worker.py Loading commit data...
canary_worker.py Loading commit data...
characters.py Loading commit data...
chunking.py Loading commit data...
colibri_worker.py Loading commit data...
compose.py Loading commit data...
crisperwhisper_worker.py Loading commit data...
custom_pipelines.py Loading commit data...
diarization.py Loading commit data...
ds4_kv_janitor.py Loading commit data...
ds4_worker.py Loading commit data...
embeddings.py Loading commit data...
environments.py Loading commit data...
faceswap.py Loading commit data...
fastlog.py Loading commit data...
h3_worker.py Loading commit data...
host_worker.py Loading commit data...
hub_compat.py Loading commit data...
images.py Loading commit data...
k3_worker.py Loading commit data...
kt_worker.py Loading commit data...
lipsync.py Loading commit data...
log.py Loading commit data...
longcat_worker.py Loading commit data...
loop_watch.py Loading commit data...
lora_archs.py Loading commit data...
loras.py Loading commit data...
melotts_worker.py Loading commit data...
model_test.py Loading commit data...
models_transfer.py Loading commit data...
mux_remote.py Loading commit data...
ocr.py Loading commit data...
parler_worker.py Loading commit data...
pipelines.py Loading commit data...
pod_http.py Loading commit data...
pod_telemetry.py Loading commit data...
pod_tls.py Loading commit data...
podlog.py Loading commit data...
prompt_cache.py Loading commit data...
pyannote_worker.py Loading commit data...
ratelimit.py Loading commit data...
remote_gateway.py Loading commit data...
rerank.py Loading commit data...
runpod_client.py Loading commit data...
runpod_fleet_share.py Loading commit data...
runpod_ledger.py Loading commit data...
runpod_worker.py Loading commit data...
serving_load.py Loading commit data...
spatial.py Loading commit data...
speaker_embeddings.py Loading commit data...
speaker_registry.py Loading commit data...
state.py Loading commit data...
stt_backends.py Loading commit data...
text.py Loading commit data...
thread_pool.py Loading commit data...
transcriptions.py Loading commit data...
tts.py Loading commit data...
tts_backends.py Loading commit data...
uploads.py Loading commit data...
urlutils.py Loading commit data...
video.py Loading commit data...
vllm_worker.py Loading commit data...
voice_clone.py Loading commit data...
voice_convert.py Loading commit data...
vpr_salad_server.py Loading commit data...
xtts_worker.py Loading commit data...