Skip to content
  • Stefy Lanza (nextime / spora )'s avatar
    7649c9c2
    runpod(phase3-5): pod pool + scaler + cost ledger, budgets, global-gate bypass, spillover, stats UI · 7649c9c2
    Stefy Lanza (nextime / spora ) authored
    Completes the RunPod backend for testing.
    
    Pods pool (runpod_worker.py):
    - RunpodPodPool: provision via _select_gpu (cheaper=lowest $/hr, faster=SECURE+
      throughput, across allowed pools, spot-aware) → create_pod → wait_ready →
      vLLM health-check; least-loaded per-request acquire, provision-on-demand up to
      max_pods; background scaler thread tears down idle pods to min after
      idle_timeout_s, keeps min warm, reaps. get_pod_pool registry + pods_status().
    - Budget enforcement at provision: global $/hr rate cap + per-model & global
      cumulative spend caps (rolling period) via the ledger; refuses over budget.
    
    Cost ledger (runpod_ledger.py): persistent per-model + global spend, trailing
    rolling windows (hour/day/week/month), totals() for stats. Final pod cost booked
    on teardown; live pod cost added on top for enforcement/stats (no double count).
    
    Backend (backends/runpod.py): pods mode wired to the poo...
    7649c9c2
    runpod(phase3-5): pod pool + scaler + cost ledger, budgets, global-gate bypass, spillover, stats UI
    Stefy Lanza (nextime / spora ) authored
    Completes the RunPod backend for testing.
    
    Pods pool (runpod_worker.py):
    - RunpodPodPool: provision via _select_gpu (cheaper=lowest $/hr, faster=SECURE+
      throughput, across allowed pools, spot-aware) → create_pod → wait_ready →
      vLLM health-check; least-loaded per-request acquire, provision-on-demand up to
      max_pods; background scaler thread tears down idle pods to min after
      idle_timeout_s, keeps min warm, reaps. get_pod_pool registry + pods_status().
    - Budget enforcement at provision: global $/hr rate cap + per-model & global
      cumulative spend caps (rolling period) via the ledger; refuses over budget.
    
    Cost ledger (runpod_ledger.py): persistent per-model + global spend, trailing
    rolling windows (hour/day/week/month), totals() for stats. Final pod cost booked
    on teardown; live pod cost added on top for enforcement/stats (no double count).
    
    Backend (backends/runpod.py): pods mode wired to the poo...
Loading