• Stefy Lanza (nextime / spora )'s avatar
    runpod(phase3-5): pod pool + scaler + cost ledger, budgets, global-gate bypass, spillover, stats UI · 7649c9c2
    Stefy Lanza (nextime / spora ) authored
    Completes the RunPod backend for testing.
    
    Pods pool (runpod_worker.py):
    - RunpodPodPool: provision via _select_gpu (cheaper=lowest $/hr, faster=SECURE+
      throughput, across allowed pools, spot-aware) → create_pod → wait_ready →
      vLLM health-check; least-loaded per-request acquire, provision-on-demand up to
      max_pods; background scaler thread tears down idle pods to min after
      idle_timeout_s, keeps min warm, reaps. get_pod_pool registry + pods_status().
    - Budget enforcement at provision: global $/hr rate cap + per-model & global
      cumulative spend caps (rolling period) via the ledger; refuses over budget.
    
    Cost ledger (runpod_ledger.py): persistent per-model + global spend, trailing
    rolling windows (hour/day/week/month), totals() for stats. Final pod cost booked
    on teardown; live pod cost added on top for enforcement/stats (no double count).
    
    Backend (backends/runpod.py): pods mode wired to the pool (per-request acquire/
    release; async stream acquires off the event loop); serverless unchanged. Config
    adds global_cost_limit_usd + global_cost_period.
    
    Global-gate bypass (app.py _model_capacity): runpod models get an effectively
    unlimited LOCAL queue cap — remote GPUs aren't bound by the local concurrency
    gate; RunPod's max_pods/workers are the real limit.
    
    Spillover (app.py): a local model can burst to a RunPod SERVERLESS target when
    local can't serve it — front reverse-proxies to the serverless OpenAI URL, guarded
    to fall back to local on any error. Wired at the direct path for on_no_gpu
    (no engine) + on_concurrency_full (queue full). NOTE partial: broker path,
    streaming-concurrency, pods-target and on_local_error triggers are follow-ups.
    
    UI: dedicated /admin/runpod stats page (pods, live $/hr vs caps, spend by period +
    model) + nav link; Tasks page RunPod box; Settings split into category tabs
    (JS-only, IDs/save untouched) + global spend cap fields.
    
    Verified: imports in-image (8b16914ee24b), ledger + GPU selection (cheaper/faster)
    unit-tested, all templates Jinja-parse. Live RunPod calls (provision/serverless)
    still need an end-to-end test with the user's account.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    7649c9c2
runpod_ledger.py 4.29 KB