• Stefy Lanza (nextime / spora )'s avatar
    runpod(phase3-5): pod pool + scaler + cost ledger, budgets, global-gate bypass, spillover, stats UI · 7649c9c2
    Stefy Lanza (nextime / spora ) authored
    Completes the RunPod backend for testing.
    
    Pods pool (runpod_worker.py):
    - RunpodPodPool: provision via _select_gpu (cheaper=lowest $/hr, faster=SECURE+
      throughput, across allowed pools, spot-aware) → create_pod → wait_ready →
      vLLM health-check; least-loaded per-request acquire, provision-on-demand up to
      max_pods; background scaler thread tears down idle pods to min after
      idle_timeout_s, keeps min warm, reaps. get_pod_pool registry + pods_status().
    - Budget enforcement at provision: global $/hr rate cap + per-model & global
      cumulative spend caps (rolling period) via the ledger; refuses over budget.
    
    Cost ledger (runpod_ledger.py): persistent per-model + global spend, trailing
    rolling windows (hour/day/week/month), totals() for stats. Final pod cost booked
    on teardown; live pod cost added on top for enforcement/stats (no double count).
    
    Backend (backends/runpod.py): pods mode wired to the pool (per-request acquire/
    release; async stream acquires off the event loop); serverless unchanged. Config
    adds global_cost_limit_usd + global_cost_period.
    
    Global-gate bypass (app.py _model_capacity): runpod models get an effectively
    unlimited LOCAL queue cap — remote GPUs aren't bound by the local concurrency
    gate; RunPod's max_pods/workers are the real limit.
    
    Spillover (app.py): a local model can burst to a RunPod SERVERLESS target when
    local can't serve it — front reverse-proxies to the serverless OpenAI URL, guarded
    to fall back to local on any error. Wired at the direct path for on_no_gpu
    (no engine) + on_concurrency_full (queue full). NOTE partial: broker path,
    streaming-concurrency, pods-target and on_local_error triggers are follow-ups.
    
    UI: dedicated /admin/runpod stats page (pods, live $/hr vs caps, spend by period +
    model) + nav link; Tasks page RunPod box; Settings split into category tabs
    (JS-only, IDs/save untouched) + global spend cap fields.
    
    Verified: imports in-image (8b16914ee24b), ledger + GPU selection (cheaper/faster)
    unit-tested, all templates Jinja-parse. Live RunPod calls (provision/serverless)
    still need an end-to-end test with the user's account.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    7649c9c2
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...