• Stefy Lanza (nextime / spora )'s avatar
    runpod(phase2): serverless request path + per-model config + no-local-download · 17604ab4
    Stefy Lanza (nextime / spora ) authored
    End-to-end serverless: a model tagged backend:runpod (mode:serverless) proxies
    OpenAI chat/completions to a RunPod serverless endpoint. Runs on a remote GPU,
    holds NO local weights.
    
    - codai/api/runpod_worker.py: RunpodModelConfig + parse_model_runpod (mode/pools/
      criteria/gpu/price/scaling/cost/serverless), serverless_base_url + auth_headers.
    - codai/backends/runpod.py: RunpodBackend — thin OpenAI proxy (chat/stream, Bearer
      auth for serverless), uses_local_vram=False; pods/auto raise a clear "next phase"
      error for now.
    - manager.py: runpod in _ENGINE_BACKENDS + _runpod_name_claims (pin-only) +
      runpod_should_handle + load_model dispatch branch. _model_is_runpod() gate:
      set_default_model skips local download/cache for runpod-served models.
    - router.py + registry.py: runpod is an explicit-pin capability; added to every
      engine's default caps (no local VRAM → any node hosts the proxy; prefers primary).
    - admin routes: whitelist per-model `runpod` + `runpod_spillover` blocks; skip
      local-VRAM estimate for runpod models.
    - models.html: RunPod backend option + full per-model sub-form (mode w/ explainer,
      serverless endpoint, served model, pools/criteria/gpu/price/spot, min/max pods,
      idle timeout, image/disk/port/env, spend limit+period) + prefill/collect.
    
    Serverless verified wired in-image (ef1fd7a3802c). Pods pool/scaler, cost ledger,
    spillover routing, and stats UI are later phases.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01R6iugoNeEgt9PkajKrFyq4
    17604ab4
runpod.py 11 KB