runpod: a region and a warm-pod schedule per model, spend over the API (v0.2.82)

Three gaps, all configurable from the model page:

- data_center pins a model's pods to one RunPod region. Precedence is
  volume > model > account: a pod in another region cannot attach the
  volume, so the volume still decides.

- schedule_enabled/start/end/days/tz gate the WARM FLOOR only. Outside
  the window the floor is 0 and the warm pod is released without waiting
  out idle_timeout_s; a request arriving out of hours still cold-starts
  one, exactly as min_pods=0 always has. An end before the start runs
  overnight and belongs to the day it started on. An incomplete or
  unparseable window is treated as no schedule rather than a floor of
  zero, and an unknown time zone falls back to local, not UTC.

- GET /v1/runpod/spend serves the pods, ledger, caps and schedule status
  to an API key marked admin, so an external monitor can poll spend
  without an admin session. Opt-in per token; an ordinary key gets 403.

pools_schedule_status() reports in_window/next_change/effective_min_pods
on /admin/api/runpod/stats and on the new endpoint.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.2.81"
+__version__ = "0.2.82"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/auth.py

@@ -377,6 +377,22 @@ class SessionManager:
                 return True
         return False
 
+    def token_is_admin(self, token: str) -> bool:
+        """True when this API token carries the admin scope.
+
+        Ordinary keys can call the model endpoints but must not read what the
+        install spends; a key is privileged only when it was explicitly marked
+        so, never by default.
+        """
+        if not token:
+            return False
+        auth_data = self._load_auth_data()
+        for t in auth_data.get("tokens", []):
+            stored = t.get("token", "")
+            if stored and hmac.compare_digest(stored, token):
+                return bool(t.get("admin"))
+        return False
+
     def delete_user(self, username: str) -> bool:
         """Delete a user.
         

codai/admin/routes.py

@@ -3089,7 +3089,9 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                   # where the POD gets the weights
                   "source", "hf_repo", "model_url", "quantization",
                   "network_volume_id", "volume_mount_path", "volume_path",
-                  "venv_name", "slim_image"):
+                  "venv_name", "slim_image",
+                  # region + warm-pod schedule
+                  "data_center", "schedule_start", "schedule_end", "schedule_tz"):
             v = src.get(k)
             if isinstance(v, str) and v.strip():
                 rpo[k] = v.strip()
@@ -3103,9 +3105,15 @@ async def api_model_configure(request: Request, username: str = Depends(require_
             ct = None
         if ct:
             rpo["cloud_types"] = ct
+        # schedule_days: list of 0=Mon…6=Sun, or a comma string of those/names
+        sd = src.get("schedule_days")
+        if isinstance(sd, str):
+            sd = [d.strip() for d in sd.split(",") if d.strip()]
+        if isinstance(sd, list) and sd:
+            rpo["schedule_days"] = [str(d).strip().lower() for d in sd if str(d).strip()]
         # bools
         for k in ("allow_spot", "allow_open_pod", "sticky_sessions", "keep_warm",
-                  "model_url_is_tar", "venv_on_volume"):
+                  "model_url_is_tar", "venv_on_volume", "schedule_enabled"):
             if k in src and src.get(k) is not None and src.get(k) != "":
                 sv = src.get(k)
                 rpo[k] = (sv.lower() in ("1", "true", "on", "yes")
@@ -4988,6 +4996,10 @@ async def api_runpod_stats(username: str = Depends(require_admin)):
         ledger = runpod_ledger.totals()
     except Exception:
         ledger = {"per_model": {}, "global": {}}
+    try:
+        schedules = runpod_worker.pools_schedule_status()
+    except Exception:
+        schedules = []
     live_hourly = sum((p.get("hourly_usd") or 0.0) for p in pods if p.get("healthy"))
     live_cost = sum((p.get("live_cost_usd") or 0.0) for p in pods)
     cfg = config_manager.config.runpod if config_manager and config_manager.config else None
@@ -4998,6 +5010,7 @@ async def api_runpod_stats(username: str = Depends(require_admin)):
                 "global_cost_period": cfg.global_cost_period,
                 "enabled": cfg.enabled}
     return {"success": True, "pods": pods, "ledger": ledger, "caps": caps,
+            "schedules": schedules,
             "live_hourly_usd": round(live_hourly, 4),
             "live_uncommitted_usd": round(live_cost, 4)}
 

codai/admin/templates/models.html

@@ -918,6 +918,40 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
             </label>
             <span class="form-hint" style="font-size:11px">No cold-boot wait (image pull + weight download, minutes) — but it bills every hour of every day, including the ones nobody uses it. Off by default.</span>
           </div>
+          <div class="form-row" style="margin:.5rem 0 0">
+            <label style="display:flex;align-items:center;gap:.4rem;cursor:pointer">
+              <input type="checkbox" id="cfg-rp-schedule-enabled" style="width:auto">
+              <span class="form-label" style="margin:0">Only keep it warm during these hours</span>
+            </label>
+            <span class="form-hint" style="font-size:11px">Outside the window the warm pod is terminated and billing stops. Requests still work — one arriving out of hours cold-starts a pod, exactly as it does with no warm pod at all.</span>
+          </div>
+          <div style="display:grid;grid-template-columns:1fr 1fr 1.4fr;gap:.5rem;margin-top:.5rem">
+            <div class="form-row" style="margin:0"><label class="form-label">From</label><input type="time" id="cfg-rp-schedule-start" class="form-input"></div>
+            <div class="form-row" style="margin:0"><label class="form-label">To</label><input type="time" id="cfg-rp-schedule-end" class="form-input"></div>
+            <div class="form-row" style="margin:0">
+              <label class="form-label">Time zone</label>
+              <input type="text" id="cfg-rp-schedule-tz" class="form-input" placeholder="blank = server local, e.g. Europe/Rome">
+            </div>
+          </div>
+          <div class="form-row" style="margin:.4rem 0 0">
+            <label class="form-label">Days</label>
+            <div id="cfg-rp-schedule-days" style="display:flex;gap:.6rem;flex-wrap:wrap">
+              <label style="display:flex;align-items:center;gap:.25rem;font-size:12px"><input type="checkbox" value="0" style="width:auto"> Mon</label>
+              <label style="display:flex;align-items:center;gap:.25rem;font-size:12px"><input type="checkbox" value="1" style="width:auto"> Tue</label>
+              <label style="display:flex;align-items:center;gap:.25rem;font-size:12px"><input type="checkbox" value="2" style="width:auto"> Wed</label>
+              <label style="display:flex;align-items:center;gap:.25rem;font-size:12px"><input type="checkbox" value="3" style="width:auto"> Thu</label>
+              <label style="display:flex;align-items:center;gap:.25rem;font-size:12px"><input type="checkbox" value="4" style="width:auto"> Fri</label>
+              <label style="display:flex;align-items:center;gap:.25rem;font-size:12px"><input type="checkbox" value="5" style="width:auto"> Sat</label>
+              <label style="display:flex;align-items:center;gap:.25rem;font-size:12px"><input type="checkbox" value="6" style="width:auto"> Sun</label>
+            </div>
+            <span class="form-hint" style="font-size:11px">None ticked = every day. An end earlier than the start runs overnight, and belongs to the day it starts on.</span>
+            <span class="form-hint" id="cfg-rp-schedule-status" style="font-size:11px"></span>
+          </div>
+          <div class="form-row" style="margin:.5rem 0 0">
+            <label class="form-label">Region (data centre)</label>
+            <input type="text" id="cfg-rp-data-center" class="form-input" placeholder="blank = the account setting, e.g. EU-RO-1">
+            <span class="form-hint" style="font-size:11px">Pin this model's pods to one RunPod data centre. A network volume still wins: a pod in another region cannot attach it.</span>
+          </div>
           <div style="display:grid;grid-template-columns:1fr 1fr;gap:.5rem;margin-top:.5rem">
             <div class="form-row" style="margin:0">
               <label class="form-label">Boot timeout (s)</label>
@@ -4847,6 +4881,23 @@ function _populateRunpod(s){
   set('cfg-rp-model-url', rp.model_url || '');
   chk('cfg-rp-url-tar', rp.model_url_is_tar);
   chk('cfg-rp-keep-warm', rp.keep_warm);
+  chk('cfg-rp-schedule-enabled', rp.schedule_enabled);
+  set('cfg-rp-schedule-start', rp.schedule_start || '');
+  set('cfg-rp-schedule-end', rp.schedule_end || '');
+  set('cfg-rp-schedule-tz', rp.schedule_tz || '');
+  set('cfg-rp-data-center', rp.data_center || '');
+  (function(){
+    // Days come back normalised to 0=Mon…6=Sun; an empty list means every day,
+    // which is shown as nothing ticked rather than everything.
+    var want = {};
+    (rp.schedule_days || []).forEach(function(d){ want[String(d)] = true; });
+    var box = document.getElementById('cfg-rp-schedule-days');
+    if (box) box.querySelectorAll('input[type=checkbox]').forEach(function(cb){
+      cb.checked = !!want[cb.value];
+    });
+    var st = document.getElementById('cfg-rp-schedule-status');
+    if (st) st.textContent = rp.schedule_status || '';
+  })();
   _updateRunpodSource();
   set('cfg-rp-api-key', rp.api_key || '');
   chk('cfg-rp-open', rp.allow_open_pod);
@@ -5036,6 +5087,15 @@ function _collectRunpod(){
     scale_up_inflight_per_pod: intOrU('cfg-rp-scale-up'),
     max_inflight_per_pod:      intOrU('cfg-rp-max-inflight'),
     sticky_sessions:    document.getElementById('cfg-rp-sticky').checked,
+    schedule_enabled:   document.getElementById('cfg-rp-schedule-enabled').checked,
+    schedule_start:     (val('cfg-rp-schedule-start') || '').trim(),
+    schedule_end:       (val('cfg-rp-schedule-end') || '').trim(),
+    schedule_tz:        (val('cfg-rp-schedule-tz') || '').trim(),
+    schedule_days:      Array.from(
+                          document.querySelectorAll('#cfg-rp-schedule-days input[type=checkbox]'))
+                          .filter(function(cb){ return cb.checked; })
+                          .map(function(cb){ return cb.value; }),
+    data_center:        (val('cfg-rp-data-center') || '').trim(),
   };
   return o;
 }

codai/api/app.py

@@ -754,6 +754,71 @@ async def list_models():
     return ModelList(data=models)
 
 
+def _admin_scoped_token(request) -> str:
+    """The caller's bearer token when it carries the admin scope, else "".
+
+    BearerAuthMiddleware has already established that the request is authorised;
+    this is the narrower question of whether this particular key may read what
+    the install spends.
+    """
+    auth = request.headers.get("authorization", "")
+    if not auth.lower().startswith("bearer "):
+        return ""
+    token = auth[7:].strip()
+    try:
+        # The live SessionManager is the one the admin app built; auth.py only
+        # defines the class. Imported lazily to keep app import order unchanged.
+        from codai.admin.routes import session_manager
+        if session_manager is None:
+            return ""
+        return token if session_manager.token_is_admin(token) else ""
+    except Exception:
+        return ""
+
+
+@app.get("/v1/runpod/spend", summary="RunPod spend and pod status", tags=["Core"])
+async def runpod_spend(request: Request):
+    """What the rented GPUs are costing, for an admin-scoped API key.
+
+    The same figures the admin page shows, served over the normal API so an
+    external monitor can poll them without an admin session. Ordinary keys are
+    refused: a key that may ask a model a question has no business reading the
+    account's spend.
+    """
+    if not _admin_scoped_token(request):
+        raise HTTPException(status_code=403,
+                            detail="this endpoint needs an admin-scoped API key")
+    from codai.api import runpod_worker, runpod_ledger
+    try:
+        pods = runpod_worker.pods_status()
+    except Exception:
+        pods = []
+    try:
+        ledger = runpod_ledger.totals()
+    except Exception:
+        ledger = {"per_model": {}, "global": {}}
+    try:
+        schedules = runpod_worker.pools_schedule_status()
+    except Exception:
+        schedules = []
+    live_hourly = sum((p.get("hourly_usd") or 0.0) for p in pods if p.get("healthy"))
+    live_cost = sum((p.get("live_cost_usd") or 0.0) for p in pods)
+    caps = {}
+    try:
+        from codai.config import config_manager
+        cfg = config_manager.config.runpod if config_manager and config_manager.config else None
+        if cfg is not None:
+            caps = {"global_max_hourly_usd": cfg.global_max_hourly_usd,
+                    "global_cost_limit_usd": cfg.global_cost_limit_usd,
+                    "global_cost_period": cfg.global_cost_period,
+                    "enabled": cfg.enabled}
+    except Exception:
+        caps = {}
+    return {"pods": pods, "ledger": ledger, "caps": caps, "schedules": schedules,
+            "live_hourly_usd": round(live_hourly, 4),
+            "live_uncommitted_usd": round(live_cost, 4)}
+
+
 @app.get("/coderai/capabilities", summary="Server capability document", tags=["Core"])
 async def get_broker_capabilities():
     """Return broker capability metadata."""

codai/frontproxy/admin_data.py

@@ -66,7 +66,8 @@ def register_admin_data(app: FastAPI, config_dir, config=None,
         for t in auth_data.get("tokens", []):
             out.append({"id": t["id"], "name": t["name"], "token": t["token"],
                         "provider": t["provider"], "created_at": t["created_at"],
-                        "last_used": t.get("last_used")})
+                        "last_used": t.get("last_used"),
+                        "admin": bool(t.get("admin"))})
         return JSONResponse(out)
 
     @app.post("/admin/api/tokens", include_in_schema=False)
@@ -80,6 +81,8 @@ def register_admin_data(app: FastAPI, config_dir, config=None,
             data = {}
         name = data.get("name")
         provider = data.get("provider", "openai")
+        # Admin scope lets this key read spend over the served API. Opt-in only.
+        is_admin = bool(data.get("admin"))
         if not name:
             return JSONResponse({"detail": "Token name is required"}, status_code=400)
         import secrets
@@ -94,13 +97,40 @@ def register_admin_data(app: FastAPI, config_dir, config=None,
                 "provider": provider,
                 "created_at": datetime.now(timezone.utc).isoformat(),
                 "last_used": None,
+                "admin": is_admin,
             }
             tokens.append(new_token)
             return True, new_token
 
         nt = sm.update_auth_data(_mut)
         return JSONResponse({"token": nt["token"], "id": nt["id"],
-                             "name": nt["name"], "provider": nt["provider"]})
+                             "name": nt["name"], "provider": nt["provider"],
+                             "admin": bool(nt.get("admin"))})
+
+    @app.patch("/admin/api/tokens/{token_id}", include_in_schema=False)
+    async def _update_token(token_id: int, request: Request):
+        """Grant or revoke a token's admin scope."""
+        _u, err = _admin(request)
+        if err:
+            return err
+        try:
+            data = await request.json()
+        except Exception:
+            data = {}
+        if "admin" not in data:
+            return JSONResponse({"detail": "nothing to change"}, status_code=400)
+        want = bool(data.get("admin"))
+
+        def _mut(auth_data):
+            for t in auth_data.get("tokens", []):
+                if t.get("id") == token_id:
+                    t["admin"] = want
+                    return True, True
+            return False, False
+
+        if not sm.update_auth_data(_mut):
+            return JSONResponse({"detail": "token not found"}, status_code=404)
+        return JSONResponse({"success": True, "id": token_id, "admin": want})
 
     @app.delete("/admin/api/tokens/{token_id}", include_in_schema=False)
     async def _delete_token(token_id: int, request: Request):

docs/runpod.md

@@ -144,6 +144,11 @@ sync.) In `models.json` that is a `backend` pin plus a `runpod` block:
 | `max_pods` | `1` | Hard ceiling on concurrent pods for this model. |
 | `scale_up_inflight_per_pod` | `4` | Add a pod when in-flight requests per pod exceeds this. |
 | `idle_timeout_s` | `300` | Destroy a pod this long after its **last** request. |
+| `data_center` | — | Pin this model's pods to one RunPod data centre (region), e.g. `EU-RO-1`. Blank uses the account-wide setting. A network volume still wins over both: a volume lives in one region and a pod elsewhere cannot attach it. |
+| `schedule_enabled` | `false` | Apply the warm floor (`min_pods`/`keep_warm`) only inside a daily window. Outside it the floor is `0` and the warm pod is terminated, so a model used in office hours stops billing overnight. It does **not** block requests: one arriving out of hours cold-starts a pod exactly as `min_pods: 0` always has. Ignored unless both `schedule_start` and `schedule_end` are set. |
+| `schedule_start` / `schedule_end` | — | `HH:MM`. The start is inclusive, the end exclusive. An end **earlier** than the start runs overnight and belongs to the day it started on, so `22:00`–`06:00` on `fri` covers Friday night into Saturday morning. |
+| `schedule_days` | every day | Which days the window opens on: `0`=Mon … `6`=Sun, or names (`mon,tue`). Empty means every day. |
+| `schedule_tz` | server local | IANA zone for the window, e.g. `Europe/Rome`. An unknown name falls back to the server's local time rather than silently shifting to UTC. |
 | `boot_timeout_s` | `900` | Give up if the pod never exposes its port. The port appears only *after* the image is pulled, so this covers the download — 15 GB at a cold machine's ~25 MB/s is ten minutes. |
 | `load_timeout_s` | `600` | Give up if the server never answers `/v1/models`. |
 
@@ -430,6 +435,7 @@ Admin API:
 | `GET /admin/api/runpod/gpu-types` | Live GPU catalogue + prices |
 | `GET /admin/api/runpod/status` | Pools, pods and their states |
 | `GET /admin/api/runpod/stats` | Cost tiles + per-model spend |
+| `GET /v1/runpod/spend` | The same pods, ledger, caps and schedule status over the **served** API, for an external monitor. Needs an API key marked `admin` (Admin → Tokens); an ordinary key gets `403`. |
 | `GET /admin/api/runpod/pod-logs` | Container / vLLM log for one pod |
 
 ---