Skip to content

C
coderai
Commit cd24ff1f authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files
Options

front: batch page-load reads into one request; bound admin-API GETs 

The Models page fired ~10 /admin/api reads on load. Browsers cap ~6 connections
per host, so during a generation (engine event loop GIL-busy) the proxied reads
pile up, saturate the limit and freeze the whole page behind them.

Two fixes:

1. New front endpoint POST /admin/api/batch: fans out several engine GET reads
   CONCURRENTLY server-side (front's own un-capped pool), each individually
   bounded, and returns them in one response. A slow/blocked sub-call returns an
   error marker without holding up the rest. base.html gains bootstrapPage() +
   pageFetch(): the page batch-loads its first-paint reads in ONE request, and the
   existing loaders transparently serve from the bundle (falling back to a normal
   fetch on miss). models.html now collapses settings/cache-stats/cached-models/
   models/quantize-status/downloads into a single batched call, plus an in-flight
   guard on the downloads poller so ticks can't stack up.

2. Bound /admin/api/* GET proxying (catch-all) with a finite read timeout +
   graceful 503 fallback, so a busy engine can never hang a page forever (HF hub
   lookups and streaming endpoints keep the unbounded path).

Verified: batch unit test (concurrent aggregation, error markers, path filtering,
auth) and node syntax-check of the bootstrap script.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
parent e7001716
Hide whitespace changes
Inline Side-by-side
Showing with 155 additions and 14 deletions
codai/admin/templates/base.html
View file @ cd24ff1f
...@@ -10,6 +10,44 @@ ...@@ -10,6 +10,44 @@
<link href="https://fonts.googleapis.com/css2?family=Plus+Jakarta+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet"> <link href="https://fonts.googleapis.com/css2?family=Plus+Jakarta+Sans:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
<link rel="stylesheet" href="{{ root_path }}/static/admin/style.css"> <link rel="stylesheet" href="{{ root_path }}/static/admin/style.css">
<script>const ROOT_PATH = "{{ root_path }}";</script> <script>const ROOT_PATH = "{{ root_path }}";</script>
<script>
// ── Batched page bootstrap ────────────────────────────────────────────────
// A page's first paint often needs ~10 /admin/api reads. Fired as separate
// fetches they hit the browser's ~6-connections-per-host cap and, during a
// generation (engine busy), pile up and freeze the page. bootstrapPage() asks
// the front to fetch them all CONCURRENTLY server-side and return one response;
// pageFetch() then transparently serves each from that bundle (once), falling
// back to a normal fetch on miss/expiry. Render code keeps using pageFetch like
// fetch — no other change needed.
const _bootCache = new Map();
async function bootstrapPage(paths){
try{
const r = await fetch(ROOT_PATH + '/admin/api/batch', {
method:'POST', headers:{'Content-Type':'application/json'},
body: JSON.stringify({paths})});
if(!r.ok) return;
const d = await r.json();
const now = Date.now();
for(const [p, res] of Object.entries(d.results||{})){
_bootCache.set(p, {res, at: now});
}
}catch{}
}
// Drop-in replacement for fetch(ROOT_PATH + path) on GET reads: returns the
// batched result if present (consumed once, valid ~8s), else a real fetch.
function pageFetch(path){
const hit = _bootCache.get(path);
if(hit && (Date.now() - hit.at) < 8000){
_bootCache.delete(path);
const res = hit.res || {};
const status = res.status || 200;
const body = ('json' in res) ? JSON.stringify(res.json) : (res.text || '');
return Promise.resolve(new Response(body, {status,
headers:{'Content-Type': ('json' in res) ? 'application/json' : 'text/plain'}}));
}
return fetch(ROOT_PATH + path);
}
</script>
{% block head %}{% endblock %} {% block head %}{% endblock %}
</head> </head>
<body> <body>
......
codai/admin/templates/models.html
View file @ cd24ff1f
...@@ -1116,7 +1116,7 @@ let _ds4Enabled = false; // whether the ds4 (DeepSeek V4) engine is enabled gl ...@@ -1116,7 +1116,7 @@ let _ds4Enabled = false; // whether the ds4 (DeepSeek V4) engine is enabled gl
async function loadGlobalSettings(){ async function loadGlobalSettings(){
try{ try{
const r = await fetch(ROOT_PATH + '/admin/api/settings'); const r = await pageFetch('/admin/api/settings');
if(r.ok){ if(r.ok){
const d = await r.json(); const d = await r.json();
_defaultOffloadDir = d.offload?.directory || './offload'; _defaultOffloadDir = d.offload?.directory || './offload';
...@@ -1894,10 +1894,17 @@ async function startDownload(){ ...@@ -1894,10 +1894,17 @@ async function startDownload(){
/* ── active downloads strip ──────────────────────────── */ /* ── active downloads strip ──────────────────────────── */
let _pollTimer = null; let _pollTimer = null;
let _dlInFlight = false;
async function pollDownloads(){ async function pollDownloads(){
// Skip this tick if the previous request is still pending (engine busy) or the
// tab is hidden — firing every 2s while each call takes seconds would stack up
// requests and saturate the browser's per-host connection limit, freezing the page.
if(_dlInFlight) return;
if(document.visibilityState === 'hidden') return;
_dlInFlight = true;
try{ try{
const r = await fetch(ROOT_PATH + '/admin/api/downloads'); const r = await pageFetch('/admin/api/downloads');
if(!r.ok) return; if(!r.ok) return;
const all = await r.json(); const all = await r.json();
const active = all.filter(d=>d.status!=='done'&&d.status!=='error'&&d.status!=='cancelled'); const active = all.filter(d=>d.status!=='done'&&d.status!=='error'&&d.status!=='cancelled');
...@@ -1934,6 +1941,7 @@ async function pollDownloads(){ ...@@ -1934,6 +1941,7 @@ async function pollDownloads(){
</div>`; </div>`;
}).join('<div style="border-top:1px solid var(--border);margin:.3rem 0"></div>'); }).join('<div style="border-top:1px solid var(--border);margin:.3rem 0"></div>');
}catch{} }catch{}
finally{ _dlInFlight = false; }
} }
function startPolling(){ function startPolling(){
...@@ -1946,13 +1954,13 @@ function stopPolling(){ ...@@ -1946,13 +1954,13 @@ function stopPolling(){
if(_pollTimer){ clearInterval(_pollTimer); _pollTimer=null; } if(_pollTimer){ clearInterval(_pollTimer); _pollTimer=null; }
} }
// Single one-shot check on page load; polling only runs while downloads are active. // One-shot download check is driven from the batched bootstrap sequence below
pollDownloads(); // (so it reuses the bundle instead of firing its own request before it lands).
/* ── cache stats & local models ──────────────────────── */ /* ── cache stats & local models ──────────────────────── */
async function loadCacheStats(){ async function loadCacheStats(){
try{ try{
const r = await fetch(ROOT_PATH + '/admin/api/cache-stats'); const r = await pageFetch('/admin/api/cache-stats');
if(!r.ok) return; if(!r.ok) return;
const s = await r.json(); const s = await r.json();
document.getElementById('stat-hf-size').textContent = fmtBytes(s.hf_bytes); document.getElementById('stat-hf-size').textContent = fmtBytes(s.hf_bytes);
...@@ -1981,7 +1989,7 @@ async function _loadEngineInfo(){ ...@@ -1981,7 +1989,7 @@ async function _loadEngineInfo(){
if (er.ok) _engineNames = ((await er.json()).engines || []).map(e => e.name); if (er.ok) _engineNames = ((await er.json()).engines || []).map(e => e.name);
} catch(e) {} } catch(e) {}
try { try {
const d = await (await fetch(ROOT_PATH + '/admin/api/settings')).json(); const d = await (await pageFetch('/admin/api/settings')).json();
if (!_engineNames.length) _engineNames = (d.server && d.server.engine_names) || []; if (!_engineNames.length) _engineNames = (d.server && d.server.engine_names) || [];
_defaultEngine = (d.server && d.server.default_engine) || ''; _defaultEngine = (d.server && d.server.default_engine) || '';
} catch(e) {} } catch(e) {}
...@@ -2443,13 +2451,13 @@ async function loadCachedModels(){ ...@@ -2443,13 +2451,13 @@ async function loadCachedModels(){
const _firstLoad = !hfEl.dataset.loaded; const _firstLoad = !hfEl.dataset.loaded;
if(_firstLoad) hfEl.innerHTML = ggufEl.innerHTML = '<span class="muted small">Loading…</span>'; if(_firstLoad) hfEl.innerHTML = ggufEl.innerHTML = '<span class="muted small">Loading…</span>';
try{ try{
const r = await fetch(ROOT_PATH + '/admin/api/cached-models'); const r = await pageFetch('/admin/api/cached-models');
if(!r.ok) throw new Error((await r.json()).detail||r.statusText); if(!r.ok) throw new Error((await r.json()).detail||r.statusText);
const d = await r.json(); const d = await r.json();
// Only whisper-server RUNNERS (they carry a model_path) belong in this card. // Only whisper-server RUNNERS (they carry a model_path) belong in this card.
// A whisper MODEL config (a .gguf entry with backend=whisper-server but no // A whisper MODEL config (a .gguf entry with backend=whisper-server but no
// model_path) is shown on the GGUF row instead, as that file's model config. // model_path) is shown on the GGUF row instead, as that file's model config.
const whisperModels = (await fetch(ROOT_PATH + '/admin/api/models').then(r=>r.ok?r.json():[])) const whisperModels = (await pageFetch('/admin/api/models').then(r=>r.ok?r.json():[]))
.filter(m => m.backend === 'whisper-server' && m.model_path); .filter(m => m.backend === 'whisper-server' && m.model_path);
// HF models // HF models
...@@ -2695,7 +2703,7 @@ let _quantJobs = {}; ...@@ -2695,7 +2703,7 @@ let _quantJobs = {};
let _quantSig = ''; let _quantSig = '';
async function refreshQuantJobs(){ async function refreshQuantJobs(){
try{ try{
const r = await fetch('/admin/api/quantize-status'); const r = await pageFetch('/admin/api/quantize-status');
_quantJobs = (await r.json()).jobs || {}; _quantJobs = (await r.json()).jobs || {};
}catch(e){ /* keep last-known on transient error */ } }catch(e){ /* keep last-known on transient error */ }
// Signature buckets progress to 5% so we only re-render on meaningful change. // Signature buckets progress to 5% so we only re-render on meaningful change.
...@@ -2729,11 +2737,22 @@ setInterval(async () => { ...@@ -2729,11 +2737,22 @@ setInterval(async () => {
if (sig !== _quantSig) { _quantSig = sig; loadCachedModels(); } if (sig !== _quantSig) { _quantSig = sig; loadCachedModels(); }
}, 5000); }, 5000);
loadGlobalSettings(); // Collapse the first-paint engine reads into ONE batched request (fetched
// Load engine/card info first so the per-model card tags render on the first paint, // concurrently server-side by the front) so the page load doesn't fire ~10 calls
// then re-render once it's available (covers the fetch resolving after the list). // that saturate the browser's per-host connection limit and freeze during a
_loadEngineInfo().then(() => loadCachedModels()); // generation. pageFetch() in these loaders transparently serves from the bundle.
refreshLocal(); (async () => {
await bootstrapPage([
'/admin/api/settings', '/admin/api/cache-stats', '/admin/api/cached-models',
'/admin/api/models', '/admin/api/quantize-status', '/admin/api/downloads',
]);
loadGlobalSettings();
// Engine/card info first so per-model card tags render on first paint, then
// re-render once it resolves (covers the fetch resolving after the list).
_loadEngineInfo().then(() => loadCachedModels());
refreshLocal();
pollDownloads(); // one-shot; reuses the batched bundle
})();
// Toggle the acceleration / TurboQuant sections as model types are checked/unchecked. // Toggle the acceleration / TurboQuant sections as model types are checked/unchecked.
document.querySelectorAll('.cfg-type-cb').forEach(cb => document.querySelectorAll('.cfg-type-cb').forEach(cb =>
......
codai/frontproxy/app.py
View file @ cd24ff1f
...@@ -475,6 +475,57 @@ class FrontProxy: ...@@ -475,6 +475,57 @@ class FrontProxy:
return JSONResponse({"cards": [], "error": str(exc)}) return JSONResponse({"cards": [], "error": str(exc)})
return JSONResponse({"cards": cards}) return JSONResponse({"cards": cards})
async def batch(self, request: Request) -> Response:
"""Fan out several engine GET reads CONCURRENTLY (server-side) and return
them in one response.
A page that needs ~10 ``/admin/api/*`` reads otherwise fires ~10 browser
requests; the browser caps ~6 connections per host, so during a generation
(engine event loop GIL-busy) the stuck requests saturate that limit and the
whole page freezes. Here the front issues all of them at once over its own
(un-capped) connection pool, each individually bounded, so the browser makes
ONE request and a slow/blocked sub-call returns an error marker without
holding up the rest. Body: {"paths": ["/admin/api/models", …]}. Returns
{"results": {path: {status, json|text|error, stale?}}}."""
if not self._has_cred(request):
return JSONResponse({"detail": "Unauthorized"}, status_code=401)
try:
payload = await request.json()
paths = payload.get("paths") or []
except Exception:
paths = []
prim = self.registry.primary()
if prim is None or not isinstance(paths, list) or not paths:
return JSONResponse({"results": {}})
headers = self._filter_headers(request.headers, _DROP_REQ)
_bound = httpx.Timeout(connect=10.0, read=12.0, write=12.0, pool=12.0)
async def _one(p):
if (not isinstance(p, str) or "/admin/api/" not in p
or "stream" in p):
return p, {"status": 400, "error": "unsupported path"}
try:
r = await self._long.request("GET", prim.url + p, headers=headers,
timeout=_bound)
ctype = (r.headers.get("content-type") or "").lower()
out = {"status": r.status_code}
if "application/json" in ctype:
try:
out["json"] = r.json()
except Exception:
out["text"] = r.text
else:
out["text"] = r.text
return p, out
except Exception:
return p, {"status": 503, "error": "engine busy (generating)",
"stale": True}
import asyncio as _asyncio
# Cap fan-out width so a pathological request can't spawn unbounded calls.
pairs = await _asyncio.gather(*[_one(p) for p in paths[:24]])
return JSONResponse({"results": dict(pairs)})
def _canonical_loaded(self, keys) -> list: def _canonical_loaded(self, keys) -> list:
"""Map an engine's loaded-model keys to canonical model ids, deduped. """Map an engine's loaded-model keys to canonical model ids, deduped.
...@@ -796,6 +847,32 @@ class FrontProxy: ...@@ -796,6 +847,32 @@ class FrontProxy:
{"error": "No engine is ready yet (still starting/loading)."}, {"error": "No engine is ready yet (still starting/loading)."},
status_code=503) status_code=503)
# Bound page-data reads with a finite timeout + graceful fallback, so a
# busy engine can never hang a page *forever*. The catch-all otherwise uses
# _long (no read timeout): a handful of stuck /admin/api GETs then saturate
# the browser's ~6-connections-per-host limit and freeze the whole page
# behind them. Excludes endpoints that are legitimately slow (HF hub
# lookups) or streaming (…-stream, SSE), which keep the unbounded path.
if (method == "GET" and "/admin/api/" in path and "stream" not in path
and "/hf-" not in path
and "text/event-stream"
not in (request.headers.get("accept", "").lower())):
try:
r = await self._long.request(
"GET", engine.url + path,
headers=self._filter_headers(request.headers, _DROP_REQ),
params=request.query_params,
timeout=httpx.Timeout(connect=10.0, read=12.0, write=12.0,
pool=12.0))
except Exception:
return JSONResponse(
{"error": "engine busy (generating); data temporarily "
"unavailable", "stale": True}, status_code=503)
return Response(
content=r.content, status_code=r.status_code,
headers=dict(self._filter_headers(r.headers, _DROP_RESP)),
media_type=r.headers.get("content-type"))
# Front-managed generation queue (text only). Acquire a per-model slot # Front-managed generation queue (text only). Acquire a per-model slot
# before dispatching: if all max_instances slots are busy this awaits # before dispatching: if all max_instances slots are busy this awaits
# (showing as "queued" on the Tasks page) until one frees; if too many are # (showing as "queued" on the Tasks page) until one frees; if too many are
...@@ -1062,6 +1139,13 @@ def build_app(config, config_dir=None) -> FastAPI: ...@@ -1062,6 +1139,13 @@ def build_app(config, config_dir=None) -> FastAPI:
async def _gpu_stats(request: Request): async def _gpu_stats(request: Request):
return await front.gpu_stats(request) return await front.gpu_stats(request)
# Aggregate several engine reads into ONE response (concurrent, bounded) so a
# page makes one browser request instead of ~10 — avoids saturating the
# browser's per-host connection limit and freezing during a generation.
@app.post("/admin/api/batch", include_in_schema=False)
async def _batch(request: Request):
return await front.batch(request)
# /v1/models is the union across engines (each engine registers only the models # /v1/models is the union across engines (each engine registers only the models
# the front assigned to it). Registered before the catch-all so it's aggregated. # the front assigned to it). Registered before the catch-all so it's aggregated.
@app.get("/v1/models", include_in_schema=False) @app.get("/v1/models", include_in_schema=False)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment