- 23 Jun, 2026 8 commits
-
-
Stefy Lanza (nextime / spora ) authoredThe streaming content gate only ran when the request carried a tools array. But the AISBF relay can drop tools (as it does max_tokens), so a gemma model still emitting <|tool_call>call:NAME{…} from its system prompt had that markup streamed straight to the client as content — the spill that poisoned history and fed the tool-call loop. _gate_tool_content now: * ALWAYS withholds the unambiguous native special-token markers (<|tool_call>, <|tool_response>, DeepSeek DSML) and their cross-chunk partials, even with no tools declared; * gates the ambiguous <tool>…</tool> XML form only when tools are present; * gates the gemma-4 call:NAME{…} form by the per-model gemma_tool_parser mode (off: never; full: always; restricted: only declared tool names), so legit call:foo{…} prose/code is streamed instead of withheld+dropped. The gate is now invoked for every stream (not just tools-present), and the final flush likewise, with the mode/tool context resolved once per request. Bump version to 0.1.2. Co-Authored-By:Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
-
Stefy Lanza (nextime / spora ) authored
The gemma-4 native call:NAME{…} heuristic could eat legitimate text: the content stripper removed ANY call:/response:{…} span with no tool-name restriction, so a coding reply containing e.g. call:foo{…} in prose or a snippet got silently deleted; and with no tools declared the parser matched any call:word{. Add a 3-way mode, resolved per-model (models.json "gemma_tool_parser") over the global models.gemma_tool_parser (default "restricted"): * full - parse & strip every call:/response: span (old behavior) * restricted - only when NAME is a declared tool; legit call:foo{…} prose/code is preserved, real declared-tool calls still parse & strip * off - disable the gemma heuristic entirely (bigger models that emit standard structured tool calls) resolve_gemma_tool_mode() + a centralized strip_gemma_native() now gate both the parse (GemmaParser, via the dispatcher) and the strip (ModelParserAdapter and the streaming ToolCallParser, which now stashes the declared tool names from extract_tool_calls so 'restricted' works on the streaming finalizer too). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
A model can get stuck re-issuing the same tool call with the same arguments when each attempt fails (e.g. memory replace old_text=X → "No entry matched" on the wrong memory tier) — or when the call spills as un-parsed call:NAME{…} markup into assistant content. Each brokered request carries the full history, so coderai can see the repetition the agent's own tool-runner didn't break. _detect_tool_loop scans the recent history for a (tool, arguments) signature repeated >= threshold times where those attempts failed (per the tool result) or spilled as markup, and injects one system reminder before generation telling the model to stop repeating the call. Covers both structured tool_calls and spilled gemma-style markup. Configurable on every model: per-model models.json tool_loop_guard / tool_loop_repeats override the global models.tool_loop_* (default on, repeats=3); repeats<=0 disables. Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
ChatMessage.convert_content_array_to_string ran at parse time (mode=before) and unconditionally joined any multipart content list into a string, replacing each image_url part with the literal text "[image_url content]". That destroyed the image before text.py's vision pipeline (_vision_ok / _normalize_vision_content, which the backend's mmproj/MTMDChatHandler consume) ever saw it — so a vision model (e.g. lisa = Gemma-4 + mmproj) received only text and answered "no image attached", and agents looped retrying. text.py's end-to-end vision handling was effectively dead code because content was always pre-stringified. Now flatten only TEXT-ONLY multipart arrays (the KiloCode case); preserve the list whenever it carries an image_url (or any non-text part) so the multimodal backend receives the image. Non-vision models still degrade to the placeholder downstream (text.py:1439), and text-only/plain-string paths are unchanged. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The broker streaming relay counted tokens into engine.active["step"] but never set ["rate"], so _merge_engine_tasks (which overlays both onto the running task) showed token progression with a frozen 0 speed. Compute tok/s from the first streamed token (so the model-load/queue wait doesn't drag the average down) and publish it as m["rate"], mirroring the engine-side text path. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Ships the broker load-status / max_tokens fixes, the run-as-invoking-user default (+ --root opt-out), nginx console silencing, and the expanded installer help. --versioned builds now tag coderai:full_all_0.1.1. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Three operational fixes for the distributable image: run_oci.sh: default the container to the invoking user (uid:gid, SUDO_UID-aware) instead of the image's root default, with a new --root opt-out. The image has no USER directive and supervisord sets no user=, so a run without --user created root-owned dirs (logs, coderai-tmp, hf cache) in the bind-mounted /cache; a later --user run then couldn't write them ("cannot write /cache/logs ... logging to stdout only"). Running as the user by default makes a fresh install safe regardless of run order. --root restores the old behaviour (and throwaway config copy) for shared root-managed data dirs. nginx.conf: error_log crit (was info) + access_log off. nginx's [notice] startup/worker lines and per-request access logs were piped to the container console by supervisord and buried coderai's own output on attach / docker logs. Real failures (crit/alert/emerg) still surface. install.sh: print an extensive post-install guide (quick start, --local + --map, where data lives, the run-as-you default, file logging, docker logs/stop, --help). Co-Authored-By: -
Stefy Lanza (nextime / spora ) authored
The engine-side "Waiting for model reply..." content injection ran inside the GIL-blocked engine handler during model load, so it never streamed live -- it only got concatenated into the final reply and polluted the output. Removed it. Load-status now lives in the responsive front-proxy/broker layer: - SSE: broker_execute_stream yields a NON-content chunk (empty delta.content + x_queue_info status) during each not-ready wait (engine down / starting / model loading), so a watching client sees progress without reply pollution. - out-of-band: the existing broker `pending` keepalives are kept. Both are gated by a single flag, resolved by _load_status_enabled(model): - global models.load_status_updates (default True) - per-model "load_status_updates" override in models.json (unset = inherit) The same gate drives the SSE side and, via client.status_gate, the out-of-band `pending` keepalives. Unknown model defaults to on so the relay deadline stays protected. Co-Authored-By:
-
- 22 Jun, 2026 32 commits
-
-
Stefy Lanza (nextime / spora ) authored
Newer OpenAI clients send max_completion_tokens instead of the deprecated max_tokens. Our request model only read max_tokens, so such a request arrived with max_tokens=None and fell back to the default — ignoring the client's limit. Add a max_completion_tokens field and a model_validator that maps it onto max_tokens when max_tokens is omitted (max_tokens wins when both are present). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
A remote client (Hermes) reportedly sends max_tokens=512, but requests arrive at the engine with no max_tokens (defaulting to 2048). Our broker client preserves whatever payload it's handed, so log the inbound max_tokens / max_completion_tokens and the body keys when a brokered request arrives — this shows definitively whether the value survives the aisbf relay into our payload or is dropped upstream (remote), so we fix the right layer. Diagnostic only. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Two fixes from live debugging: 1. broker_execute_stream: remove the "the model is thinking..." keepalive chunk. It leaked into the assistant reply (saved turns began with the placeholder) and isn't wanted — we just wait for the real response. The engine waits ~5min for a load and the broker sends protocol-level `pending` keepalives, so liveness holds without injecting content. Reverts the streaming body to the simple retry-on-connection/transient-5xx form (no queue/keepalive). 2. _clamp_max_tokens: always resolve a concrete max_tokens, never None. The main stream/generate paths pass request.max_tokens bare; when the client omits it the GGUF backend's `max_tokens or 512` default truncated replies at ~512 tokens mid-sentence. Now: model-level cap is authority (client honored only if smaller); no cap configured -> keep client value, else default 2048. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Add a model-level max_tokens authority: per-model models.json "max_tokens" wins, else a new node-wide models.max_tokens (default None = no cap). A client's requested max_tokens is honored only when it is SMALLER than the model-level cap; a larger or absent request is clamped to the model-level value. Applied before auto-compaction so the window-fit trim works off the clamped value. When no cap is configured, behavior is unchanged (client value, or the 2048 fallback). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Even after the readiness retries, a brokered stream could still reach the client empty: when the model falls back to CPU offload after a VRAM OOM, the load + first prefill can take tens of seconds during which the engine sends no SSE bytes, so the relay/client times out and shows an empty/errored reply. Keep the stream alive end-to-end: await the engine response and read its stream via a background task, pulling with a timeout. On any gap (long load wait, or slow first token) emit a visible "the model is thinking..." chunk once - a reasoning_content delta when thinking is enabled (enable_thinking/thinking, or a known reasoning model, unless reasoning_effort==none), otherwise a plain content message - then cheap SSE-comment pings to hold the connection without further polluting the reply. Fast responses see no gap and thus no keepalive. Slowness is fine; an empty reply is not. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The first pass only retried when pick_engine returned no engine or the engine returned a non-200. But the real empty-reply case is a CONNECTION failure: during the engine startup/restart window the engine process isn't accepting yet, so self._long.send raises "All connection attempts failed" — caught by the outer except and relayed as a single empty SSE chunk (chunks=1) over the broker. Move the send inside the retry loop on both broker paths (streaming + buffered): retry on a connection exception and on not-ready statuses (425/429/500/502/503/504), attempt-bound (6 x 5s), and only surface the error once exhausted. A 4xx is a real client error and is still relayed as-is. Confirmed against debug.log: all failures were in the engine-startup window; requests after the engine came up streamed fine. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Brokered inference (both the streaming and buffered front executors) hard-failed in the window where no engine/model was ready yet — e.g. engine still warming up, or an OOM-triggered evict+reload in progress. Over the broker that not-ready reply lands at the client as a single empty SSE chunk (the "empty reply / looks like a dead session" symptom), even though a direct API request to the same engine just waits (the engine's own /v1/chat/completions handler retries ~5min for a load). Make the broker paths behave like the direct path: when pick_engine returns no ready engine (or the engine returns a non-200 not-ready status), wait and retry, attempt-bound (6 × 5s), and only surface an error once retries are exhausted. A non-200 means no tokens were generated, so re-sending the body is safe (no duplicate output). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Root cause (from [timing] logs): chat "build" = 39.6s while prefill+first-token = 0.31s. The ~40s is create_chat_completion rendering the chat template via llama-cpp-python's ImmutableSandboxedEnvironment, whose per-operation security checks make a heavy template (gemma-4's recursive / O(n^2) macros over a big Kilocode conversation) take tens of seconds — every request. Fix (generic for ALL gguf/llama.cpp models): render the prompt ourselves with a plain jinja2.Environment compiled ONCE from the model's tokenizer.chat_template (same trim/lstrip, IgnoreGenerationTags + loopcontrols extensions, tojson filter, same render vars), then tokenize with add_bos=False + special=True (matching llama-cpp's added_special handling — no double BOS) and generate via create_completion. Output is byte-identical to the sandboxed render (verified), but without the sandbox overhead. Plus a small per-message render cache and finer timing ([timing] chat render / tokenize / create_completion setup / first-token). Falls back to create_chat_completion if the model has no chat_template or rendering fails. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The front's broker path buffered the entire SSE response and sent one envelope, so broker clients (lisa, and any OpenAI client via AISBF) got the whole reply at once. Now the front streams it: - streaming.py: stream_chunk_envelope uses event="chunk" with payload.chunk (the shape the AISBF relay consumes), finalize_stream uses event="done". - dispatcher.py: extracted resolve_broker_request() (op routing + body decode + validation) shared by the buffered dispatch and the new streaming path, so they can't diverge; added BrokerDispatchError. - app.py: broker_execute_stream() proxies to the engine with stream=True and yields each SSE chunk (sharing the per-model queue + in-flight tracking); start_broker wires client.stream_dispatcher. - client.py: handle_message, for stream=true requests, relays each yielded chunk as a `chunk` envelope and ends with a `done` (instead of the single buffered reply). Requires the matching AISBF change (send_request returns early for streaming and the relay drains the chunk queue) — committed in the aisbf repo. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Direct Kilocode requests showed ~40s before the first token even though prefill was a cache hit (14551 prefix-match, 1 token to eval) — so the delay is before decode. Added timing in vulkan generate_chat_stream: logs "[timing] chat build (template+tokenize)" around create_chat_completion (eager chat-template render + tokenize) and "[timing] chat lock+prefill+first-token" around the first streamed token. One request now reveals whether the 40s is the gemma chat-template Jinja render (huge template + many tool schemas, done in Python) or lock/prefill. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Capability detection is already name-based (microsecond-fast), so caching it wouldn't help — the real cost is huggingface_hub.scan_cache_dir() + the per-file size sums walking the whole cache. So instead of re-walking on a TTL, the cached-models/cache-stats layer now computes a cheap change-signature (the names + mtimes of the top-level model dirs/files — a handful of stat() calls) and only does the expensive scan when that signature changes. Unchanged caches return the last result instantly and never re-walk; a change refreshes in the background. A long 600s backstop TTL still re-scans occasionally. Mutating ops reset the signature. Net: the models page is instant on every load and the background refresh only runs when models are actually added/removed/downloaded. Verified: unchanged signature → 0 rescans; changed signature → one background rescan. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The models page was slow because the work itself is slow, not because of routing: cached-models walks the HF/GGUF cache doing per-model capability detection, and cache-stats walks every file summing sizes — many seconds each on a large cache. - routes.py: cached-models + cache-stats are now stale-while-revalidate (TTL 25s). First call computes; afterwards they return the last result instantly and refresh in a background thread. Mutating ops (clear-cache, delete cached model, free-disk) force the next refresh. The coderai-system worker pre-warms both scans on startup, so even the first page load is fast. - gpu_detect.gpu_stats(): short 1.5s TTL cache — the Tasks page polls it ~every second and nvidia-smi is slow on a saturated GPU, so collapse repeats. Verified: _cached returns stale instantly and refreshes in the background (2 underlying calls across first+expire); all modules compile. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The front is the authority for config READS (settings GET, status, queue capacity). A settings SAVE stays on the engine because it applies live runtime changes (thermal, RAM monitor, broker runtime) that only exist there and then persists config.json. The front now detects the config.json mtime change and re-reads it into self.config IN PLACE (so the supervisor/admin_data references see it), keeping everything it serves current without a restart. Triggered from status() and the settings GET handler (via an on_config_read callback) and lazily in the capacity helper. This is the pragmatic Phase 2: front owns/serves config; engine applies+persists; front stays consistent. (Fully relocating the 362-line apply logic to the front isn't sound — the live application must run on the engine.) Verified: front reflects an external config.json change (queue_max 6→11) with no restart. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The archive endpoints (list/get/delete/file/settings) are served by the front (codai/frontproxy/admin_data.py, from disk via archive_manager), so the engine's copies in admin/routes.py were dead. Removed. Note: the cache/download/HF handlers in routes.py are NOT removed — the coderai-system worker serves them through the same admin_router, so they're only unrouted on the engine, not dead code. Fully removing them from the engine would require splitting routes.py into engine vs system routers (future cleanup). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The front issues every load/unload, so it now updates its own per-engine resident set in the registry the moment the action succeeds — the models page (whose loaded status is front-native) reflects it immediately instead of waiting for the next supervisor engine-state poll. The poll still reconciles the exact keys. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The engine emits a lot of debug output during generation. Builtin print() goes through a buffered, lock-guarded text stream — per-call overhead plus a lock that can serialize against other threads on the hot path. New codai/api/fastlog.py fast_print writes each line with a single os.write(1) syscall: unbuffered, no Python-level lock, atomic for pipe writes up to PIPE_BUF (log lines don't interleave), with a fallback to real print() for stderr/failures. Imported as print() in the generation hot path: api/text.py, backends/vulkan.py, backends/ cuda.py. Verified: fast_print mirrors print's signature (sep/end/file/flush), writes to fd 1, falls back for stderr; all three modules compile. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
#1 /v1/models: the front fanned out to every engine's /v1/models (slow during generation). It now fetches the full list once from the coderai-system worker (config-based list_models, off the GPU engines) — added a /v1/models route to the system app — with a last-good cache and per-engine fallback if the worker is down. #3 quant status: the models page polled /admin/api/quantize-status (engine) every 5s unconditionally. It now polls only while a quant job is actually running (and the tab is visible); startQuantize re-arms it. No idle engine hits. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Fixes the models page's 6-7s stall and reduces engine hits generally: - loadCachedModels awaited /admin/api/models (engine) before painting the cached- models cards, so the fast list waited on the slow engine. /admin/api/models (and accel-presets/accel-loras/turboquant-info/model-enable/model-disable) are pure config reads/edits, so they now route to the coderai-system worker (it has config_manager + models.json) — fast, off the engine. - model-loaded-status is now served from the FRONT's own state: loaded comes from the registry (the front issues every load/unload; the supervisor keeps each engine's resident set fresh via the cheap engine-state poll) and running from the front's in-flight tracking. Per-model instance detail is synced from the engine ONLY when it's idle (inflight==0) and cached otherwise — no engine hit during generation. This is the "front maintains its own list, syncs when not under load" model. - The 1s Tasks poll now skips the engine entirely when the primary is mid- generation, serving the merged/synthesized list from cache + live in-flight tracking instead of burning the timeout. Verified: model-loaded-status returns front-native loaded/running with no engine call while busy; compiles. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The /admin/api/batch aggregator fans every sub-path to the PRIMARY ENGINE. After moving cache-stats/cached-models/downloads to the coderai-system worker and settings to the front, batching them still sent them to the (busy) engine — so the models page kept loading its data from the slow engine. Those endpoints now route to fast targets (front + system worker), so the batch's connection-saving no longer applies and was actually the bottleneck. The models page now loads each first-paint read directly to its correct target: settings → front, cache-stats/cached-models/downloads → coderai-system, only models/ quantize-status → engine. The main model cards come from cached-models (system worker), so the list paints fast even mid-generation. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Phase 1 of moving non-GPU work off the engine. The front now spawns and supervises a third process type — coderai-system — a lightweight (torch-free) worker that owns the slow, I/O-bound, GPU-irrelevant endpoints: cache scan (cached-models, cache-stats, cache delete), HF downloads (model-download, download-stream SSE, downloads, cancel), HF lookups (hf-search/files/model-info/ model-files, ds4 defaults), and disk/bookkeeping (model-upload, model-free-disk, model-add-known, model-mark/unmark-download). The front routes those /admin/api paths to the worker instead of an engine, so they stay responsive during generation and survive engine restarts — fixing the models/archive-style hangs for the cache/download sections. Mechanics: - cli.py: --system-only flag. main.py: proc title coderai-system + run branch. - codai/system_app.py: slim FastAPI = admin_router only + session/config init + internal-auth gate + /healthz + /internal/engine-state + reload-config. Stays torch-free (the cache/download/HF handlers import no torch). - engine_supervisor: spawns one role="system" worker on the next internal port (_engine_cmd gains a mode arg), health-polls it, and excludes it from model assignment (it owns no models) while still queueing it model-json reloads. - registry: Engine.role; can_serve()==False for role="system" so it's never an inference target. Front excludes it from the engine tiles. - app.py: _SYSTEM_PATHS + _system_engine() + _proxy_passthrough() route the cache/download paths to the worker (long client handles JSON + SSE). - routes.get_current_user now validates ANY session/session_<port> cookie by signature, so front/engine/system (different ports) all accept one browser cookie. Engine still physically defines these handlers (now dead — front routes to the worker); stripping them is a later cleanup. Verified: slim app serves cache-stats with cross-port cookie + internal token (403 without), engine-state; front routing picks the worker and excludes it from inference/tiles. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The front already tracks which models are in-flight (engine.active); model-loaded- status now returns a front-native `running` list. The models page shows a pulsing green dot inside each model's engine/GPU tag while that model is serving a request, toggled in place via a 2s poll of the fast front endpoint (no list re-render, so it never re-hits the slow cache scan). Dot styling lives in base.html. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The archive page's data (/admin/api/archive[-settings], get/delete/file) was proxied to the GIL-busy engine, so it hung during generation. codai/api/archive.py is torch-free and pure on-disk, so the front now serves these directly via archive_manager (configured from config.archive), with disk scans run in a thread. The engine's archive handlers are now dead (front handles before the catch-all) and can be stripped in a later cleanup. Verified: list/settings/get(404) via TestClient. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
- /admin/api/system-stats now built on the front (psutil for CPU/RAM, torch-free gpu_detect for GPU/VRAM) instead of poll-proxying to the GIL-busy engine, so the Tasks header tiles stay live during generation. - Removed from codai/admin/routes.py (engine), now all owned by the front: the dead page-GET routes (login/admin/models/tokens/users/tasks/chat/settings/ archive/change-password GET — the front renders these), api_gpu_stats and api_system_stats. Kept the auth POSTs (login/logout/change-password), _tmpl, _do_task_cancel, build_settings_dict and api_save_settings. Verified: front _build_system_stats returns cpu/ram/gpu/vram; routes.py imports. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
With --single-process gone, the engine only ever runs behind the front, which serves these before its catch-all — so the engine copies were dead code. Removed from codai/admin/routes.py: - api_status (GET /admin/api/status) — front builds status from its own state - api_list_tokens/api_create_token/api_delete_token — front admin_data (auth.json) - api_create_user/api_delete_user — front admin_data (auth.json) - api_get_settings (GET) — front serves via the shared build_settings_dict Kept: build_settings_dict (imported by the front), api_save_settings (POST — the engine still persists/applies config), api_list_models, and the auth POSTs (login/logout/change-password) the front still proxies. The engine now does generation + model management only; the front owns pages, stats and these data reads. Verified routes.py imports and compiles. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
coderai now always boots as the front proxy + supervised engine subprocess(es); the one-process "UI/API and model work in the same process" mode is gone. This unblocks removing the engine's now-dead UI/stats/data handlers (the engine only ever runs as --engine-only behind the front). Removed: the --single-process CLI flag (cli.py), config.server.single_process (config.py + save), the proc-title branch and the run-mode branch (main.py), and the arg from the engine spawn filter (engine_supervisor.py). Existing config.json files with a stale single_process key are tolerated (_dc ignores unknown keys). Verified: argparse rejects --single-process; --engine-only still parses. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Per "always use the front when it can serve it; the engine only generates and gets stats from the front": /admin/api/status no longer calls the engine at all. It's built entirely from front state — registry (loaded models, in-flight), config (backend/load_mode), models.json (enabled models/aliases), live torch-free GPU stats (VRAM) and psutil (system RAM). Recent activity is now tracked on the front (it relays every request) via a ring buffer recorded in both the direct proxy and broker dispatch paths, so the Overview activity table is front-native too. Removed the now-dead _overlay_engine_totals/_cached_status/_status_cache. Engine-side removal of the now-unused handlers (api_status, tokens/users CRUD, api_get_settings) is pending a decision on the legacy --single-process mode, which still serves the UI from the engine app. Verified: _native_status() returns vram+ram+recent_activity via standalone test. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Overview's only call (/admin/api/status) was proxied to the engine, so during a generation (engine GIL-busy) it timed out and the dashboard showed nothing. The front now builds status from its OWN state — registry (loaded models, per- engine in-flight), config (backend, load_mode), models.json (enabled models + aliases) and live torch-free GPU stats (VRAM summed across cards) — with no engine round-trip. It's used whenever the engine can't answer, and the front skips the engine entirely when the primary is mid-generation (inflight > 0) so Overview is instant instead of waiting out the timeout. Engine-only fields (recent_activity, RAM watcher) are carried over from the last good status; when the engine is idle the richer engine status is still used. Verified: _native_status() builds a complete payload (status/backend/load_mode/ enabled_models/vram/requests) via TestClient/standalone. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Root-cause finding (Path B): the engine event loop is genuinely freed during GGUF generation (work runs via asyncio.to_thread; llama_decode releases the GIL), so it's not a hard-blocked loop. But under CONTINUOUS generation the per-token Python work in llama-cpp-python keeps re-grabbing the GIL and starves the engine's own API handlers (sync ones especially) — an intrinsic ceiling that can't be cleanly removed while generating. So per "B first, A fallback", falling back to A: serve page data from the front (a separate process the GIL can't touch). This commit: /admin/api/settings GET is now served by the front. api_get_settings was refactored into a pure build_settings_dict(config, gpu_cards) shared by the engine handler and the front (front holds the same Config), so both return an identical payload. Front uses its torch-free gpu_detect.gpu_cards(). POST (save) still falls through to the engine, which persists + applies it. Verified: build_settings_dict standalone + front /admin/api/settings via TestClient. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Pages opened from the front but their DATA was still proxied to the engine, so while a generation ran (engine event loop GIL-busy) Tokens/Users/etc. showed no data. Tokens and users are backed purely by auth.json under config_dir, which the front already has — so it can answer them from disk with zero engine round-trip. New codai/frontproxy/admin_data.py (register_admin_data): front handlers for GET/POST/DELETE /admin/api/tokens and /admin/api/users, mirroring the engine handlers exactly (same shapes/status codes), with sessions validated locally against the shared secret and admin gating. Registered before the catch-all so they're served here, not forwarded. The engine still reads auth.json fresh for bearer/session auth, so tokens created via the front work for inference. This fully unblocks the Tokens and Users pages during generation. Settings/ status/models-list are the next candidates (config + models.json + registry are all local to the front too). Verified via TestClient: full CRUD, 401/403 gating, and pass-through isolation. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The Models page fired ~10 /admin/api reads on load. Browsers cap ~6 connections per host, so during a generation (engine event loop GIL-busy) the proxied reads pile up, saturate the limit and freeze the whole page behind them. Two fixes: 1. New front endpoint POST /admin/api/batch: fans out several engine GET reads CONCURRENTLY server-side (front's own un-capped pool), each individually bounded, and returns them in one response. A slow/blocked sub-call returns an error marker without holding up the rest. base.html gains bootstrapPage() + pageFetch(): the page batch-loads its first-paint reads in ONE request, and the existing loaders transparently serve from the bundle (falling back to a normal fetch on miss). models.html now collapses settings/cache-stats/cached-models/ models/quantize-status/downloads into a single batched call, plus an in-flight guard on the downloads poller so ticks can't stack up. 2. Bound /admin/api/* GET proxying (catch-all) with a finite read timeout + graceful 503 fallback, so a busy engine can never hang a page forever (HF hub lookups and streaming endpoints keep the unbounded path). Verified: batch unit test (concurrent aggregation, error markers, path filtering, auth) and node syntax-check of the bootstrap script. Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
Move request queuing from the engine to the front so the engine does only generation. New codai/frontproxy/reqqueue.py FrontQueue: a per-model concurrency gate sized to the model's max_instances (per-model from models.json, else the global models.max_model_instances). A text-generation request acquires a slot before dispatch; when all slots are busy it waits (FIFO) and shows as "queued" with its position on the Tasks page; beyond server.queue_max_size waiting it gets HTTP 503. A client disconnect while queued cancels the wait and drops it from the queue with no slot leak. Wired into both dispatch paths in app.py (direct proxy + broker route) so direct and brokered requests share one queue, scoped to text generation only (images/ audio/embeddings pass through unqueued). Slots are released on stream completion, send failure, or disconnect. Queue key is the canonical model id so all aliases/ paths of a model share one gate. Front never over-subscribes the engine, so the engine's own queue stays a safety net that effectively never fills. The engine-side QueueManager + KV-affinity instance pool are intentionally left in place for now (trimming them is a follow-up once this is verified live). Verified: FrontQueue unit tests (capacity, FIFO hand-off, QueueFull, cancel cleanup, position, capacity>1) and a FrontProxy integration test (capacity resolution, alias-key collapsing, queued task surfacing). Co-Authored-By:
-
Stefy Lanza (nextime / spora ) authored
The front reverse-proxied every page navigation to the primary engine, so opening any page while that engine was mid-generation (event loop busy) made the whole UI hang until generation finished. Now the front renders the UI pages itself (dashboard, models, tokens, users, tasks, settings, archive, chat, login, change-password) and serves the admin static assets + favicon locally. Sessions are validated locally: the cookie is HMAC-signed with the secret under config_dir (shared via auth.json), so the front authenticates it with no round-trip to a busy engine — and matches the port-derived session_<port> cookie name by signature, not by exact name. Mutating auth actions (login/logout/change-password POST) and all /admin/api/* data calls still fall through to the catch-all proxy and reach the engine. New: codai/frontproxy/ui_pages.py (register_ui_pages), wired in build_app before the catch-all. Verified via TestClient: local render, unauth->login redirect, authed render, static, and POST/logout pass-through. Co-Authored-By:
-