{% extends "base.html" %} {% block title %}Settings — CoderAI{% endblock %} {% block content %}

Settings

Server configuration — restart CoderAI to apply changes

Server
IP address or hostname to bind to (0.0.0.0 = all interfaces)
Off = a session, an API token or the cluster token is required. Exported: requests per key/model/kind/status/engine with wall time, engines and nodes up, VRAM, in-flight, RPC servers, discovery, RunPod spend.
Maximum number of concurrent queued requests. Authenticated requests arriving when the queue is full receive a 429 response.
First internal port for engine subprocesses (the front assigns this and the next free ports). Keep it different from the listen port above. Restart to apply.
Concurrency
How many requests each engine runs at once. Each engine is its own process, so this applies per-engine — total concurrency is the sum across engines.
Default number of concurrent copies of one model. Override per model on the Models page.
Storage
Models will inherit this as default when configured
Server-wide ceiling on host RAM (process-tree RSS). New model loads get a CPU-offload budget clamped to the remaining headroom, so the overflow spills to the offload directory (disk) instead of pushing past the limit. Applied live on save.
Unload least-recently-used idle models to free real RAM before forcing disk offload.
Default for every model: pool one model's layers across all GPUs of all backends (e.g. NVIDIA 3090 + Radeon) for more total VRAM. Off = each model stays on its own backend's card(s) (multiple same-backend cards still split). A per-model "Engine / card → All GPUs" setting overrides this. The slower card bottlenecks each token.
Default for split models when no explicit ratio is set. VRAM fits the largest model; Speed keeps most layers on the fast card so the slow one bottlenecks less. Per-model setting overrides this.
Detecting cards…
Cap how much VRAM the auto-split may place on each specific card on this machine — applies to every engine. e.g. set the RX 580 to 4 GB and the RTX 3090 to 20 GB independently; the rest stays on another card or spills to CPU. Blank/0 = no cap for that card.
Comma-separated ratio in llama.cpp device order (CUDA cards first, then Vulkan), e.g. 0.8,0.2. Blank = automatic per the strategy above.
Background watcher samples RSS; when it keeps climbing while idle or nears the cap it runs gc / CUDA cache release / heap trim and (if enabled) evicts idle models.
When the watcher mitigates, let it call torch.cuda.empty_cache(). Turn this off on a GPU that drops off the bus (Xid 79) to stop the background thread from touching CUDA — host-RAM mitigation (gc / heap trim / idle eviction) still runs. Loads are always skipped regardless.
How often the watcher samples process-tree RSS.
Fraction of the RAM cap at which the mitigation ladder engages.
Scratch space for frame extraction, upscaling and interpolation. Point it at a large volume — 4× upscaling can exhaust a small /tmp ("No space left on device"). Applied live on save.
Video Enhancement

Upscaling and FPS interpolation run on in-process torch models by default (ESRGAN upscaler, RIFE/FILM interpolator) — no external tools. Enable these only to fall back to external tools when no model is configured.

Off = use PyAV + models only. On = ffmpeg may be used as a fallback.
Off = use an in-process RIFE/FILM model. On = the external rife-ncnn-vulkan binary may be used when no model is configured.
Generation Archive
When enabled, every generation (image, video, audio, text) is logged with its prompt, model, and output files.
Absolute path or relative to the config directory. Leave blank to use the default.
Archived entries older than this are automatically deleted. Takes effect immediately on save.
Thermal Protection
Before serving a request against a loaded model, wait until temperatures are safe so a long sequence of heavy generations can't overheat the machine and trip its power-off protection. The wait is non-blocking (other requests keep being accepted) and takes effect immediately on save. Temperatures in °C.
How often to re-read temperatures while waiting for cooldown.
Before the hard pause, gently slow generation (short per-step sleeps) once the CPU enters the warm band, so its temperature climbs slower and the full cooldown is rarely hit. CPU only — GPU is unaffected.
Background Jobs
Controls how interrupted LoRA training is handled when CoderAI restarts. Equivalent to the --no-resume-jobs launch flag.
When off, a training job that was running at restart is marked cancelled instead of resuming. Its checkpoint is kept, so you can still restart it manually from the Tasks page.
AISBF Broker
Registers this CoderAI instance with an AISBF broker so it can receive brokered requests.
DeepSeek V4 (ds4)

Run DeepSeek V4 through antirez's native ds4 / DwarfStar engine as a managed subprocess. When enabled, requests for a matching model name are proxied to ds4-server. First use clones + builds ds4 and downloads several GB of weights — this is slow and needs lots of RAM (96 GB+).

GLM-5.2 (colibri)

Run GLM-5.2 through JustVugg's pure-C colibri MoE engine. Unlike ds4, colibri ships no server — coderai drives the C engine binary directly over its stdin/stdout mux protocol (no colibri Python at runtime). First use clones + builds the engine (CUDA when available). The GLM-5.2 int4 container (~372 GB directory) is not downloaded automatically — point the model path at it (see the int4 g64 + int8-MTP container).

Kimi-K3 (kimi-k3-in-c)

Run Kimi-K3 through FareedKhan-dev's portable-C kimi-k3-in-c engine. coderai patches in a resident serve loop and drives it over the same mux protocol as colibri. CPU-only (AVX2+FMA) and needs ~1.7 TB fast local storage (the ~1.56 TB checkpoint + a ~109 GB packed trunk from the repo's scripts/pack-trunk.sh). Nothing is downloaded automatically.

ktransformers (SGLang)

Run many large MoE families (DeepSeek, Kimi, Qwen, GLM, MiniMax) through ktransformers' CPU+GPU heterogeneous engine, served over SGLang's OpenAI API. coderai launches python -m sglang.launch_server as a managed subprocess and proxies to it. Requires SGLang + kt-kernel installed out of band (best on AMX/AVX-512 CPUs). Selected per model via an explicit backend: kt pin or the model id below — never by a broad name marker.

vLLM (high-concurrency)

Serve an LLM (or VLM) through vLLM's OpenAI server with continuous batching — far higher aggregate throughput than serialized single-instance backends. coderai launches vllm.entrypoints.openai.api_server as a managed subprocess and proxies to it. vLLM pins its own torch/CUDA, so it runs in an isolated venv (CUDA-only). Selected per model via a backend: vllm pin or the model id below. Also serves Surya2 for OCR.

Cluster

Other coderai installs on your network become engines of this one: this front polls each node's state, assigns models to it, pins a model to it by name on the Models page (Engine / card), and routes requests there by capability and load — the node's own front picks the card. A node needs only an API token created on its Tokens page. This section also lists the llama.cpp rpc-server processes this machine runs to lend its cards to a GGUF loaded on another node (per model: Cards on other machines). Reachability of everything is on the Cluster page; the full story is in docs/cluster.md.

What a head calls this install; the key of a model's per-node path.
The address other machines reach this one at (RPC endpoints, Ray head).
Type the same on every install. Nodes accept it on /cluster/*; a head uses it for discovered nodes and for listed nodes with a blank token. Blank keeps the stored one, - clears it.
Announces this install as _coderai._tcp and lists the others on the Cluster page. Link-local: same subnet, container on the host network.
Every discovered install with the same token becomes a node, no listing needed.
A conversation's next turn goes back to the engine or node that served it (warm KV cache) when the model is loaded in several places. TTL in seconds.
Nodes — listed by hand; discovered members are added on their own
Verify: system = public certificate, pem = paste the node's self-signed certificate, off = no check (plain http, or a LAN you trust). Capabilities blank = whatever the node reports.
RPC servers this machine runs
One rpc-server process per row (built by packaging/build-rpc-server.sh, from the same llama.cpp as the bundled llama-cpp-python — the protocol is versioned). Device = a ggml device name (CUDA0, Vulkan1 …, blank = default). Unauthenticated protocol: bind to a LAN or WireGuard address, never the internet.
RunPod (remote GPUs)

Burst inference to rented GPUs on RunPod — as Pods (coderai provisions, load-balances and autoscales them) or Serverless endpoints (RunPod autoscales). Uses no local VRAM: coderai proxies OpenAI requests to the remote GPU. Set the account key here, then tag a model in the Models list with backend: runpod and its per-model pod/serverless block. These are the account-wide settings shared by every RunPod model.

Remote capabilities

Images, video, embeddings, TTS, STT and the rest load their model in this process, so there is no per-model way to move them. Point a capability somewhere else and matching /v1 requests are forwarded there instead — to another coderai, or to a RunPod pod coderai provisions, scales and reaps for you. Per-model service_url still wins over anything set here. Chat and completions are not listed: those are per-model, on the model's own page.

OCR (document → text)

Dedicated OCR engines (detection + recognition), not a vision LLM — faithful text with bounding boxes and layout, batched on GPU. Serve PaddleOCR, docTR or Surya, selectable per request. Endpoints: POST /v1/ocr, POST /v1/ocr/batch. Heavy deps install out of band (pip install -r requirements-ocr.txt or ./build.sh … --ocr); a missing engine returns HTTP 503.

Auto-compact context

Global defaults for shrinking an over-long chat history before generation so requests don't error out on context overflow. Per-model settings on the Models page override these. OFF by default.

Credits & acknowledgements

CoderAI stands on the shoulders of remarkable open-source work. Heartfelt thanks to the developers whose engines and research make these capabilities possible.

colibri — GLM-5.2 on consumer hardware
A brilliant pure-C MoE engine by JustVugg that streams a 744B-parameter model across VRAM/RAM/disk to run it on a single GPU. github.com/JustVugg/colibri
ds4 / DwarfStar — DeepSeek-V4 native engine
A superb from-scratch DeepSeek-V4 inference engine by Salvatore Sanfilippo (antirez). github.com/antirez/ds4
llama.cpp & whisper.cpp — GGUF LLM inference & Whisper STT
The foundational C/C++ inference engines by Georgi Gerganov and contributors. llama.cpp · whisper.cpp
Visual place recognition & geolocation research
EigenPlaces (Gabriele Berton et al.) · DINOv2-SALAD (Sergio Izquierdo, Javier Civera) · GeoCLIP (Vicente Vivanco et al.) · DINOv2 (Meta AI) — the models that power visual matching & geolocation.

…and the wider open-source ecosystem (PyTorch, Hugging Face Transformers/Diffusers, and many more) that CoderAI builds on.

{% endblock %} {% block scripts %} {% endblock %}