Exact filename or suffix pattern. Leave blank to download all .gguf files.
Will download the full repository using the HuggingFace snapshot API. This is the correct method for safetensors / non-GGUF models. Large repos may take a while.
Preparing…
———0%
Upload GGUF model
Uploading…
0%—
Edit whisper-server model
Configure model
Capabilities
Text / Language
Image generation & editing
Image analysis & processing
Video
Audio
3D
Shown on the config pill. Falls back to alias, then "Config N".
Backend
Auto runs the model on ONE card (picked by capability / most free VRAM). Pin it to a specific engine — a card here or a cluster node (another coderai on your network) — or pick a Split — <card> first option to spread one model across all GPUs of this machine (more total VRAM) with that card leading. Options an engine cannot run are greyed out for the selected backend, and vice versa. Only shown when more than one engine or a node is available.When off (default), a request fails if the pinned engine is down/busy-unreachable instead of running on a different card.
VRAM packs the most onto every card (best for fitting a large model) but the slow card bottlenecks throughput. Speed keeps as much as possible on the lead/fast card and spills only the overflow to the slower one — faster, as long as it still fits.Max VRAM the auto-split may put on each secondary (slow) card, e.g. 4 = at most 4 GB on the RX 580; the remainder stays on the lead card or spills to CPU. Lower = the slow card bottlenecks less. Blank = use the global default; 0 = no cap.Optional explicit ratio for the lead card then the rest, e.g. 0.8,0.2. Overrides the strategy. Blank = automatic per the strategy above.
The node registers the model under this path when it does not share this machine's filesystem. Blank works when the path exists on the node too (NFS, same layout) or the model is a HuggingFace id.
Splits what a request carries — n images or videos, a list to embed or rerank, a long text to speak (by sentences), a long recording to transcribe (cut at silences), a batch of documents to OCR — over every engine that has this model, here and on cluster nodes, and merges the answers. One generation is never split; that is Cards on other machines / vLLM nodes above.
A video model is several models run one after the other; what passes between them is small. Name a cluster node for a part and that part runs there: the text encoder (UMT5-XXL, 11 GB) encodes the prompt elsewhere; the low-noise expert of a Wan 2.2 dual-expert model takes over at the expert boundary (latents handed over, a few MB); the VAE decodes elsewhere and returns the finished video. Blank = here. The node needs the same model.
Each host:port where an rpc-server listens becomes one more GPU in this model's layer split, after this machine's cards in that order (so a Weight distribution ratio reads: local cards…, then these). Known servers from the Cluster page are suggested. Decode over a LAN is workable (activations per token are small); loading and prompt processing are wire-bound — 1 GbE hurts, 10 GbE is fine. The protocol is unauthenticated: LAN or WireGuard only. Tick Split above or leave Auto — a model naming RPC servers is split by definition.
Applies to local cards and RPC servers alike. Row split trades one all-reduce per layer for using every card on every token: a win on one PCIe bus or a fast fabric, a loss on 1 GbE.
Blank = inherit the global vLLM settings. Tensor parallel splits every layer across GPUs; with nodes listed it may exceed this machine's cards and span the Ray cluster (an all-reduce per layer crosses the wire: 10 GbE is the floor, RDMA/InfiniBand the point). Pipeline parallel gives each machine a slice of layers — the one to use on ordinary Ethernet. Both can be combined (tensor × pipeline GPUs in total). With nodes listed (or pipeline > 1) coderai starts a Ray head here, runs each node's command, waits until Ray sees tensor×pipeline GPUs, then launches vLLM on Ray.
Keeps enough cache for this many requests at the model's full context. 0 = size by GiB only.
{ray_address}, {head} and {port} are filled in. The coderai-vllm image joins as a Ray worker when started with CODERAI_RAY_ADDRESS; every node must run the same vLLM build.
On one GPU, decode is limited by how fast the weights can be read, so it cannot be tuned past the card's memory bandwidth — only drafting beats that. Prefill can be: it is compute-bound, and a long prompt is where nearly all of a long-context request's time goes.
Leave blank. This declares how the weights on disk are already packed — it converts nothing — and vLLM reads that from the checkpoint itself and picks its fastest kernel for it. Naming the wrong format moves the engine onto a slower kernel or stops it booting, and a repo whose name says AWQ can still be compressed-tensors.
vLLM prefills this many tokens per engine step, so at the default a 60k-token prompt takes ~30 sequential passes of small, tensor-core-starved GEMMs. Raising it (8192 is a good first try) cuts time-to-first-token on long prompts. The cost comes out of the KV cache: check Available KV cache memory and the concurrency in the boot log afterwards.
Reuses the KV of a prompt prefix that was computed before, which turns a whole prefill into almost nothing when the same document is asked about again and again. vLLM's own default is on for ordinary models but off for hybrid-attention ones, where it is supported yet still opt-in upstream — so for those this switch is the only way to get it. Worth re-running a retrieval check after turning it on.
ngram_gpu needs no draft model and no VRAM: it guesses the next tokens by finding the current text earlier in the prompt. That pays off precisely when the answer quotes its source — extraction, citation, summarising a document — and does nothing for free-form prose. Prefer it over plain ngram, which vLLM lists as incompatible with async scheduling and so turns that off, paying for the draft with a throughput loss elsewhere. Anything else is passed to vLLM as --speculative-config JSON, so a real draft model can be configured here too.
Runs in an isolated Python 3.10 venv (torch 2.6+cu124, transformers 4.41) behind a managed subprocess, because those pin against the server's own 3.13 / 2.11 / 5.x. Blank fields inherit the global Settings → LongCat section. Generation is three coarse-to-fine stages; Quality picks which run. Long video is generated in segments whose first conditioning frames re-render the previous segment's tail, so each segment adds the remainder — ~11 segments is a minute at 15fps.
INT8 is only supported by the official avatar-1.5 weights, which also require distilled sampling. Offloading the KV cache trades speed for VRAM across segments.
Blank = inherit the global ktransformers settings. With nodes > 1 this machine is rank 0; ranks 1… are started by the commands below (templated with {rank}, {nnodes}, {dist_init_addr}, {model}) before rank 0 launches.
These tune ds4-server for THIS model (they vary by quant/size/context). Leave blank to inherit the global ds4 settings. Context window is set by n_ctx above.
On for big models that don't fit VRAM; off for small ones that do.
Set just above this model's resident weights (+~2 GiB). ds4 defaults to half the card, which starves the cache.
Cap the expert cache (a count or NGB), prefill-chunk, kv-disk-space, etc. Blank = inherit global.
KEY=VALUE pairs. e.g. DS4_CUDA_WEIGHT_ARENA_CHUNK_MB=512 fixes weight-arena OOM under heavy expert caching. Blank = inherit global.
⚡ Performance: put the GGUF on NVMe/SSD (≈10× prefill for a streamed MoE) ·
cap the expert cache with --ssd-streaming-cache-experts N (a count, e.g. 400) — higher helps decode but too high OOMs the weight arena ·
set reserve ≈ weights + ~2 GiB · add DS4_CUDA_WEIGHT_ARENA_CHUNK_MB=512 · avoidDS4_CUDA_WEIGHT_CACHE (slower). Decode of a model bigger than VRAM is streaming-bound (~0.5 tok/s) — the real fix is a smaller quant. Context = n_ctx above.
These tune the colibri engine for THIS model. Leave blank to inherit the global colibri settings. Context window is set by n_ctx above.
1–16 concurrent cached conversations.
Engine worker cap.
Exports CUDA_EXPERT_GB. Blank = inherit global.
KEY=VALUE pairs for colibri's env-only tuning. Blank = inherit global.
Placement — where this model runs
RunPod and Host are exclusive: the model is never loaded here. To keep it local and only overflow to RunPod, leave "This machine" and use "Burst" below.
RunPod — remote GPU
This model runs on a rented RunPod GPU and is not downloaded locally. Set the model Path/ID (top) to the served model id (e.g. an HF repo) — the remote worker pulls it. Requires RunPod enabled in Settings.
The window the pod is launched with (--max-model-len), and the one this model reports to clients and sizes auto-compaction from. The general context field above does not reach a pod. It takes effect on the next pod: a running one keeps the window it started with, and a pod that is re-adopted keeps it too. With a shared pool, the first model to create the pool sets this for every model on it — so set the same value on all of them.
No cold-boot wait (image pull + weight download, minutes) — but it bills every hour of every day, including the ones nobody uses it. Off by default.
Outside the window the warm pod is terminated and billing stops. Requests still work — one arriving out of hours cold-starts a pod, exactly as it does with no warm pod at all.
None ticked = every day. An end earlier than the start runs overnight, and belongs to the day it starts on.
How many requests the pod's own server admits at once. The defaults are sized for a shared workstation and will throttle a dedicated card long before its GPU is busy — everything over the limit gets a 429. Set these to roughly the remote engine's batch width (e.g. vLLM's max_num_seqs). The numbers above apply to any card without a row in the table below.
GPU
parallel
queue
scale up at
in‑flight cap
A mixed fleet needs a number per card, not one number scaled: a figure measured on one GPU does not describe another, and leaving it to be multiplied means changing the rental preference silently re-scales it. Leave a card out until you have measured it — it then falls back to the numbers above (or to its VRAM ratio). scale up at is the in-flight depth at which that card justifies renting another pod, and it also decides that card’s share of the load: two cards with depths 32 and 93 are given work in that ratio.
Pin this model's pods to one RunPod data centre. A network volume still wins: a pod in another region cannot attach it.
Wait for the pod to expose its port; a stuck machine is dropped and the next one tried.
Wait for vLLM to serve the model — raise this for big models (long image pull + weight download).
⚠ Frontier-size weights. Models on this engine are 100 GB and up — DeepSeek-V4 alone is ~154 GB, Kimi-K3 is measured in terabytes. Without a network volume every cold pod downloads all of it before answering, typically one to two hours of rental per boot. Set a network volume and, better, the "weights already on the volume" path so nothing is downloaded at all. The pod's disk is sized from Weights (GB) when no volume is set.
A local .gguf path means nothing on a rented machine — name the HF repo the pod should download.
Weights, uploads and adapters land here and outlive the pod. Secure Cloud only; pods are pinned to the volume's region.
Keeps this model's region fallback list AND its storage, which one volume id cannot. The region is picked first, then its volume. Regions missing here are not rented for this model.
Relative to the volume mount. The pod loads this instead of downloading — the only sane way to run a 150 GB+ engine model (ds4, colibri, k3).
Boots a small image and runs from a venv on the volume — the first pod builds it, later pods skip the 7 GB pull. Needs a volume; imports are slower off network storage.
Direct has no request cutoff and no body cap; coderai images bring a certificate from this install's own CA so nothing travels in clear. vLLM / llama.cpp images cannot take a certificate, so on the direct path they would be plain HTTP — the proxy is their TLS.
Models naming the same pool share one set of pods and one budget until it saturates. Needs a coderai or custom multi-model image — a vLLM or llama.cpp pod is launched for ONE model and sharing is refused.
For an OCR VLM the image also chooses where the OCR pipeline runs,
and it is the only switch:
coderai-vllm — the pod serves the model and this box rasterises every
page and sends it as a base64 PNG (~1 MB per A4 page at 250 DPI,
and again for each resumption round of a long page).
coderai-vllm-ocr — the pod carries surya and the rasteriser, so it is
sent a one-page PDF (~474 KB, 1.2 ms of CPU here instead of
~105 ms to render) and reads it on the card, talking to its own vLLM
over loopback. Either way one page is one request, so every pod in the pool
still shares the work. Page budgets, DPI, continuation and repeat-detector
settings travel to the pod, so it reads a page the same way.
The pod is launched requiring this token and coderai sends it. Blank does NOT mean open — one is generated.
Anyone with the pod URL can use your GPU. Only for images with no auth support.
Add a pod once the least-loaded one carries this many requests.
Give a pod back once the load no longer needs it and has not for this long. One per pass, least-loaded first, draining rather than dropping work. 0 = keep every pod until it goes fully idle (idle_timeout_s), which on a trickle of traffic means paying for the peak.
Above this a request waits for a slot instead of piling on.
How many pods may be booting together when the pool grows. This is also how many GPUs start billing together, which is why it is not simply max_pods: a pool filling a floor of 16 four at a time reaches it in waves, and a mistake costs one wave rather than the whole fleet. Raise it when a batch is waiting and the ramp is the bottleneck.
Send a conversation back to the pod whose cache still holds it.
Host — your own machine
A coderai capability image (or any coderai) running on a machine you control. Always-on if the start command is blank; otherwise coderai runs the start command, waits for /healthz, and runs the stop command after the idle timeout. See docs/remote-execution.md.
Several machines form a pool: each request goes to the healthy one with the fewest in flight, a dead one is skipped for 30 s and the request moves on, and an on-demand machine is started only when none is up.
Pins it here even when its whole capability is set remote — so one video model can stay local while another runs on a pod.
Another coderai, llama-server, vLLM, a pod. Nothing is downloaded and no local VRAM is used.
Local stays primary. If a burst can't be opened the request just queues locally.
When on, the real measured VRAM is re-recorded every run and used for eviction even if "Used VRAM" is set above.
Parallel copies in VRAM
Inference
-1 = all on GPU (auto-offloads the overflow to CPU if it won't fit). N = N layers on GPU, the rest on CPU (slower, frees VRAM).
Prompt-ingestion batch. Lowering it (e.g. 256) shrinks the compute buffer so a big-context model fits at the VRAM ceiling — slower prefill. Blank = default (512).
Concurrent sequence slots. Each slot reserves its own KV/compute VRAM; keep at 1 to avoid reserving for unused parallelism. Blank = default (1). Ignored by older llama-cpp-python builds.
Pairs an mmproj GGUF with this model to add image/vision input (passed to llama.cpp as --mmproj). Auto-selected when a matching projector is detected.
Note: KV-cache quantization only applies to GGUF models on
the llama.cpp backend. HF-transformers models (incl. gemma and
other sliding-window / hybrid linear-attention architectures) ignore this and
keep an fp16 KV cache — their quantized-cache path crashes during generation.
For those, lower n_ctx to shrink the KV VRAM reserve instead.
Auto-compact context
Compacts down to the size below when the prompt reaches this %.
Lower leaves more room to grow before the next compaction.
From the first user turn. 0 = keep nothing at the start.
0 = at most half the target, so the start never starves the recent end.
Recent conversation kept verbatim.
A separate (e.g. smaller/faster) model can write the summary while the main model answers. The dropped history is chunked to fit the chosen model's own context. Empty = the request's own model summarizes. Leave empty to inherit the global default.
By default the model's thinking is returned as a separate reasoning / reasoning_content field (think/thinking/thought tags, including the bare </think> Qwen pre-fill emits) and kept out of content. Enable this to discard it entirely. A request can override per-call with reasoning:{exclude:true}, reasoning_effort:"none", or suppress_reasoning.
A thinking model generates its thought first and charges it against max_tokens, so on extraction and retrieval work it is most of the latency and none of the answer — and on a tight budget it can consume the whole allowance before the answer begins, returning an empty one. This is the default for requests that say nothing; enable_thinking on the request always wins. Different from Suppress reasoning output above, which still pays for the tokens and merely hides them.
The dtype the model runs in. This is not replaced by the quantization boxes above: 4-bit and 8-bit quantize the weights, while the embeddings, lm_head, the norms, the activations and the KV cache all stay in this dtype — so a 4-bit model left in fp32 still pays fp32 for all of them. fp32 also disables Flash Attention 2, which accepts only fp16 and bf16. Leave it on auto only if you want the loader to pick; set it explicitly when the card has to fit a known budget.
bitsandbytes NF4 is the slowest 4-bit option. Quantizing to GPTQ/AWQ uses
Marlin kernels (2–4× faster) but is a heavy one-time background job; the
produced checkpoint is used automatically on the next load. Falls back to
bitsandbytes if unavailable or the architecture is unsupported.
Component Quantization (image / video pipelines)
Quantize individual pipeline components. Default uses the global 4-bit/8-bit checkboxes above.
2-bit needs optimum-quanto; GGUF file (Q5_K/Q6_K…) needs a pre-quantized .gguf path/URL.
Tip: backbone 4-bit + text encoder 8-bit + VAE None = best quality-per-GB.
Offload
Auto goes straight to model offload when the model's peak exceeds free VRAM (no full-GPU gamble). Auto borderline-aware is the same, except when the model only marginally overshoots (≤3 GB) it tries full-GPU first to keep both experts resident and use the free VRAM, falling back to model offload on OOM. model keeps only the active component on GPU (best for two-expert Wan2.2 — near full-GPU speed). group streams blocks with prefetch (lowest VRAM, fast; incompatible with bitsandbytes 4-bit → falls back to sequential). balanced fills GPU to the % below then spills.
sd.cpp Video Options
Prompt Handling (image models)
Takes the text embedding this many layers from the end of CLIP. The booru-tag SDXL fine-tune families — Pony, Illustrious, NoobAI and their derivatives — are trained at 2 and look washed out and off-prompt without it. Set it here because a client speaking the OpenAI image API has no field to ask for it; a request's own clip_skip still wins.
On, the A1111 grammar is parsed: (tag:1.3) emphasises, [tag] attenuates, bare (tag) is 1.1, nesting multiplies, \( is a literal bracket, and BREAK starts a new 75-token window. Off, brackets reach the model as text. Either way the prompt is encoded in as many windows as it needs, so nothing is cut at 77 tokens.
Comma-joined in front of whatever the caller sent. This is how a model trained on a quality ladder gets it on every request, including from a client that speaks the plain OpenAI image API and only sends a sentence. Applied at most once — a prompt that already starts with it is left alone — and a request can override it, or send an empty string to switch it off.
Learned prompt tokens, so their trigger words work in any prompt sent to this model. A bare name is looked up in the server's embeddings directory (image.embeddings_dir — everything in it is loaded anyway); a path or a HuggingFace repo id also works, and ref:token renames the trigger word. A request's own embeddings list is loaded on top of these.
When the model won't fit entirely in VRAM, fill this % of free VRAM then spill the rest to CPU RAM. Also the cap used by auto-balanced fallback.
Lists distill-LoRA repos found in the local cache. Choose one, then pick its high/low (or single) LoRA below.
Wan2.2 A14B has two experts; the distill LoRA must be fused into both or the clip collapses to a solid colour at 4 steps. Leave blank to apply the single LoRA above to both.
A request's quantization field (e.g. turbo4) overrides the default bits. With encoding_format:"base64" the response is compact packed bytes; otherwise it returns the lossy reconstruction as floats.
simultaneous embeds on the GPU (1 = strict serialise; use for fragile cards like Polaris/RX580)
requests queued beyond concurrency before returning 429 (0 = never shed)
minimum gap between GPU starts; paces a flooding client (0 = no pacing)
Components
Works on text models too (LoRA and QLoRA). A HuggingFace repo id also works — and is the only form a vLLM/llama.cpp pod can load, since it cannot read this disk. llama.cpp needs a GGUF-converted adapter.
Tip: use <lora:name:strength> in prompts to apply at inference time
When this image model is a transformer/DiT (Z-Image, Flux, SD3) the in-process LoRA
trainer can't target it. Set a UNet-based SD1.x/SDXL model here to train identity LoRAs
against, while generation still uses this model. Leave empty to train on this model.
Every LoRA/QLoRA trained on this model runs on this machine and the nodes named
here, each on its own card with its share of the samples; the adapter's gradients are
averaged every step (a few MB — fine on 1 GbE), rank 0 here saves the result. The nodes
need the same base model. A request's own nodes overrides this.
Generation
—
Lip-sync engines
Video-driven presenter engines. They are not models.json entries: each runs in
its own interpreter as a subprocess, so the model manager cannot load one into
VRAM and they carry no row in the VRAM ledger. The inference code ships in the
repo; installing fetches the weights and builds the engine's venv, which takes
minutes and several GB.