Exact filename or suffix pattern. Leave blank to download all .gguf files.
Will download the full repository using the HuggingFace snapshot API. This is the correct method for safetensors / non-GGUF models. Large repos may take a while.
Preparing…
———0%
Upload GGUF model
Uploading…
0%—
Edit whisper-server model
Configure model
Capabilities
Text / Language
Image generation & editing
Image analysis & processing
Video
Audio
3D
Shown on the config pill. Falls back to alias, then "Config N".
Backend
Auto runs the model on ONE card (picked by capability / most free VRAM). Pin it to a specific engine — a card here or a cluster node (another coderai on your network) — or pick a Split — <card> first option to spread one model across all GPUs of this machine (more total VRAM) with that card leading. Options an engine cannot run are greyed out for the selected backend, and vice versa. Only shown when more than one engine or a node is available.When off (default), a request fails if the pinned engine is down/busy-unreachable instead of running on a different card.
VRAM packs the most onto every card (best for fitting a large model) but the slow card bottlenecks throughput. Speed keeps as much as possible on the lead/fast card and spills only the overflow to the slower one — faster, as long as it still fits.Max VRAM the auto-split may put on each secondary (slow) card, e.g. 4 = at most 4 GB on the RX 580; the remainder stays on the lead card or spills to CPU. Lower = the slow card bottlenecks less. Blank = use the global default; 0 = no cap.Optional explicit ratio for the lead card then the rest, e.g. 0.8,0.2. Overrides the strategy. Blank = automatic per the strategy above.
The node registers the model under this path when it does not share this machine's filesystem. Blank works when the path exists on the node too (NFS, same layout) or the model is a HuggingFace id.
Splits what a request carries — n images or videos, a list to embed or rerank, a long text to speak (by sentences), a long recording to transcribe (cut at silences), a batch of documents to OCR — over every engine that has this model, here and on cluster nodes, and merges the answers. One generation is never split; that is Cards on other machines / vLLM nodes above.
A video model is several models run one after the other; what passes between them is small. Name a cluster node for a part and that part runs there: the text encoder (UMT5-XXL, 11 GB) encodes the prompt elsewhere; the low-noise expert of a Wan 2.2 dual-expert model takes over at the expert boundary (latents handed over, a few MB); the VAE decodes elsewhere and returns the finished video. Blank = here. The node needs the same model.
Each host:port where an rpc-server listens becomes one more GPU in this model's layer split, after this machine's cards in that order (so a Weight distribution ratio reads: local cards…, then these). Known servers from the Cluster page are suggested. Decode over a LAN is workable (activations per token are small); loading and prompt processing are wire-bound — 1 GbE hurts, 10 GbE is fine. The protocol is unauthenticated: LAN or WireGuard only. Tick Split above or leave Auto — a model naming RPC servers is split by definition.
Applies to local cards and RPC servers alike. Row split trades one all-reduce per layer for using every card on every token: a win on one PCIe bus or a fast fabric, a loss on 1 GbE.
Blank = inherit the global vLLM settings. Tensor parallel splits every layer across GPUs; with nodes listed it may exceed this machine's cards and span the Ray cluster (an all-reduce per layer crosses the wire: 10 GbE is the floor, RDMA/InfiniBand the point). Pipeline parallel gives each machine a slice of layers — the one to use on ordinary Ethernet. Both can be combined (tensor × pipeline GPUs in total). With nodes listed (or pipeline > 1) coderai starts a Ray head here, runs each node's command, waits until Ray sees tensor×pipeline GPUs, then launches vLLM on Ray.
{ray_address}, {head} and {port} are filled in. The coderai-vllm image joins as a Ray worker when started with CODERAI_RAY_ADDRESS; every node must run the same vLLM build.
Runs in an isolated Python 3.10 venv (torch 2.6+cu124, transformers 4.41) behind a managed subprocess, because those pin against the server's own 3.13 / 2.11 / 5.x. Blank fields inherit the global Settings → LongCat section. Generation is three coarse-to-fine stages; Quality picks which run. Long video is generated in segments whose first conditioning frames re-render the previous segment's tail, so each segment adds the remainder — ~11 segments is a minute at 15fps.
INT8 is only supported by the official avatar-1.5 weights, which also require distilled sampling. Offloading the KV cache trades speed for VRAM across segments.
Blank = inherit the global ktransformers settings. With nodes > 1 this machine is rank 0; ranks 1… are started by the commands below (templated with {rank}, {nnodes}, {dist_init_addr}, {model}) before rank 0 launches.
These tune ds4-server for THIS model (they vary by quant/size/context). Leave blank to inherit the global ds4 settings. Context window is set by n_ctx above.
On for big models that don't fit VRAM; off for small ones that do.
Set just above this model's resident weights (+~2 GiB). ds4 defaults to half the card, which starves the cache.
Cap the expert cache (a count or NGB), prefill-chunk, kv-disk-space, etc. Blank = inherit global.
KEY=VALUE pairs. e.g. DS4_CUDA_WEIGHT_ARENA_CHUNK_MB=512 fixes weight-arena OOM under heavy expert caching. Blank = inherit global.
⚡ Performance: put the GGUF on NVMe/SSD (≈10× prefill for a streamed MoE) ·
cap the expert cache with --ssd-streaming-cache-experts N (a count, e.g. 400) — higher helps decode but too high OOMs the weight arena ·
set reserve ≈ weights + ~2 GiB · add DS4_CUDA_WEIGHT_ARENA_CHUNK_MB=512 · avoidDS4_CUDA_WEIGHT_CACHE (slower). Decode of a model bigger than VRAM is streaming-bound (~0.5 tok/s) — the real fix is a smaller quant. Context = n_ctx above.
These tune the colibri engine for THIS model. Leave blank to inherit the global colibri settings. Context window is set by n_ctx above.
1–16 concurrent cached conversations.
Engine worker cap.
Exports CUDA_EXPERT_GB. Blank = inherit global.
KEY=VALUE pairs for colibri's env-only tuning. Blank = inherit global.
Placement — where this model runs
RunPod and Host are exclusive: the model is never loaded here. To keep it local and only overflow to RunPod, leave "This machine" and use "Burst" below.
RunPod — remote GPU
This model runs on a rented RunPod GPU and is not downloaded locally. Set the model Path/ID (top) to the served model id (e.g. an HF repo) — the remote worker pulls it. Requires RunPod enabled in Settings.
No cold-boot wait (image pull + weight download, minutes) — but it bills every hour of every day, including the ones nobody uses it. Off by default.
Outside the window the warm pod is terminated and billing stops. Requests still work — one arriving out of hours cold-starts a pod, exactly as it does with no warm pod at all.
None ticked = every day. An end earlier than the start runs overnight, and belongs to the day it starts on.
How many requests the pod's own server admits at once. The defaults are sized for a shared workstation and will throttle a dedicated card long before its GPU is busy — everything over the limit gets a 429. Set these to roughly the remote engine's batch width (e.g. vLLM's max_num_seqs).
Pin this model's pods to one RunPod data centre. A network volume still wins: a pod in another region cannot attach it.
Wait for the pod to expose its port; a stuck machine is dropped and the next one tried.
Wait for vLLM to serve the model — raise this for big models (long image pull + weight download).
⚠ Frontier-size weights. Models on this engine are 100 GB and up — DeepSeek-V4 alone is ~154 GB, Kimi-K3 is measured in terabytes. Without a network volume every cold pod downloads all of it before answering, typically one to two hours of rental per boot. Set a network volume and, better, the "weights already on the volume" path so nothing is downloaded at all. The pod's disk is sized from Weights (GB) when no volume is set.
A local .gguf path means nothing on a rented machine — name the HF repo the pod should download.
Weights, uploads and adapters land here and outlive the pod. Secure Cloud only; pods are pinned to the volume's region.
Relative to the volume mount. The pod loads this instead of downloading — the only sane way to run a 150 GB+ engine model (ds4, colibri, k3).
Boots a small image and runs from a venv on the volume — the first pod builds it, later pods skip the 7 GB pull. Needs a volume; imports are slower off network storage.
Direct has no request cutoff and no body cap; coderai images bring a certificate from this install's own CA so nothing travels in clear. vLLM / llama.cpp images cannot take a certificate, so on the direct path they would be plain HTTP — the proxy is their TLS.
Models naming the same pool share one set of pods and one budget until it saturates. Needs a coderai or custom multi-model image — a vLLM or llama.cpp pod is launched for ONE model and sharing is refused.
The pod is launched requiring this token and coderai sends it. Blank does NOT mean open — one is generated.
Anyone with the pod URL can use your GPU. Only for images with no auth support.
Add a pod once the least-loaded one carries this many requests.
Give a pod back once the load no longer needs it and has not for this long. One per pass, least-loaded first, draining rather than dropping work. 0 = keep every pod until it goes fully idle (idle_timeout_s), which on a trickle of traffic means paying for the peak.
Above this a request waits for a slot instead of piling on.
Send a conversation back to the pod whose cache still holds it.
Host — your own machine
A coderai capability image (or any coderai) running on a machine you control. Always-on if the start command is blank; otherwise coderai runs the start command, waits for /healthz, and runs the stop command after the idle timeout. See docs/remote-execution.md.
Several machines form a pool: each request goes to the healthy one with the fewest in flight, a dead one is skipped for 30 s and the request moves on, and an on-demand machine is started only when none is up.
Pins it here even when its whole capability is set remote — so one video model can stay local while another runs on a pod.
Another coderai, llama-server, vLLM, a pod. Nothing is downloaded and no local VRAM is used.
Local stays primary. If a burst can't be opened the request just queues locally.
When on, the real measured VRAM is re-recorded every run and used for eviction even if "Used VRAM" is set above.
Parallel copies in VRAM
Inference
-1 = all on GPU (auto-offloads the overflow to CPU if it won't fit). N = N layers on GPU, the rest on CPU (slower, frees VRAM).
Prompt-ingestion batch. Lowering it (e.g. 256) shrinks the compute buffer so a big-context model fits at the VRAM ceiling — slower prefill. Blank = default (512).
Concurrent sequence slots. Each slot reserves its own KV/compute VRAM; keep at 1 to avoid reserving for unused parallelism. Blank = default (1). Ignored by older llama-cpp-python builds.
Pairs an mmproj GGUF with this model to add image/vision input (passed to llama.cpp as --mmproj). Auto-selected when a matching projector is detected.
Note: KV-cache quantization only applies to GGUF models on
the llama.cpp backend. HF-transformers models (incl. gemma and
other sliding-window / hybrid linear-attention architectures) ignore this and
keep an fp16 KV cache — their quantized-cache path crashes during generation.
For those, lower n_ctx to shrink the KV VRAM reserve instead.
Auto-compact context
Compacts to ~65% of the context window when the prompt reaches this %.
A separate (e.g. smaller/faster) model can write the summary while the main model answers. The dropped history is chunked to fit the chosen model's own context. Empty = the request's own model summarizes. Leave empty to inherit the global default.
By default the model's thinking is returned as a separate reasoning / reasoning_content field (think/thinking/thought tags, including the bare </think> Qwen pre-fill emits) and kept out of content. Enable this to discard it entirely. A request can override per-call with reasoning:{exclude:true}, reasoning_effort:"none", or suppress_reasoning.
bitsandbytes NF4 is the slowest 4-bit option. Quantizing to GPTQ/AWQ uses
Marlin kernels (2–4× faster) but is a heavy one-time background job; the
produced checkpoint is used automatically on the next load. Falls back to
bitsandbytes if unavailable or the architecture is unsupported.
Component Quantization (image / video pipelines)
Quantize individual pipeline components. Default uses the global 4-bit/8-bit checkboxes above.
2-bit needs optimum-quanto; GGUF file (Q5_K/Q6_K…) needs a pre-quantized .gguf path/URL.
Tip: backbone 4-bit + text encoder 8-bit + VAE None = best quality-per-GB.
Offload
Auto goes straight to model offload when the model's peak exceeds free VRAM (no full-GPU gamble). Auto borderline-aware is the same, except when the model only marginally overshoots (≤3 GB) it tries full-GPU first to keep both experts resident and use the free VRAM, falling back to model offload on OOM. model keeps only the active component on GPU (best for two-expert Wan2.2 — near full-GPU speed). group streams blocks with prefetch (lowest VRAM, fast; incompatible with bitsandbytes 4-bit → falls back to sequential). balanced fills GPU to the % below then spills.
sd.cpp Video Options
When the model won't fit entirely in VRAM, fill this % of free VRAM then spill the rest to CPU RAM. Also the cap used by auto-balanced fallback.
Lists distill-LoRA repos found in the local cache. Choose one, then pick its high/low (or single) LoRA below.
Wan2.2 A14B has two experts; the distill LoRA must be fused into both or the clip collapses to a solid colour at 4 steps. Leave blank to apply the single LoRA above to both.
A request's quantization field (e.g. turbo4) overrides the default bits. With encoding_format:"base64" the response is compact packed bytes; otherwise it returns the lossy reconstruction as floats.
simultaneous embeds on the GPU (1 = strict serialise; use for fragile cards like Polaris/RX580)
requests queued beyond concurrency before returning 429 (0 = never shed)
minimum gap between GPU starts; paces a flooding client (0 = no pacing)
Components
Works on text models too (LoRA and QLoRA). A HuggingFace repo id also works — and is the only form a vLLM/llama.cpp pod can load, since it cannot read this disk. llama.cpp needs a GGUF-converted adapter.
Tip: use <lora:name:strength> in prompts to apply at inference time
When this image model is a transformer/DiT (Z-Image, Flux, SD3) the in-process LoRA
trainer can't target it. Set a UNet-based SD1.x/SDXL model here to train identity LoRAs
against, while generation still uses this model. Leave empty to train on this model.
Every LoRA/QLoRA trained on this model runs on this machine and the nodes named
here, each on its own card with its share of the samples; the adapter's gradients are
averaged every step (a few MB — fine on 1 GbE), rank 0 here saves the result. The nodes
need the same base model. A request's own nodes overrides this.