docs: network volumes, pools, and a recipe per kind of model
The guide told an operator which keys exist but not how to lay out a real
fleet. Adds a section covering:
- why a network volume is the single biggest lever: without one a pod
re-downloads its weights on EVERY cold start, which is ten minutes for a
15 GB repo and one to two hours for a 150 GB MoE, paid as rental before
it answers anything;
- that the volume's region overrides data_center and the account setting,
because a pod elsewhere cannot attach it — so two regions means two
volumes;
- a suggested volume layout, and venv_on_volume;
- pooling: two models sharing `pool` share pods, but only CoderAI-engine
models can, since a vLLM pod is launched --model X;
- worked blocks for vLLM, llama.cpp/GGUF, embeddings (warm + scheduled),
native-engine OCR, and the big MoE engine pods, which REQUIRE a volume
because colibri and k3 never download;
- LoRA/QLoRA: portable adapters travel as a reference and vLLM gets
--enable-lora wired automatically, while a local-only adapter forces a
CoderAI pod because vLLM and llama.cpp have no endpoint to receive one;
the volume is the durable answer. Plus the two QLoRA traps — a
quantization that must match what the adapter was trained against (it
loads fine and silently degrades otherwise), and llama.cpp taking exactly
one GGUF-converted adapter;
- a checklist for before you add the tenth model.
Co-Authored-By:
Claude Opus 5 <noreply@anthropic.com>
Showing
This diff is collapsed.
Please
register
or
sign in
to comment