-
Stefy Lanza (nextime / spora ) authored
The guide told an operator which keys exist but not how to lay out a real fleet. Adds a section covering: - why a network volume is the single biggest lever: without one a pod re-downloads its weights on EVERY cold start, which is ten minutes for a 15 GB repo and one to two hours for a 150 GB MoE, paid as rental before it answers anything; - that the volume's region overrides data_center and the account setting, because a pod elsewhere cannot attach it — so two regions means two volumes; - a suggested volume layout, and venv_on_volume; - pooling: two models sharing `pool` share pods, but only CoderAI-engine models can, since a vLLM pod is launched --model X; - worked blocks for vLLM, llama.cpp/GGUF, embeddings (warm + scheduled), native-engine OCR, and the big MoE engine pods, which REQUIRE a volume because colibri and k3 never download; - LoRA/QLoRA: portable adapters travel as a reference and vLLM gets --enable-lora wired automatically, while a local-only adapter forces a CoderAI pod because vLLM and llama.cpp have no endpoint to receive one; the volume is the durable answer. Plus the two QLoRA traps — a quantization that must match what the adapter was trained against (it loads fine and silently degrades otherwise), and llama.cpp taking exactly one GGUF-converted adapter; - a checklist for before you add the tenth model. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com>
3703d7c3