• Stefy Lanza (nextime / spora )'s avatar
    docs: network volumes, pools, and a recipe per kind of model · 3703d7c3
    Stefy Lanza (nextime / spora ) authored
    The guide told an operator which keys exist but not how to lay out a real
    fleet. Adds a section covering:
    
    - why a network volume is the single biggest lever: without one a pod
      re-downloads its weights on EVERY cold start, which is ten minutes for a
      15 GB repo and one to two hours for a 150 GB MoE, paid as rental before
      it answers anything;
    - that the volume's region overrides data_center and the account setting,
      because a pod elsewhere cannot attach it — so two regions means two
      volumes;
    - a suggested volume layout, and venv_on_volume;
    - pooling: two models sharing `pool` share pods, but only CoderAI-engine
      models can, since a vLLM pod is launched --model X;
    - worked blocks for vLLM, llama.cpp/GGUF, embeddings (warm + scheduled),
      native-engine OCR, and the big MoE engine pods, which REQUIRE a volume
      because colibri and k3 never download;
    - LoRA/QLoRA: portable adapters travel as a reference and vLLM gets
      --enable-lora wired automatically, while a local-only adapter forces a
      CoderAI pod because vLLM and llama.cpp have no endpoint to receive one;
      the volume is the durable answer. Plus the two QLoRA traps — a
      quantization that must match what the adapter was trained against (it
      loads fine and silently degrades otherwise), and llama.cpp taking exactly
      one GGUF-converted adapter;
    - a checklist for before you add the tenth model.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    3703d7c3
orchestrator-runpod-guide.md 30.2 KB