• Stefy Lanza (nextime / spora )'s avatar
    runpod: resolve, download or upload a model's weights; optional warm pods · 249da5af
    Stefy Lanza (nextime / spora ) authored
    A pod cannot see this machine's disk, so "use my local models on RunPod"
    needed a way for the pod to GET them. Three, now, picked by `source`:
    
      auto   — an explicit hf_repo, a path that already is a repo id, or a
               repo id recovered from the HuggingFace cache path
               (models--Owner--Repo). Most of a normal catalogue resolves
               with nothing typed: weights downloaded through coderai live
               in that cache, and the directory name still says where from.
      hf     — download hf_repo from HuggingFace.
      url    — download model_url directly, which is how most one-off GGUFs
               are had.
      upload — send the weights from here.
    
    Each server gets only what it can use, refused at configuration time
    rather than after five minutes of pod boot: vLLM is launched `--model`
    and takes a repo id only; llama.cpp takes `-hf` or downloads a URL with
    `-mu`; upload works only on a coderai pod, because vLLM and llama.cpp
    need the file before their container starts while a coderai pod loads
    models on demand.
    
    Upload is a real option, not a token one — on symmetric gigabit a 30 GB
    model is a few minutes — but it is the cold cost of every NEW pod, so it
    is never chosen automatically. /v1/models/upload streams to disk in
    chunks (never into memory), extracts a tar for multi-file models with
    every member checked against the destination, and registers the entry in
    memory only. An existence check by name and size makes a repeat free and
    refuses a half-copy.
    
    keep_warm keeps one pod running so nothing waits for a cold boot. OFF by
    default and deliberately: a warm pod bills every hour of every day,
    including the ones nobody uses it. Warm pods start at startup, since a
    pool created on first use would still pay the boot it exists to avoid.
    
    All of it is in the UI: weights source with the right field shown per
    choice, keep-warm, and an "always run this model locally" pin.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    249da5af
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...