• Stefy Lanza (nextime / spora )'s avatar
    runpod: resolve, download or upload a model's weights; optional warm pods · 249da5af
    Stefy Lanza (nextime / spora ) authored
    A pod cannot see this machine's disk, so "use my local models on RunPod"
    needed a way for the pod to GET them. Three, now, picked by `source`:
    
      auto   — an explicit hf_repo, a path that already is a repo id, or a
               repo id recovered from the HuggingFace cache path
               (models--Owner--Repo). Most of a normal catalogue resolves
               with nothing typed: weights downloaded through coderai live
               in that cache, and the directory name still says where from.
      hf     — download hf_repo from HuggingFace.
      url    — download model_url directly, which is how most one-off GGUFs
               are had.
      upload — send the weights from here.
    
    Each server gets only what it can use, refused at configuration time
    rather than after five minutes of pod boot: vLLM is launched `--model`
    and takes a repo id only; llama.cpp takes `-hf` or downloads a URL with
    `-mu`; upload works only on a coderai pod, because vLLM and llama.cpp
    need the file before their container starts while a coderai pod loads
    models on demand.
    
    Upload is a real option, not a token one — on symmetric gigabit a 30 GB
    model is a few minutes — but it is the cold cost of every NEW pod, so it
    is never chosen automatically. /v1/models/upload streams to disk in
    chunks (never into memory), extracts a tar for multi-file models with
    every member checked against the destination, and registers the entry in
    memory only. An existence check by name and size makes a repeat free and
    refuses a half-copy.
    
    keep_warm keeps one pod running so nothing waits for a cold boot. OFF by
    default and deliberately: a warm pod bills every hour of every day,
    including the ones nobody uses it. Warm pods start at startup, since a
    pool created on first use would still pay the boot it exists to avoid.
    
    All of it is in the UI: weights source with the right field shown per
    choice, keep-warm, and an "always run this model locally" pin.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    249da5af
Name
Last commit
Last update
..
static Loading commit data...
templates Loading commit data...
__init__.py Loading commit data...
auth.py Loading commit data...
download_worker.py Loading commit data...
routes.py Loading commit data...