• Stefy Lanza (nextime / spora )'s avatar
    runpod: let a pod learn a model at runtime · 92c264f2
    Stefy Lanza (nextime / spora ) authored
    A pod knew only what it was seeded with at boot, so any request naming
    another model got "not available" for the rest of that pod's life. That
    makes a pod a fixed catalogue rather than an extension of the local
    system, which is the whole point of it.
    
    When a remote refuses a model it does not know, the gateway now teaches
    it and retries once: resolve the model to something the remote can fetch
    (HuggingFace id, URL, or an upload when it exists nowhere else), POST it
    to the new /v1/models/register, and replay the request. Registration is
    in memory — a pod is disposable — and idempotent, so a model already
    loaded is never disturbed. Taught once per pod, tracked by (base, model).
    
    The retry is bounded to one attempt and only fires on a 400/404 whose
    body actually says the model is unknown: a 500 or an out-of-memory must
    never turn into a retry loop against a rented GPU.
    
    A model that resolves to no fetchable source is refused with that reason
    rather than registered as something the pod could never load.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    92c264f2
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...