• Stefy Lanza (nextime / spora )'s avatar
    runpod: let a pod learn a model at runtime · 92c264f2
    Stefy Lanza (nextime / spora ) authored
    A pod knew only what it was seeded with at boot, so any request naming
    another model got "not available" for the rest of that pod's life. That
    makes a pod a fixed catalogue rather than an extension of the local
    system, which is the whole point of it.
    
    When a remote refuses a model it does not know, the gateway now teaches
    it and retries once: resolve the model to something the remote can fetch
    (HuggingFace id, URL, or an upload when it exists nowhere else), POST it
    to the new /v1/models/register, and replay the request. Registration is
    in memory — a pod is disposable — and idempotent, so a model already
    loaded is never disturbed. Taught once per pod, tracked by (base, model).
    
    The retry is bounded to one attempt and only fires on a 400/404 whose
    body actually says the model is unknown: a 500 or an out-of-memory must
    never turn into a retry loop against a rented GPU.
    
    A model that resolves to no fetchable source is refused with that reason
    rather than registered as something the pod could never load.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    92c264f2
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...