• Stefy Lanza (nextime / spora )'s avatar
    runpod: let a pod learn a model at runtime · 92c264f2
    Stefy Lanza (nextime / spora ) authored
    A pod knew only what it was seeded with at boot, so any request naming
    another model got "not available" for the rest of that pod's life. That
    makes a pod a fixed catalogue rather than an extension of the local
    system, which is the whole point of it.
    
    When a remote refuses a model it does not know, the gateway now teaches
    it and retries once: resolve the model to something the remote can fetch
    (HuggingFace id, URL, or an upload when it exists nowhere else), POST it
    to the new /v1/models/register, and replay the request. Registration is
    in memory — a pod is disposable — and idempotent, so a model already
    loaded is never disturbed. Taught once per pod, tracked by (base, model).
    
    The retry is bounded to one attempt and only fires on a 400/404 whose
    body actually says the model is unknown: a 500 or an out-of-memory must
    never turn into a retry loop against a rented GPU.
    
    A model that resolves to no fetchable source is refused with that reason
    rather than registered as something the pod could never load.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    92c264f2
test_remote_execution.py 57.4 KB