• Stefy Lanza (nextime / spora )'s avatar
    runpod: let several models share one pod until it saturates · 69d944a9
    Stefy Lanza (nextime / spora ) authored
    Two models that both offload rented a card each, even when neither came
    close to filling one. They can now name a shared `pool` — on the model's
    runpod block or on a burst target — and land on the same pod; each
    request carries its own model name, and the pool rents a second pod only
    once the first is carrying scale_up_inflight_per_pod requests.
    
    Sharing is refused, loudly, for pod servers pinned to one model at
    launch: a vLLM pod is `--model X` and a llama.cpp pod `-hf one.gguf`.
    Allowing it would send requests to a pod that answers with the wrong
    model's weights, or 404s — a clear error at configuration time beats
    debugging that. `engine: coderai` (a whole coderai, which picks the model
    per request) is what makes a shared pool possible.
    
    The first model to create a shared pool sets its shape; later joiners do
    not overwrite it, so whichever model happens to load last cannot silently
    redefine the budget everyone is sharing. Capability pools now use the
    same rule instead of their own ad-hoc guard.
    
    Settable in the UI on both the model's pod section and its burst target.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    69d944a9
Name
Last commit
Last update
..
test_admin_auth.py Loading commit data...
test_admin_remote_ui.py Loading commit data...
test_audio_ml_backends.py Loading commit data...
test_audio_ml_endpoints.py Loading commit data...
test_broker_config.py Loading commit data...
test_broker_dispatch.py Loading commit data...
test_broker_protocol.py Loading commit data...
test_broker_streaming.py Loading commit data...
test_burst_placement.py Loading commit data...
test_manual_multimodal_test_client.py Loading commit data...
test_remote_execution.py Loading commit data...
test_studio_composed_surfaces.py Loading commit data...
test_whisper_server_local_models.py Loading commit data...