• Stefy Lanza (nextime / spora )'s avatar
    runpod: let any capability rent, scale and reap its own pod · 3fb97ff1
    Stefy Lanza (nextime / spora ) authored
    The pod pool never cared that it was serving one LLM — it provisions,
    health-checks, scales and reaps whatever image it is given. Two things
    were hardcoded: the image and the /v1/models readiness probe. Both are
    now per-pool, via pod_plan():
    
      auto      vLLM, or llama.cpp for a GGUF   -> /v1/models
      coderai   your image (required)           -> /healthz
      custom    your image + docker_args        -> health_path
    
    So a capability can now be a pool instead of a fixed URL. Configure
    `remotes.pods: {"images": {...}}` (the same fields a model's runpod block
    takes) and the gateway borrows a pod per request and hands it back;
    `"endpoints": {"images": "runpod"}` is the shorthand. Everything that
    already guards LLM pods now guards these: $/hr ceilings, rolling spend
    limits, the global budget, min/max autoscaling, idle teardown, the stale
    -pod reaper, and the stats page (as `capability:<name>`).
    
    That closes the gap this work left open: an idle image or video pod is
    exactly the kind you forget about, and until now nothing would reap it.
    
    A pod that cannot be provisioned — RunPod disabled, budget hit, no GPU
    under the ceiling — returns 502 rather than falling back to running the
    model locally. Silent local fallback is a failure mode you only notice by
    watching your own VRAM vanish.
    
    22 tests, including pod acquire/release (the handle is returned even on
    the happy path, so the reaper can see it), the runpod shorthand, the
    provisioning-failure 502, and every pod_plan branch.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    3fb97ff1
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...