• Stefy Lanza (nextime / spora )'s avatar
    the fleet-status endpoint could neither answer nor be believed (v0.4.34) · 9265a7ee
    Stefy Lanza (nextime / spora ) authored
    /v1/runpod/status is the one surface an operator has for seeing WHY a
    server is busy. During a pod boot wave it was both unanswerable and,
    when it did answer, wrong about the only number anyone was watching.
    
    IT COULD NOT ANSWER. Measured on a live orchestrator:
    
        /healthz            200 in   2.9 ms   (no thread needed)
        /metrics            200 in 180   ms   (no thread needed)
        /v1/runpod/status   no answer in 25 s (needs a thread)
        /v1/runpod/scale    no answer in 25 s (needs a thread)
    
    Nothing was broken. Those two are the only endpoints that hand work to a
    thread, and they drew it from the DEFAULT executor -- the pool the
    request path also draws on. 25 s of silence reads as a dead server, and
    it also meant a client lease could not be RELEASED while the fleet was
    busy, so pods kept billing with no way to stop them but the console.
    
    Three changes, each closing one way that happened:
    
      * a CONTROL pool (16 threads, fixed) that nothing in the request path
        may use -- status, scale and release run there. Its whole value is
        being empty when everything else is full, and a test pins that no
        request-path module touches it.
      * the PDF split moves to a two-wide PDFIUM pool. pdfium admits exactly
        one caller, so hundreds of concurrent documents each took a general
        thread and then queued on the same global lock: the serialisation
        was unchanged while the threads they held were the control plane's.
        Queueing two-wide costs nothing that was not already being paid.
      * the snapshot's three sources now run concurrently, each with its own
        5 s deadline, and whatever is late is reported as `degraded` rather
        than waited for. pools_schedule_status() was the half never bounded
        in v0.4.21 -- pods_status() gave up on a busy pool and this one
        waited for it -- and the ledger snapshot reads without the lock it
        shares with the writer that fires on every termination. Enforcement
        still takes that lock properly: only the snapshot may degrade.
    
    IT COULD NOT BE BELIEVED. `live_hourly_usd` summed only HEALTHY pods.
    RunPod bills from creation and a pod spends ~12 minutes booting without
    ever being healthy, so a fleet entirely in paid boot reported as free:
    
        reported  $0.59/hr      actual  $4.72/hr
        reported  $0.00/hr      actual  $7.67/hr   (13 pods, 8 serving)
    
    A client polling it concluded nothing was being spent and kept feeding
    work. It now counts every pod in the listing, and reports the serving
    and booting halves separately -- their ratio is how much of the bill is
    boot, which is the figure that says a ramp is too steep, and it did not
    exist before. Both admin pages show it; "0 pod(s) · $7.67/hr" was a real
    reading and is self-contradictory.
    
    29 tests. Full suite 3741 passed, 15 pre-existing failures unchanged.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    9265a7ee
test_live_hourly_counts_booting_pods.py 5.43 KB