• Stefy Lanza (nextime / spora )'s avatar
    runpod: a warm floor is not satisfied by a pod that no longer exists (v0.3.44) · 9c82c3f1
    Stefy Lanza (nextime / spora ) authored
    `keep_warm` stopped re-renting whenever a pod went away behind the pool's
    back — terminated from the console, reclaimed as spot, lost with its host —
    and the scaler said nothing about it. Both halves had one cause: nothing in
    a pool ever re-checks a pod's health. A handle's `healthy` flag goes false
    only when something tried to USE the pod and failed, or when the pool itself
    retired it, and a warm-floor pod is precisely the pod with no traffic to
    fail. So `maintain()` counted the handle, found `healthy_n >= keep_min`, and
    neither topped up nor had anything to log. The floor stayed "satisfied" by a
    machine that had not existed for hours, and the first real request paid the
    full ~12-minute cold boot that keep_warm is bought to avoid.
    
    (The class docstring claimed the scaler "re-checks health" for months. It
    never has. Corrected rather than implemented: a timer-probe would have to
    decide what an unanswered probe means, and that decision is what killed
    seven A40s in v0.3.41.)
    
    `reconcile_pools` instead reads the account listing the stale-pod reaper
    already fetches every ~30s — authoritative, free, and unable to mistake a
    busy pod for a dead one. Only positive evidence acts:
    
      * a dead status (TERMINATED/EXITED/DEAD/FAILED) drops the pod at once;
      * a mere absence drops it only after two listings in a row, because the
        API is eventually consistent and one page omitting a live pod used to be
        enough to unregister it — a reappearance resets the count;
      * a pod that served a request in the last 60s is kept whatever the listing
        says, which cannot mask this bug since a warm-floor pod has no traffic;
      * a provisioning pod is shielded, and a FAILED listing never gets here at
        all, so an API outage cannot empty every pool and re-rent the fleet.
    
    `PodPool.forget()` is `discard()` minus the kill: nothing is left to
    terminate, but the bill is closed in the ledger exactly as a real
    termination closes it, since the pod was billed up to its death. A borrowed
    `(shared)` pod is dropped without a ledger entry — its owner holds the bill.
    The registry is pruned through the same two-sweep rule it used to skip.
    
    The top-up now announces itself before spending, because a warm floor
    silently failing is indistinguishable from a warm floor that is fine.
    
    21 new tests in tests/test_runpod_warm_floor_rerent.py pin the caution, not
    just the fix: the failure mode in the other direction is forgetting a pod
    that is alive and billing. test_runpod_client_scale's source assertion moved
    to the new seam.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    9c82c3f1
Name
Last commit
Last update
..
conftest.py Loading commit data...
ocr_lane_stub_worker.py Loading commit data...
test_admin_auth.py Loading commit data...
test_admin_forward_auth.py Loading commit data...
test_admin_remote_ui.py Loading commit data...
test_audio_ml_backends.py Loading commit data...
test_audio_ml_endpoints.py Loading commit data...
test_autoupgrade.py Loading commit data...
test_autoupgrade_recovers_a_down_server.py Loading commit data...
test_autoupgrade_sees_training.py Loading commit data...
test_broker_config.py Loading commit data...
test_broker_dispatch.py Loading commit data...
test_broker_protocol.py Loading commit data...
test_broker_shutdown.py Loading commit data...
test_broker_streaming.py Loading commit data...
test_burst_placement.py Loading commit data...
test_character_studio_fetch.py Loading commit data...
test_cluster.py Loading commit data...
test_compose.py Loading commit data...
test_forward_auth_peer_trust.py Loading commit data...
test_front_api_auth.py Loading commit data...
test_front_inflight_leak.py Loading commit data...
test_front_ocr_swap_gate.py Loading commit data...
test_image_has_c_compiler.py Loading commit data...
test_longcat_avatar_layout.py Loading commit data...
test_longcat_bsa.py Loading commit data...
test_longcat_cancellable.py Loading commit data...
test_longcat_checkpoint_resolve.py Loading commit data...
test_longcat_component_quant.py Loading commit data...
test_longcat_config_ui.py Loading commit data...
test_longcat_external_training.py Loading commit data...
test_longcat_int8_load.py Loading commit data...
test_longcat_lora.py Loading commit data...
test_longcat_orchestration.py Loading commit data...
test_longcat_packaging.py Loading commit data...
test_longcat_placement.py Loading commit data...
test_longcat_progress.py Loading commit data...
test_longcat_prompt_cache.py Loading commit data...
test_longcat_quantize.py Loading commit data...
test_longcat_scale.py Loading commit data...
test_longcat_tasks.py Loading commit data...
test_longcat_variants.py Loading commit data...
test_longcat_vendored_source.py Loading commit data...
test_longcat_venv_env.py Loading commit data...
test_longcat_vram_estimate.py Loading commit data...
test_longcat_worker.py Loading commit data...
test_manual_multimodal_test_client.py Loading commit data...
test_model_scan_cache.py Loading commit data...
test_ocr_accent_repair.py Loading commit data...
test_ocr_chandra.py Loading commit data...
test_ocr_confidence.py Loading commit data...
test_ocr_fanout.py Loading commit data...
test_ocr_language.py Loading commit data...
test_ocr_olmocr.py Loading commit data...
test_ocr_served_model.py Loading commit data...
test_ocr_token_trace.py Loading commit data...
test_ocr_unreadable_and_dpi.py Loading commit data...
test_ocr_vram_share_and_fallback.py Loading commit data...
test_ocr_worker_lanes.py Loading commit data...
test_olmocr_pod_tls.py Loading commit data...
test_oom_priority.py Loading commit data...
test_pod_log_drain.py Loading commit data...
test_proxy_logprobs.py Loading commit data...
test_remote_execution.py Loading commit data...
test_runpod_client_scale.py Loading commit data...
test_runpod_fleet_status.py Loading commit data...
test_runpod_global_volume.py Loading commit data...
test_runpod_ocr_remotization.py Loading commit data...
test_runpod_placement.py Loading commit data...
test_runpod_pod_extra_args.py Loading commit data...
test_runpod_reaper_grace.py Loading commit data...
test_runpod_scale_down.py Loading commit data...
test_runpod_schedule_region_spend.py Loading commit data...
test_runpod_warm_floor_rerent.py Loading commit data...
test_runpod_warmup.py Loading commit data...
test_studio_composed_surfaces.py Loading commit data...
test_surya_runpod_fixes.py Loading commit data...
test_text_lora_to_pod.py Loading commit data...
test_township_api_key.py Loading commit data...
test_township_clip_geometry.py Loading commit data...
test_township_longcat.py Loading commit data...
test_township_page_js.py Loading commit data...
test_township_templates.py Loading commit data...
test_township_video_dataset.py Loading commit data...
test_upgrade_launcher.py Loading commit data...
test_video_as_b64_scope.py Loading commit data...
test_video_geometry.py Loading commit data...
test_vllm_share_self_tuning.py Loading commit data...
test_vram_releasers.py Loading commit data...
test_weight_stores.py Loading commit data...
test_whisper_server_local_models.py Loading commit data...